Amazon Web Services (AWS) offers various kinds of cloud storage systems, and users can sometimes be confused about which one is the right choice for their cloud storage operation.
This article will talk about three common storage services: AWS Elastic Block Store (EBS), Simple Storage Service (Amazon S3), and Elastic File System (EFS). Though there have been other discussions on this topic, this comparison of the three systems will go in-depth and give you the tools to identify the optimal storage option for you.
EBS, EFS, and Amazon S3
AWS Amazon S3 is a static storage service useful for static website hosting, media distribution, version management, big data analytics, and archiving. AWS EBS is a persistent storage device that can be used as a file system for databases, application hosting and storage, and plug and play devices.
Compared with these two storage systems, AWS EFS is the new kid on the block. EFS is excellent as a managed network file system wherein the file system can be shared across different EC2 instances. EFS works like NAS devices and works well for big data analytics, media processing workflows, and content management.
With that brief introduction, it is very important to know which is a more suitable storage service for your specific needs. People make their decisions based on their requirement as well as parameters such as cost, performance, and access type.
|Storage Size||No limit on number of objects||Maximum storage size of 16 TB||No limitation on the size of the file system|
File Size Limitation
|Individual Amazon S3 objects can range from a minimum of 0 bytes to a maximum of 5TB||No limitation on file size in EBS disk||Single files have a maximum size of 52TiB|
Data Throughput and I/O
|•Supports multipart upload. It is recommended for capability of objects larger than 100MB •The largest size of a single object uploaded using PUT API can be of 5GB||SSD- and HDD-backed storage types •Use of SSD backed and Provisioned IOPS is recommended for dedicated IO operations as needed||Default throughput of 3GB|
|•Highly scalable managed service supports 100 PUT/LIST/DELETE requests per second by default•Can automatically scale till 300 PUT/LIST/DELETE requests per second or more than 800 GET requests per second||•Manually Scale the size of the volumes without stopping instance.•Baseline performance of 3 IOPS per GB for General Purpose volume•Use Provisioned IOPS for increased performance||•Highly Scalable Managed Service•Supports up to 7000 file system operations per second|
|•Stored data stays in the region. •Replicas are made within the region in multiple availability zones•Amazon S3 objects can be copied to other region using the cross region replication feature||•Data stored stays in the same Availability zone. •Replicas are made within the AZ for higher durability||•Data stored stays in the region.•Replicas are made within the region|
|Accessible over internet based on access policy configured||Can be accessed only by single EC2 instance||Can be accessed by 1 to 1000s of EC2 instances from multiple AZs, concurrently|
File Permissions/ File System
|•Can be mounted as a file system, but it is not recommended•Unlike traditional file systems, bucket permissions do not pass on to the folders by default (bucket policies can be used to achieve this)||Supports various file systems, including ext3 and ext4||•File storage service for use with AWS EC2 •EFS can be used as network file system for on-premise servers too using AWS Direct Connect.|
Supported Encryption Mechanisms
|•Server Side Encryption with Amazon S3-Managed Keys (SSE-Amazon S3),AWS KMS-Managed Keys (SSE-KMS), and with Customer-Provided Keys (SSE-C) •Client Side Encryption using an AWS KMS–Managed Customer Master Key (CMK) andusing a client-side master key||Uses an AWS KMS–Managed Customer Master Key (CMK) and AES 256-bit Encryption standards||Uses an AWS KMS–Managed Customer Master Key (CMK) and AES 256-bit Encryption standards|
|•Using bucket policies and user policies
- Managed with ACLs
- Pre-signed URL access based on IAM
- User based Authentication such as IAM
|99.99% available||99.99% available||Highly available (No public SLA)|
Availability Zone Failure
|Can withstand up to two concurrent AZ failures||Cannot withstand AZ failure without point-in time EBS Snapshots||Every file system object is redundantly stored across multiple Availability Zones so it can survive one AZ failure.|
|Eleven 9’s of durability
|20 times more reliable than normal hard disks||Highly durable (No public SLA)|
Eventual Consistency Problem
|AWS Amazon S3 offers eventual consistency for overwrite PUTS and DELETES in all regions.||No such issues||No such issues|
Of the three options, Amazon S3 is cheapest and can be configured to be accessed from anywhere. Using Amazon S3 is like taking a write-once, read-multiple-time approach. EFS is almost 10 times more expensive than EBS, but the cheaper EBS can only be accessed by one EC2 instance at a time. This can be problem in a clustering or distributed app. However, EBS can give better performance than EFS. EFS gains a point on EBS in its ability to be used as shared network storage.
Static EBS vs. Auto-Scaled EFS
The differences between using EBS and using EFS can be compared to parking a car in a home garage versus parking a car in a rental parking lot. In general, only you use your home garage: you tend to use your car frequently and the space is highly secure.
In a rental parking lot there are lots of people including yourself sharing a common space available to accommodate a large number of individuals. In this analogy EBS is like the home garage, and EFS is that big rental parking lot.
EBS is generally used for persistent block storage. It is ideal for databases and other low latency interactive applications which require high IOPS or throughput and specifically low latency with consistent and predictable performance.
Now, there is an argument that you can use EFS instead of EBS so you won’t have to worry about space. But building a parking lot to have extra space just for yourself is not a smart idea: it will cost more and the space will go unused. Plus, EBS volumes can scale up or down with a single API call and without stopping the instance.
So, when is it a good idea use EFS?
EFS is designed for the huge amounts of data that big data workloads and analytic applications generate; amounts of data that it are so large it is impossible for an EBS EC2 instance to access for processing or analyzing. Because of that, EFS outweighs EBS in media processing workflows, content management, and web serving.
Inexpensive Amazon S3 vs. High Performance EFS
Because of Amazon S3’s eventual consistency problem, Amazon S3 might be little slow when compared to EFS, which has Max I/O performance modes that will have higher write speeds and throughput.
EFS can be accessed by multiple EC2 instances from multiple Availability Zones in the same region at any point in time; Amazon S3 can be accessed from anywhere by API’s. Amazon S3 guarantees eleven 9’s of durability but EFS does not. There is also a limit to the number of EFS instances that an account can launch.
If you’re thinking of trying to use EFS for regular use, individual instances, you are not the first person to think of it. Many have tried to implement that solution, but in the end, they’ve all discovered t that EFS does not do well with small general purpose workloads and is better suited for larger workloads.
Conclusion: A Balance of Cost and Necessity
In the end, the deciding factor will most likely come down to how much you can afford to pay for storage. Amazon S3 comes with a lot of features, but comes various pricing parameters such as;
- Storage used per month
- Number of Requests made (Eg: POST, GET etc)
- Amazon S3 Inventory
- Amazon S3 Analytics
- Storage Class Analysis
- Amazon S3 Object Tagging
- Data Transfer per GB out of Amazon S3
- Amazon S3 Transfer Acceleration
Compared to Amazon S3, EBS pricing is simpler and it includes per/GB storage allocated per month, Provisioned IOPS, and EBS snapshots. EFS pricing is even more straightforward: you just pay for the storage used measured in GB/month.
So Amazon S3 looks the cheapest but that might not always be the case. For performance, EBS is the fastest due to high IOPS, but when it comes to shared and scalable file systems EFS is the best option.
When deciding on a storage system there is more to consider than just costs. As no one size fits all, the right storage system for your operation will be the one that meets your specific needs.
The first step is to identify your use case parameters such as storage used, requests made, bandwidth consumed, and performance requirement. This information will help you calculate your expected storage costs and enable you to find the storage system that best meets your requirements without overpaying for them.