Gone are the days when there was a need to provision raw storage inside the server for “fast” data access. Today, there are so many cloud storage options competing for your data, and you can mix and match storage across the different providers.
Server-less apps can run Lambda functions drawn from an AWS S3 bucket, convert a legacy OS from an Azure Blob into a cloud server, mount volumes from Amazon EBS and EFS via NFS or use Azure File Storage via SMB, which every Windows PC user loves.
Collective Wisdom of the Cloud
In startup culture, there’s little hesitation before reaching straight for the cloud to cover all storage needs. Startups do not even think about purchasing massive onsite network attached storage (NAS). They prefer not to invest in infrastructure: wifi and an electrical socket at a shared working space is all the infrastructure they need (plus a laptop!)
As startups grow, they can benefit from the collective storage wisdom available in the cloud by using systems like Cloud Network Attached Storage, saving them development time and offering high availability.
Flexible startups build their applications in micro-services that talk to each other. Each micro-service uses its own ideal solution. Some of these require fast, cheap, and flexible NoSQL databases stored at either Azure or AWS, while images are stored on an AWS Blob.
In order to be safe, a startup might store the backups in a compressed format in AWS S3 for a month, then archive it in cold storage like Amazon Glacier or Azure Cold Storage. During the development stage, they keep the NoSQL DB on their AWS EBS or Azure Virtual Disk. When they go live, this is shifted to High Availability Cloud Databases that are regularly backed up and patched by AWS or Azure.
How about more established small or large enterprises?
They’re also migrating their data to the cloud. Old onsite NAS systems are too clunky and expensive to be upgraded, and with pay-as-you-go billing, enterprises can benefit from a reduced IT budget that comes from moving to the cloud. They no longer need to purchase the storage they think they’ll need for the next few years since they can grow as they go.
Enterprises usually need a hybrid solution to combine their legacy NAS and local block storage systems with their new cloud storage needs. Great hybrid solutions are available that enterprise storage features, such as control over all their legacy storage, in the same window in which they control their highly available distributed data on AWS and Azure.
Block Storage in AWS and Azure
Block storage divides your files into equally sized pieces. This allows users to change just a small piece of the data, and do so quickly.
The blocks are always connected to a cloud server, such as Azure VM or Amazon EC2. They store the OS to boot up the virtual server.
- Amazon EBS (Elastic Block Storage) is the basic storage device for EC2.
- Azure Virtual Disks, that connect to Azure Virtual Machines, also use a block storage device. The entire server image is stored in a block device.
Cloud block storage come as traditional magnetic hard drive disks or with fast Solid State Drive (SSD). An SSD is faster, since it doesn’t spin, and is an integrated circuit with memory.
(Above: Screenshot of EBS volumes connected to various EC2 instances)
For high availability, both AWS and Azure perform replication of your VM files to different zones. When you mistakenly damage your VM, this is quickly replicated. You can take snapshots as backup to recover damaged VMs for an extra fee.
The snapshots are also stored on block storage, though AWS puts them on S3. Snapshots stored on S3 cannot be accessed via the S3 panel or S3 API. On Azure, you can launch your own type of operating system that you have stored in a VHD file. Using Azure File storage, you upload the VHD to a Blob. Then, simply launch this as a VM.
Warning! When you delete your VM in Azure, the original uploaded VHD file is deleted and becomes unrecoverable.
Amazon, however, doesn’t allow you to launch from your own VHD, but you can develop with their many different OS offerings and save them as an image. Plus, when deleting a VM created from your own image, you don’t lose the image.
You need to configure a VM to mount the EBS or Virtual Disks. Once attached to a VM you can setup NFS, SMB, iSCSI, fwiw, or CIFS to access the Virtual Disk externally.
You cannot access them at all unless they are connected to a VM. Even if you need access to a tiny configuration file, the disk must first be mounted to some VM.
To extend or shrink cloud block storage, you need to first detach the volumes.
Amazon Linux automatically recognizes the new volume size; however, others usually need more work within the OS to be able to use the changed volume.
Databases and local applications generally work faster on connected block storage. AWS even offers provisioned IOPS SSD for extra fast read and writes. On Azure, provisioning extra Virtual Disks gives you extra IOPS per disk.
How Much Does Block Storage Cost?
1. AWS EBS: $.045 per GB for HDD, SDD $.10 per GB. Prices go up for greater IOPS.
Free tier: Get 30 GB SDD
2. Azure Virtual Disk: $.05 per GB for HDD. SSD are $19.71 for 128 GB per month.
Azure recently added a new Virtual Disk called Azure Managed Disks, which is more flexible and more easily backed up than EBS or Azure Virtual Disks.
Unlike the other options, you can change the volume size and available IOPs. You can also easily schedule a snapshot, though the snapshots aren’t incremental like AWS EBS snapshots. Azure Managed Disks start at $19 a month for 128 GB offering 500 IOPs.
When you think of cloud block storage, you should view them like a local drive on your physical computer.
Object Storage in AWS and Azure
Object storage is another category of cloud storage for your data. In general, it is data accessed and processed from an application.
Both AWS S3 and Azure Blob Storage are massively scalable object storage services for unstructured data. Object storage has all the data stored together. Unlike block storage, you cannot change parts of the object.
The largest S3 and Azure objects reach around 5 terabytes. You need to be well organized in your application design before you start such heavy moving. If your data is too big to send from your old data center to Amazon, you can physically ship it to them.
This is a unique Amazon solution, as Amazon.com is still the world’s largest logistical shipper. Shipping parcels to them with petabytes of data is quicker than the fastest Direct Connect available.
(Above: Screenshot of S3 - Static website settings)
Amazon’s S3 storage is the 800-pound gorilla of cloud storage. It offers a lot of functionality, it has been around the longest, and many applications have been developed that use it.
Cpanel comes with a backup solution to S3. You can’t natively mount S3 as a network drive, but Dropbox does just that and allows you to access S3 files from any device you can think of. Netflix also stores its video streaming service on S3.
S3 has the added functionality that it is easily configured as a static website for any domain. Just assign the bucket name the same as the domain and point the Amazon DNS to it. S3 also has easy security, backup, and delete scheduling from the AWS Panel.
Server-less apps using Lambda or Azure functions can draw the code from S3 or Azure storage. This allows for blurring the lines drawn by requiring block storage for VMs.
By using S3 or blobs for your applications, you can save money and benefit from extensive APIs. Azure Blobs are only accessible through the Azure Portal or programmatically via API.
You can write the program in any language you like. Azure has recently added S3 Api Proxy, so you can simply change your DNS to point to Azure with any application already designed to use S3. Azure has also added the ability to make html files stored in blobs to be static websites, also like S3.
How Much Does Object Storage Cost?
1. AWS S3: .023 per GB Per month through first 50 TB, then the price goes down. Prices are plus network usage.
Free tier: Get 5 GB
2. Azure Blob Storage: Starts at $0.0184 per GB for hot storage.
*Both S3 and Azure Blob prices go up for greater redundancy.
Other fun combinations of cloud storage are easily configured for common web applications.
WordPress can store the PHP program on ECS, EBS, or an Azure Disk. You can use the common configuration of a local database or use any of the MYSQL compatible managed database services from AWS or Azure.
WordPress can be set up to store all of its images on S3. Combining managed DB with offsite image storage greatly speeds up access to the WordPress site and can offer high availability through setting S3 to make the images available in all regions, bringing them closer to the client.
This requires less compute power, so you can use a cheap EC2 or Azure VM to run the WordPress application.
Cloud Network Attached Storage (NAS)
In the past you needed to plan for the future and purchase more storage to anticipate the application’s storage needs, whether they were hard drives, NAS storage, or tape backup systems.
Now, you can safely mount tons of data in the cloud without any initial investment or over-provisioning.
Azure has an extremely easy-to-use service called Azure File Storage.
- From your Azure storage account, add a new file share.
- Select connect to get the Windows “Net Use” command, as well as the Linux “mount” command for Samba, since Azure File storage uses the SMB protocol for mapping.
- Then your new mount is available.
You can store an unlimited number of files and are billed only for what you use.
(Right: Screenshot of Azure File Storage command to mount the volume on Windows and Linux)
AWS recently added a new service called Elastic File System (EFS) - like Azure File storage.
- You can mount EFS to EC2 servers. EFS uses NFS V4.0 /4.2 for mounting, but is presently only available in 4 AWS regions.
- You need a VPN or Direct Connect line to AWS to mount them on-premises , so EFS can’t be easily accessed.
Many startups need a more integrated system than we have discussed. More mature NAS solutions offer a console to manage not only your storage, but also your data which is spread across cloud providers, their data regions, and your own data center.
(Above: Netapp OnCommand Webpage for creating a new EBS Volume with high availability duplication)
You can manage your own data center using NetApp’s and other NAS equipment, as well as data you store in the cloud on AWS and Azure.
ONTAP Cloud offers high availability replication, bringing the data to a data center or public cloud provider near you. Existing NetApp customers can simply lift and shift data to AWS or Azure through NetApp data replication technology.
NetApp also offers data de-duplication and compression, as well as encryption for your data security. OnCommand Cloud Manager can manage your snapshots for easy backup and recovery. All of this is done from a single interface without any additional investment.
You can access your data through standard protocols like NFS, CIFS and iSCSI.
NetApp also offers Bring Your Own Licenses (BYOL) for these instances. Also note that with ONTAP storage efficiencies, you will require less public cloud storage to store and manage your data. This means that ONTAP Cloud not only offers you multi-protocol storage and enterprise class data management, but also it can be more cost effective.
What’s the Cost?
1. AWS EFS: $.30GB per month.
Free tier: Get 5GB
2. Azure File Storage: $.08 per GB per month.
*Both AWS and Azure add fees for access to the data
3. ONTAP Cloud: From $0.75 per hour to $2.90 per hour depending on capacity & subscription model. BYOL option available as well. This is in addition to the AWS or Azure and storage fees.
Consider Your Cloud Storage Options
The sky's the limit in cloud storage. Take a few moments to think about your needs, then configure them. Just change the settings if you change your mind or make a mistake, since you are only billed for what you use.
Startups have long held the reputation of being flexible and quick to evolve, whereas enterprises were all about moving slowly and taking a lot of time to change. The new scalability the cloud enables changes the nature of this perception.