Cloud Storage: The Complete Guide

Object Storage

Object storage is a type of storage that enables you to store unstructured data using metadata-based schemes rather than file hierarchies. These storage resources store data as objects, a group of file data and metadata with a unique identifier. You use this ID to recall the object as needed. Since you store objects in a flat structure rather than a hierarchy, all file context is stored in metadata. 

Block Storage

Block storage is a type of storage that uses abstraction to create volumes or blocks in a low-level storage device. These blocks serve as virtual hard drives that you can attach to instances or VMs to serve as persistent storage. 

Cloud File Sharing

Cloud file sharing is a type of storage that holds data hierarchically using files and folders. File storage is the standard storage type used by on-premises machines and servers. Typically, you access data through the Network File System (NFS) protocol or the Server Message Block (SMB) protocol.

Distributed Storage

Distributed storage is not a type of storage so much as a storage infrastructure. It enables you to split storage across multiple workstations, servers, or data centers. Each storage device serves as a node in a storage cluster that you manage centrally. You can create distributed storage infrastructures for object, block, and file storage. 

Distributed storage infrastructures are what provide the benefits of cloud storage, including:

• Scalability—you can scale storage horizontally by increasing the number of storage nodes in your cluster. 
• Redundancy—you can store multiple copies of data in remote locations for greater availability and durability. Syncing enables you to ensure data is mirrored. 
• Cost—you can use lower performance or commodity hardware that is linked together to provide the same storage volume as higher cost solutions.
• Performance—you can ensure that users can access data from nearby locations, reducing latency. You can also enable massively parallel access, splitting data retrieval across resources. 

Reduced Capital Expenses

Moving to public or hosted private cloud services reduces your need to purchase or maintain hardware. This means you are not responsible for the resources required to keep hardware up to date, to house, cool, or secure hardware, to monitor hardware, or to capacity plan. Since infrastructure is leased you also take on less technical debt and can redirect capital expenses to operational tasks.

Data Tiering for Cost Savings

Cloud storage services often offer multiple hardware or access tiers to choose from. This enables you to tier your data according to access priority and frequency. You can limit high performance storage resources to your most critical data and push lower priority data to lower performance tiers. This enables you to save costs without having to purchase or maintain additional hardware.

Data Redundancy and Replication

Cloud storage services typically include automatic data replication and redundancy features. These features distribute your data across servers, data centers, availability zones, or regions to ensure that data remains available. This redundancy helps protect you from hardware failures, natural disasters, and issues related to heavy traffic. 

Mobility

The remote accessibility of cloud storage enables you to work with distributed teams and users easily. You can access data stored in the cloud at any time from anywhere and on almost any device. Additionally, since cloud data access is centralized, it’s easier for IT teams to manage data without limiting accessibility. 

Disaster Recovery

In addition to the built-in data replication that most cloud storage services provide you can use cloud storage to enable reliable disaster recovery. Since cloud storage is typically remote you can use it to store failover systems or backups that will remain available even if your on-premises systems go down. 

Easier File Upload

Traditionally, users would upload files directly to servers using protocols like FTP and SFTP. This was cumbersome, slow, and error prone. Modern cloud storage services provide seamless methods for uploading files, such as desktop agents that synchronize files directly to the cloud, and convenient web interfaces for uploading files.

Learn more in the detailed guide to file upload

Cloud Complexity

On an individual level cloud resources are easy to provision and consume. Most users can manage setting up a Google Drive or Dropbox account. However, configuring storage resources for an organization with multiple users, diverse business goals, and a variety of compliance obligations is much more complex. 

When adopting cloud storage resources, you cannot simply move data and trust your IT team to do the rest. Instead, you need to plan your migration carefully and include staff with cloud experience and expertise.

Data Movement Challenges

Once you plan your migration, you may also run into challenges when moving data. The amount of data you need to migrate, the support for data formats, and the security of the transfer all play a role. 

Additionally, you may need to shut down or enable syncing for some applications and components to ensure no data is lost. If you do not manage data carefully, you may not be able effectively use it after migration. 

Cloud Security

When you store data in the cloud you have less control over it than when stored on-premises. This is caused by several main issues:

• Data is more accessible to outsiders due to Internet connectivity.
• Monitoring is more difficult due to distribution of data.
• You are reliant on the cloud provider for infrastructure security.

In combination, these factors put your data at greater risk, particularly if you do not understand which aspects of security are your responsibility. Not properly implementing access controls or securing storage resources can provide malicious parties full access to your data and accounts.

Public Cloud Storage

Public cloud storage services are based on resources that are owned, maintained, and operated by the cloud storage provider. Google, Azure, and Amazon are the three largest public cloud storage providers. 

Public cloud resources are designed for multiple tenants per server. Providers separate tenant or customer data through access controls, security policies, and data isolation practices. Some public cloud providers also offer dedicated servers for greater isolation. The resources you lease are made available over Internet connections and you can access data through web interfaces which generally use REST APIs. 

Public cloud storage services can be reserved or used on-demand. Reserved resources may require advance payment or an agreement to retain services for a specific period of time. On-demand services are available as needed and are paid for on a monthly basis. Services used are typically charged according to the number of gigabytes used and the amount of bandwidth used for data access or transfer. 

Hybrid Cloud Storage

Hybrid cloud storage is a combination of remote cloud storage (either public or private) and local resources. Storage is often implemented with proprietary software or appliances that sync with cloud resources via API. These infrastructures are typically used by organizations that want or need to keep data locally accessible. For example, due to mission-critical legacy applications or compliance restrictions. 

When operating a hybrid infrastructure you can have separate data stored in cloud or local resources, or you can sync data. For example, you can use policy engines to transfer infrequently accessed data to the cloud while retaining frequently accessed data. Meanwhile, syncing data enables you to gain the cloud's availability while maintaining low latency provided by on-site resources. 

Multicloud Storage

Multicloud storage is a strategy that uses multiple types of cloud storage services (public and private) or services from different vendors. The purpose of multicloud strategies is to diversify your services to avoid vendor lock-in, optimize use according to resource capability, or to enable use of otherwise incompatible services or applications. Multicloud storage strategies can combine native services, supplier-integrated services, and marketplace services. 

Private Cloud Storage

Private cloud storage services can be remote or on-premises. Services can be owned and operated in-house or by a cloud provider. These services provide dedicated, single tenant resources and allow greater control over your stored data.

Depending on the location of your private cloud services, you can base your infrastructure on your existing hardware, newly purchased hardware, or hardware leased from a provider. Also depending on location, you can access your stored data through private networks or through the Internet. 

The cost for private cloud services depends on whether it is self operated or not. If yes, costs are based on necessary hardware purchases, ongoing hardware maintenance, and cloud infrastructure operation and management. If a vendor operates private clouds, you are typically charged based on the resources you are using and the level of support needed.

Public Cloud Storage: What Do the Big Three Providers Offer?

The following table summarizes storage services offered by the big three cloud providers. 

Click the links above to read our in-depth guides about each of these services. Or read on to learn about each cloud provider’s storage services in brief.

AWS Storage Services

Amazon Web Services (AWS) is a top cloud provider, offering hundreds of cloud computing products and services, including storage. Popular and notable cloud services include AWS EBS, AWS EFS, Amazon S3, and Amazon S3 Glacier. 

Amazon Elastic Block Store (EBS)

AWS EBS lets you store data as blocks on persistent volumes. EBS was designed to provide persistent storage for Amazon Elastic Compute Cloud (Amazon EC2) instances and is commonly used for this purpose. 

You can use EBS for a range of purposes, including stream processing, relational databases like MySQL, NoSQL databases like Redis, data warehouses, and any workload that requires high throughput and low latency.

Read more in our detailed guide about AWS EBS

Amazon Elastic File System (EFS)

AWS EFS is a file system based in the cloud that can elastically scale to petabytes. You can mount EFS as a drive on EC2 instances and use to provide shared access to data. EFS is ideal for life and shift migrations, database backup, development and testing, and CMS. You can also leverage EFS as persistent storage for Kubernetes or Docker. 

Read more in our detailed guide about AWS EFS

Amazon S3

Amazon Simple Storage Service (Amazon S3) provides cloud-based object storage for all data types. S3 offers unlimited scalability and 99.999999999% durability, which you can leverage for many purposes. For example, you can use S3 for data backup and archives, storing user-generated data, and also as a data lake collecting big data.

Read more in the detailed guide about S3 Storage

Amazon S3 Glacier

Amazon S3 Glacier provides cold storage for Amazon S3, which is offered at relatively low costs. However, additional fees are charged for data retrieval. S3 Glacier is typically used for archives and long-term backup. 

Additional Storage-Related Services

Here are several useful services to consider when using AWS cloud storage:

• AWS backup—a fully-managed AWS service that helps you centralize all backup activities, including AWS resources and on-premise backup. 
• AWS Storage Gateway—helps you integrate on-premise storage with AWS-based cloud storage. 
• AWS FSx Lustre—a cloud-based file system designed and optimized for high-performance computing (HPC) and compute-intensive workloads.

AWS FSx for Windows File Server—a Microsoft Windows file system, designed to help you easily move Windows-based applications to AWS cloud environments.

Azure Storage Services

Microsoft Azure offers a wide range of cloud computing services, designed to provide scalability for enterprises and enable hybrid infrastructure implementations. Notable and popular Azure storage services include Blob Storage, Azure Files, Azure Queue Storage, Table Storage, and Azure Managed Disk.

Learn more in our detailed guide to Azure Storage services

Blob Storage

Blob Storage lets you store massive amounts of unstructured data as objects in the Azure cloud. You can organize your files in containers, which can be accessed by end-users and applications via HTTP connections. 

When you use Blob Storage, there is no need for publishing and access-policy tuning. Additionally, there is no limit to the number of blobs or containers you can use, provided you stay within the storage limits of your account.

Azure Files

Azure Files offers fully-manage file storage based in the Azure cloud. It is accessible through the Server Message Block (SMB) protocol, storage client libraries, and REST APIs. Azure Files is typically used to achieve high availability for network file shares. You can, for example, use this service to provide multiple VMs with read and write access to the same files. 

Learn more in our detailed guides to:

• Azure Files
• Azure file storage services

Azure Queue Storage

Azure Queue Storage lets you store messages and exchange data between components in the cloud or on-premises. You can store messages communicated asynchronously and share them between independent application components, using HTTP or HTTPS.

You can use Queue Storage for a variety of cases. For example, you can use this services for messages communicated between Azure Web and Worker roles, as well as for processing backlog messages. 

Table Storage

Azure Table is a NoSQL data pool that lets you store large amounts of non-relational structured data. Table allows you to insert and retrieve data through authenticated API calls or client libraries. The source can be located outside or inside the Azure cloud.  

You can use Azure Table to store flexible datasets at scale, without establishing relationships between the sets. The service lets you organize data in tables. Each table can hold collections of row entities and their properties. Common use cases include storing customer information and diagnostic logs.

Learn more in our detailed guide to Azure Table Storage

Azure Managed Disk

Azure Managed Disk provides a fully-managed block-level volumes service. A managed disk works as a virtualized physical disk, which operates in the Azure cloud and used with Azure virtual machines (VMs). A managed disk is designed as a page blob, which acts like a random storage object. You can define the size and type of the disk and then provision the disk. 

Google Cloud Storage Services

Google Cloud offers many services, designed for a wide range of industries. Popular and commonly used Google Cloud storage services include Cloud Storage, Google Cloud Persistent Disks, and Google Cloud Filestore.

Learn more in our detailed guide to Google Cloud Storage

Cloud Storage

Google Cloud Storage is an object storage service offering high durability, low latency, and geo-redundancy when storing data in multi or dual regions. Google Cloud offers many useful services, such as fine-frain permissions, object versioning, and retention policies. 

You can use Google Cloud Storage for many scenarios, including backups, archives, and ML-based analysis. Google Cloud serves as the underlining storage of many services. 

Google Cloud Persistent Disks

Persistent Disk offers highly durable block storage for VM instances. A persistent disk works like a storage device, which instances can access just like a physical disk. You attach persistent disks to instances running in Compute Engine - Google’s VM service - and Google Kubernetes Engine (GKE) - Google’s fully-managed Kubernetes service.

Learn more in our detailed guide to GCP Persistent Disks

Google Cloud Filestore

Filestore is Google’s fully-managed file storage service. You can use Filestore to create shared file systems for your applications. Filestore provides network attached storage (NAS) for your Compute Engine and GKE instances. It works in the cloud, on-premises, and in hybrid scenarios.

Learn more in our detailed guide to Google Cloud Firestore

AWS Backup

AWS Backup is a service that you can use to create backups of EBS, EFS, DynamoDB, and RDS services. It also includes an integration with AWS Storage Gateway that enables you to create backups of on-premises data. This service enables you to backup most AWS data when combined with the native snapshot capabilities that are included in many of AWS’s other services.

Through the AWS Backup console, you can manage backups from across your services. This includes determining which storage services you store backups in, who has access to backup data, and how long you retain backups for.

Another way to backup data on AWS is using AWS snapshots, also known as EBS snapshots. Elastic Block Storage (EBS) volumes serve as virtual hard disks that can be attached to EC2 instances. A snapshot creates a point-in-time backup of an EBS volume, storing it in Amazon S3.

Azure Backup

Azure Backup is a solution you can use to backup Azure service or on-premises data. It enables you to automate and manage backups and their life cycles. This service also integrates with Recovery Services vaults, storage resources designed specifically for backup data. 

Azure Backup and Recovery Services are part of a collection of services Azure offers for backup creation and management. The other main service is Azure Site Recovery. This service enables you to create and remotely store backups of your data and services which you can then use for disaster recovery or as failover services. 

Google Cloud Backup

Unlike AWS or Azure, Google Cloud does not offer a specific service for backup creation or management. Instead, it enables you to store backups in lower tier (i.e., cheaper) storage services. Your primary storage options include:

• Nearline Storage—designed for data that is accessed once a month or less. This option is best suited to your most recent backup or partial backups.
• Coldline Storage—designed for data that is accessed once a year or less. This option is best suited to disaster recovery backups or archived backup data. 

These are reasonable options in Google Cloud but are not as functional with other cloud providers. The difference is that Google’s services provide access with sub-millisecond latency while cold and archive storage in AWS or Azure can take several hours or days to retrieve. 

AWS Cloud Database Services

AWS database services include:

• Amazon RDS—a relational database service that supports your choice of six engines, including SQL Server, MariaDB, Oracle, PostgreSQL, MySQL, and Amazon Aurora. This is a fully managed service that you can manage through CLI, API, or the Console. 
• Amazon Aurora—a proprietary relational database that offers PostgreSQL and MySQL compatibility modes. This database is designed for high performance and integration with AWS services. 
• Amazon DynamoDB—a document and key-value database designed for low latency, affordable pricing, high performance, and durability. This is a fully managed service for production level workloads. 
• Amazon ElastiCache—an in-memory data store that you can use in place of a traditional database. It is compatible with Redis and Memcached and provides scalability, high performance, and super low latency. 
• Amazon Neptune—a fully managed graph database that is optimized for high speed querying. It supports RDF and Property Graph data models and SPARQL and Gremlin languages. 
• Amazon Timestream—a fully managed time series database designed for analytics, DevOps, and IoT workloads. It enables you to stream data and perform time sensitive queries. 
• Amazon Quantum Ledger Database—a fully managed ledger database that you can use to verify transactions cryptographically. Stored data is transparent and immutable, making it ideal for auditing and financial transactions. 

Azure Cloud Database Services

Azure database services include:

• Azure Cosmos DB—a fully managed, multi model database. It enables you to define schema and indexes from your workloads and application for maximum flexibility. CosmosDB includes API support for Table, SQL, MongoDB, Gremlin, Cassandra, Spark, and ectd. 
• Azure SQL Database—a fully managed database based on the SQL Server engine. It is highly available, scalable, and includes serverless options. You also have the option of bringing existing SQL Server licenses from on-premises. 
• Azure Database for MySQL—a fully managed database service based on the MySQL Community edition. You can integrate it with Azure Kubernetes Service and Azure App Service. 
• Azure Database for PostgreSQL—a fully managed database for PostgreSQL Hyperscale. You can use this service in the cloud or on-premises. It is extensible through plugins, including for Azure Data Studio, Timescale DB, Visual Studio Code, and PostGIS.
• SQL Server on Virtual Machines—a service that enables you to host SQL server on VMs with hybrid connectivity. You can use this database with Windows or Linux and use it to extend support for SQL Server 2008. 
• Azure Synapse Analytics—an analytics service that combines big data analytics with data warehousing. It is integrated with SQL and Apache Spark engines and can be integrated with CosmosDB. 
• Azure Data Explorer—a data analytics service that you can use to perform real-time analytics on streaming data. You can use it to perform time series analyses and query big data. 
• Azure Cache for Redis—a fully managed, in-memory data store that supports Redis workloads. It includes features for built-in security, scalability, and reliability. 
• Azure Database for MariaDB—a fully managed database based on the community version of MariaDB. It supports a variety of open source frameworks and includes features for high availability, scalability, and built-in security. 

Google Cloud Database Services

Google Cloud database services include:

• Cloud SQL—a fully managed database that provides support for SQL Server, PostgreSQL, and MySQL workloads. It includes features for automated backups, data replication, and failover. 
• Cloud Spanner—a fully managed relational database that is ACID compliant, globally distributed and supports automatic sharding. It also offers multi-regional availability and transparent synchronous replication. 
• BigQuery—a scalable, serverless, multi-cloud data warehouse. You can use it to perform predictive and real-time analytics with built-in machine learning. 
• Cloud Bigtable—a fully managed NoSQL database designed for operational and analytical workloads. It is cluster based and can scale to hundreds of nodes with built-in replication and high availability.
• Cloud Firestore—a fully managed document database that you can use to store, sync, and query application data. It includes client libraries for offline support and live synchronization and integrates with Firebase. 
• Firebase—a NoSQL database that provides real time data synchronization. You can use it to collaborate globally, support serverless applications, and provide offline support.
• Cloud Memorystore—a fully managed in-memory data store that supports Memcached and Redis. You can use it to migrate caching layers and create application caches. It includes features for automatic failover, monitoring, patching, and high availability.