Cloud Storage: The Complete Guide

Public Cloud Storage

Public cloud storage services are based on resources that are owned, maintained, and operated by the cloud storage provider. Google, Azure, and Amazon are the three largest public cloud storage providers. 

Public cloud resources are designed for multiple tenants per server. Providers separate tenant or customer data through access controls, security policies, and data isolation practices. Some public cloud providers also offer dedicated servers for greater isolation. The resources you lease are made available over Internet connections and you can access data through web interfaces which generally use REST APIs. 

Public cloud storage services can be reserved or used on-demand. Reserved resources may require advance payment or an agreement to retain services for a specific period of time. On-demand services are available as needed and are paid for on a monthly basis. Services used are typically charged according to the number of gigabytes used and the amount of bandwidth used for data access or transfer. 

Hybrid Cloud Storage

Hybrid cloud storage is a combination of remote cloud storage (either public or private) and local resources. Storage is often implemented with proprietary software or appliances that sync with cloud resources via API. These infrastructures are typically used by organizations that want or need to keep data locally accessible. For example, due to mission-critical legacy applications or compliance restrictions. 

When operating a hybrid infrastructure you can have separate data stored in cloud or local resources, or you can sync data. For example, you can use policy engines to transfer infrequently accessed data to the cloud while retaining frequently accessed data. Meanwhile, syncing data enables you to gain the cloud's availability while maintaining low latency provided by on-site resources. 

Multicloud Storage

Multicloud storage is a strategy that uses multiple types of cloud storage services (public and private) or services from different vendors. The purpose of multicloud strategies is to diversify your services to avoid vendor lock-in, optimize use according to resource capability, or to enable use of otherwise incompatible services or applications. Multicloud storage strategies can combine native services, supplier-integrated services, and marketplace services. 

Private Cloud Storage

Private cloud storage services can be remote or on-premises. Services can be owned and operated in-house or by a cloud provider. These services provide dedicated, single tenant resources and allow greater control over your stored data.

Depending on the location of your private cloud services, you can base your infrastructure on your existing hardware, newly purchased hardware, or hardware leased from a provider. Also depending on location, you can access your stored data through private networks or through the Internet. 

The cost for private cloud services depends on whether it is self operated or not. If yes, costs are based on necessary hardware purchases, ongoing hardware maintenance, and cloud infrastructure operation and management. If a vendor operates private clouds, you are typically charged based on the resources you are using and the level of support needed.

Object Storage

Object storage is a type of storage that enables you to store unstructured data using metadata-based schemes rather than file hierarchies. These storage resources store data as objects, a group of file data and metadata with a unique identifier. You use this ID to recall the object as needed. Since you store objects in a flat structure rather than a hierarchy, all file context is stored in metadata. 

Block Storage

Block storage is a type of storage that uses abstraction to create volumes or blocks in a low-level storage device. These blocks serve as virtual hard drives that you can attach to instances or VMs to serve as persistent storage. 

File Sharing

Cloud file sharing is a type of storage that holds data hierarchically using files and folders. File storage is the standard storage type used by on-premises machines and servers. Typically, you access data through the Network File System (NFS) protocol or the Server Message Block (SMB) protocol.

Distributed Storage

Distributed storage is not a type of storage so much as a storage infrastructure. It enables you to split storage across multiple workstations, servers, or data centers. Each storage device serves as a node in a storage cluster that you manage centrally. You can create distributed storage infrastructures for object, block, and file storage. 

Distributed storage infrastructures are what provide the benefits of cloud storage, including:

• Scalability—you can scale storage horizontally by increasing the number of storage nodes in your cluster. 
• Redundancy—you can store multiple copies of data in remote locations for greater availability and durability. Syncing enables you to ensure data is mirrored. 
• Cost—you can use lower performance or commodity hardware that is linked together to provide the same storage volume as higher cost solutions.
• Performance—you can ensure that users can access data from nearby locations, reducing latency. You can also enable massively parallel access, splitting data retrieval across resources. 

Reduced Capital Expenses

Moving to public or hosted private cloud services reduces your need to purchase or maintain hardware. This means you are not responsible for the resources required to keep hardware up to date, to house, cool, or secure hardware, to monitor hardware, or to capacity plan. Since infrastructure is leased you also take on less technical debt and can redirect capital expenses to operational tasks.

Data Tiering for Cost Savings

Cloud storage services often offer multiple hardware or access tiers to choose from. This enables you to tier your data according to access priority and frequency. You can limit high performance storage resources to your most critical data and push lower priority data to lower performance tiers. This enables you to save costs without having to purchase or maintain additional hardware.

Data Redundancy and Replication

Cloud storage services typically include automatic data replication and redundancy features. These features distribute your data across servers, data centers, availability zones, or regions to ensure that data remains available. This redundancy helps protect you from hardware failures, natural disasters, and issues related to heavy traffic. 

Mobility

The remote accessibility of cloud storage enables you to work with distributed teams and users easily. You can access data stored in the cloud at any time from anywhere and on almost any device. Additionally, since cloud data access is centralized, it’s easier for IT teams to manage data without limiting accessibility. 

Disaster Recovery

In addition to the built-in data replication that most cloud storage services provide you can use cloud storage to enable reliable disaster recovery. Since cloud storage is typically remote you can use it to store failover systems or backups that will remain available even if your on-premises systems go down. 

Cloud Complexity

On an individual level cloud resources are easy to provision and consume. Most users can manage setting up a Google Drive or Dropbox account. However, configuring storage resources for an organization with multiple users, diverse business goals, and a variety of compliance obligations is much more complex. 

When adopting cloud storage resources, you cannot simply move data and trust your IT team to do the rest. Instead, you need to plan your migration carefully and include staff with cloud experience and expertise.

Data Movement Challenges

Once you plan your migration, you may also run into challenges when moving data. The amount of data you need to migrate, the support for data formats, and the security of the transfer all play a role. 

Additionally, you may need to shut down or enable syncing for some applications and components to ensure no data is lost. If you do not manage data carefully, you may not be able effectively use it after migration. 

Cloud Security

When you store data in the cloud you have less control over it than when stored on-premises. This is caused by several main issues:

• Data is more accessible to outsiders due to Internet connectivity.
• Monitoring is more difficult due to distribution of data.
• You are reliant on the cloud provider for infrastructure security.

In combination, these factors put your data at greater risk, particularly if you do not understand which aspects of security are your responsibility. Not properly implementing access controls or securing storage resources can provide malicious parties full access to your data and accounts.

AWS Backup

AWS Backup is a service that you can use to create backups of EBS, EFS, DynamoDB, and RDS services. It also includes an integration with AWS Storage Gateway that enables you to create backups of on-premises data. This service enables you to backup most AWS data when combined with the native snapshot capabilities that are included in many of AWS’s other services.

Through the AWS Backup console, you can manage backups from across your services. This includes determining which storage services you store backups in, who has access to backup data, and how long you retain backups for.

Azure Backup

Azure Backup is a solution you can use to backup Azure service or on-premises data. It enables you to automate and manage backups and their life cycles. This service also integrates with Recovery Services vaults, storage resources designed specifically for backup data. 

Azure Backup and Recovery Services are part of a collection of services Azure offers for backup creation and management. The other main service is Azure Site Recovery. This service enables you to create and remotely store backups of your data and services which you can then use for disaster recovery or as failover services. 

Google Cloud Backup

Unlike AWS or Azure, Google Cloud does not offer a specific service for backup creation or management. Instead, it enables you to store backups in lower tier (i.e., cheaper) storage services. Your primary storage options include:

• Nearline Storage—designed for data that is accessed once a month or less. This option is best suited to your most recent backup or partial backups.
• Coldline Storage—designed for data that is accessed once a year or less. This option is best suited to disaster recovery backups or archived backup data. 

These are reasonable options in Google Cloud but are not as functional with other cloud providers. The difference is that Google’s services provide access with sub-millisecond latency while cold and archive storage in AWS or Azure can take several hours or days to retrieve. 

AWS Cloud Database Services

AWS database services include:

• Amazon RDS—a relational database service that supports your choice of six engines, including SQL Server, MariaDB, Oracle, PostgreSQL, MySQL, and Amazon Aurora. This is a fully managed service that you can manage through CLI, API, or the Console. 
• Amazon Aurora—a proprietary relational database that offers PostgreSQL and MySQL compatibility modes. This database is designed for high performance and integration with AWS services. 
• Amazon DynamoDB—a document and key-value database designed for low latency, high performance, and durability. This is a fully managed service for production level workloads. 
• Amazon ElastiCache—an in-memory data store that you can use in place of a traditional database. It is compatible with Redis and Memcached and provides scalability, high performance, and super low latency. 
• Amazon Neptune—a fully managed graph database that is optimized for high speed querying. It supports RDF and Property Graph data models and SPARQL and Gremlin languages. 
• Amazon Timestream—a fully managed time series database designed for analytics, DevOps, and IoT workloads. It enables you to stream data and perform time sensitive queries. 
• Amazon Quantum Ledger Database—a fully managed ledger database that you can use to verify transactions cryptographically. Stored data is transparent and immutable, making it ideal for auditing and financial transactions. 

Azure Cloud Database Services

Azure database services include:

• Azure Cosmos DB—a fully managed, multi model database. It enables you to define schema and indexes from your workloads and application for maximum flexibility. CosmosDB includes API support for Table, SQL, MongoDB, Gremlin, Cassandra, Spark, and ectd. 
• Azure SQL Database—a fully managed database based on the SQL Server engine. It is highly available, scalable, and includes serverless options. You also have the option of bringing existing SQL Server licenses from on-premises. 
• Azure Database for MySQL—a fully managed database service based on the MySQL Community edition. You can integrate it with Azure Kubernetes Service and Azure App Service. 
• Azure Database for PostgreSQL—a fully managed database for PostgreSQL Hyperscale. You can use this service in the cloud or on-premises. It is extensible through plugins, including for Azure Data Studio, Timescale DB, Visual Studio Code, and PostGIS.
• SQL Server on Virtual Machines—a service that enables you to host SQL server on VMs with hybrid connectivity. You can use this database with Windows or Linux and use it to extend support for SQL Server 2008. 
• Azure Synapse Analytics—an analytics service that combines big data analytics with data warehousing. It is integrated with SQL and Apache Spark engines and can be integrated with CosmosDB. 
• Azure Data Explorer—a data analytics service that you can use to perform real-time analytics on streaming data. You can use it to perform time series analyses and query big data. 
• Azure Cache for Redis—a fully managed, in-memory data store that supports Redis workloads. It includes features for built-in security, scalability, and reliability. 
• Azure Database for MariaDB—a fully managed database based on the community version of MariaDB. It supports a variety of open source frameworks and includes features for high availability, scalability, and built-in security. 

Google Cloud Database Services

Google Cloud database services include:

• Cloud SQL—a fully managed database that provides support for SQL Server, PostgreSQL, and MySQL workloads. It includes features for automated backups, data replication, and failover. 
• Cloud Spanner—a fully managed relational database that is ACID compliant, globally distributed and supports automatic sharding. It also offers multi-regional availability and transparent synchronous replication. 
• BigQuery—a scalable, serverless, multi-cloud data warehouse. You can use it to perform predictive and real-time analytics with built-in machine learning. 
• Cloud Bigtable—a fully managed NoSQL database designed for operational and analytical workloads. It is cluster based and can scale to hundreds of nodes with built-in replication and high availability.
• Cloud Firestore—a fully managed document database that you can use to store, sync, and query application data. It includes client libraries for offline support and live synchronization and integrates with Firebase. 
• Firebase—a NoSQL database that provides real time data synchronization. You can use it to collaborate globally, support serverless applications, and provide offline support.
• Cloud Memorystore—a fully managed in-memory data store that supports Memcached and Redis. You can use it to migrate caching layers and create application caches. It includes features for automatic failover, monitoring, patching, and high availability.