More about Google Cloud Storage
- Understanding Google Cloud Storage Costs
- How to Use Google Filestore with Microservices
- Google Cloud & Microservices: How to Use Google Cloud Storage with Microservices
- Google Cloud Storage Pricing: Get the Best Bang for Your Buckets
- High Availability Architecture on GCP with Cloud Volumes ONTAP
- GCP Persistent Disk Deep Dive: Tips, and Tricks
- How to Use the gsutil Command-Line Tool for Google Cloud Storage
- Storage Options in Google Cloud: Block, Network File, and Object Storage
- Provisioned IOPS for Google Cloud Persistent Disk
- How to Use Multiple Persistent Disks with the Same Google Cloud Compute Engine Instance
- Google Cloud Website Hosting with Google Cloud Storage
- Google Cloud Persistent Disk: How to Create a Google Cloud Virtual Image
- Google Cloud Storage Encryption: Key Management in Google Cloud
- How To Resize a Google Cloud Persistent Disk Attached to a Linux Instance
- How to Add and Manage Lifecycle Rules in Google Cloud Storage Buckets
- How to Switch Between Classes in Google Storage Service
- Cloud File Sharing Services: Google Cloud File Storage
Alongside compute and network, storage is one of the fundamental resources required in today’s technological systems and software development. Regardless if your systems are on-premises, in the cloud, or both, you always rely on storage components to persist your data. However, if you are used to the world of data centers, like Google Cloud Storage —with SAN, NAS, and local hard drives—and are starting to venture into public cloud, the concept of cloud-based storage and the different services available for it might prove tricky.
In this article, we are going to provide some clarity about the different storage options in the Google Cloud Platform.
Google Cloud provides three main services for different types of storage: Persistent Disks for block storage, Filestore for network file storage, and Cloud Storage for object storage. These services are at the core of the platform and act as building blocks for the majority of the Google Cloud services and, by extension, to the systems you build on top of it.
Since NetApp Cloud Volumes ONTAP is now available in the Google Cloud, NetApp users can easily take advantage of this growing cloud platform. Let’s take a closer look at each of these Google Cloud storage services, what they were designed for, and what use cases they are each best suited to handle.
Google Cloud Persistent Disks (Block Storage)
Block storage is the traditional storage type, both in the cloud and in on-premises systems. A Google Cloud Persistent Disk provides block storage and it is used by all virtual machines in Google Cloud (Google Cloud Compute Engine). The easiest way to understand it is by imagining those Persistent Disks as mere USB drives. They can be attached or detached from virtual machines and enable you to build, as the name suggests, data persistence for your services whenever virtual machines are started, stopped, or terminated.
In addition to Google Cloud Compute Engine virtual machines, these Persistent Disks are also used to power the Google Kubernetes Engine service.
Very much like a virtual disk in your local machine, a Google Cloud Persistent Disk can either be HDD or SSD, the latter for high I/O performance. In addition, there is also the ability to choose where they are located and what type of availability is needed: they can be Regional, Zonal, or Local. While local disks are not, in theory, part of the Google Cloud Persistent Disk service, it is important to mention them. These local disks are only available in the hardware where the virtual machine is running and, while providing the best I/O performance, they are not often recommended due to the low availability and redundancy. On the other hand, if you require high availability, Regional disks will offer you that out-of-the-box, with your disks being replicated behind the scenes in different zones within a region. A more moderate (and less-expensive) approach are Zonal disks, which are storage disks that are also highly available, but only within a single zone.
Other lesser known, yet great features of Google Cloud Persistent Disks are automatic encryption, flexibility to resize while-in-use, and a snapshot capability which can be used for both backup and virtual machine image creation.
Google Cloud Filestore (Network File Storage)
Filestore is the fully managed Google Cloud service that provides network file storage. Network file storage is not a new cloud concept and very much like block storage it also exists in your typical on-premises data center. If you’re used to working with NAS (Network Attached Storage) the concept should be familiar to you.
While you could argue that network file storage is technically block storage (it is!), there is a very clear distinction here. A network file storage, as the name suggests, provides a disk storage over the network. This enables the development of systems with multiple parallel services that has the ability to read and write files from the same disk storage mounted over the network.
However, the advantages this gives require some caution in its use. Compared with the usual block or object storage, the performance of file storage is, as you might expect, substantially inferior. This might lead to issues with concurrency and file permissions. Therefore, while designing cloud-native systems, you should only use this solution after careful evaluation to solve these challenges.
Google Cloud Storage (Object Storage)
What is Google Cloud Storage?
Google Cloud Storage is the object storage service offered by Google Cloud. It provides some very interesting out-of-the-box features such as object versioning or fine-grain permissions (per object or bucket), that can make development easy and help reduce operational overheads. Google Cloud Storage serves as the foundation of several different services.
What kind of benefits does this storage type have? The concept of object storage is not that easy to grasp. In typical on-premises systems where capacity is more limited and connectivity fast and exclusive, having this type of storage is not at all common. The way that object storage works, however, is beautifully simple to the end user. In simple terms, its value-proposition is such that you can get and put any file you want via a REST API—and this can expand indefinitely with each object growing up to the terabyte scale. Interesting, right? In Cloud Storage, different objects are grouped in unique “namespaces” called buckets. A bucket can hold multiple objects yet, a single object will belong to only one bucket.
This model for storage is widely popular in cloud-native systems due to its low cost (cents per GB) combined with the serverless approach and simplicity. The heavy work of data replication, availability, integrity, capacity planning, etc. is then left to the cloud provider. The drawback of object storage is that there is no other way to access the data besides the REST API; therefore, the typical approach for designing systems, managing data, and structuring a filesystem-type of access doesn’t work.
Google Cloud Storage Classes, Archival Storage, and Lifecycle Management Rules
Perhaps some of the most underrated functionalities of Google Cloud Storage are the different storage classes and putting Lifecycle Management Rules into use for the data buckets. Using these features can make a huge difference in terms of cost and the running operational expenses.
In Google Cloud Storage, you are required to select one storage class for your buckets: Standard (which can be either Regional or Multi-Regional), Nearline, or Coldline. The usual approach is to select Standard, where you can opt to have your bucket in a specific single Google Cloud Region or stored across multiple Regions. This works really well in different scenarios since you get a highly performant and highly available storage.
However, there are several cases where the data is not meant to be accessed frequently and having reduced availability is perfectly ok. In these cases, the Nearline and Coldline storage classes are options that can and should be explored. They can easily reduce the cost in >50% compared with the standard storage class.
The Nearline storage class is designed for data that is accessed less than once per month. One example use case is data that will only be used to produce an aggregate monthly report. Coldline storage, on the other hand, is designed for data that is accessed even less frequently—think, once per year or less. This storage class is therefore particularly useful for archival storage. One use case is to use the Coldline storage class to keep a copy of data that the business requires to be retained for a long period of time (e.g. 10 years) due to compliance with different regulatory requirements.
Lifecycle Management Rules
One of the challenges of properly leveraging storage classes (and other Cloud Storage features) is that the same type of data (i.e., data in the same bucket) might require different treatment over the course of its lifetime. For example, if you use Google Cloud Storage buckets to store your application logs, you might require high availability for the data during the first month (including versioning each object as a safeguard for data tampering), and perhaps less availability (without versioning) for the next six months, and eventually retain a copy of those logs for the following five plus years due to compliance obligations.
For these types of scenarios, you can enable Google Cloud Storage Lifecycle Management rules. This is a really great built-in feature of Google Cloud Storage that enables you to define business logic rules per bucket without much effort. With these rules you can define actions such as automatically transitioning objects between different storage classes, disable versioning, or even delete objects after certain defined periods of time. Leveraging these features and having lifecycle rules in place can translate into real and tangible cost savings at the end of the month.
In this article, we explored the different storage building blocks in the cloud and provided a good overview of the different storage services available in Google Cloud. They are quite straightforward to grasp and provide a huge amount of flexibility to develop different types of systems in the Cloud.
While designing and architecting your system is then important to understand the pros and cons of each storage type and the costs associated with them. What will likely appeal to current NetApp users trying to find out how to use Google Cloud storage is that Cloud Volumes ONTAP is now available for use with Google Cloud. NetApp’s groundbreaking cloud data management platform that has been a huge success for enterprises in AWS and Azure, can finally start to leverage the benefits of Google Cloud.
Learn More About Google Cloud Storage
Google Cloud & Microservices: How to Use Google Cloud Storage with Microservices
With microservices, organizations are able to break out of the monolithic IT architectures, allowing for smaller, more flexible service options. When using microservices on Google Cloud, there is an important role that Google Cloud Storage can play.
This blog will go into more detail about how microservices work and provide a working step-by-step guide on how to implement one with a same architecture using Google Cloud Storage.
Read more in Google Cloud & Microservices: How to Use Google Cloud Storage with Microservices.
Cloud File Sharing Services: Google Cloud Filestore
When it comes to using a cloud file service, cloud admins have a lot of challenges on their plate. These include ensuring data is accessible to all clients no matter the type of operating systems they use, managing backup and archive data, and making sure costs don’t go through the roof even when storing large volumes of data. Google Filestore is the newest file service on the market. How does this new offering stack up to some of the more mature options out there?
This post takes a deep dive into Cloud Filestore, Google Cloud’s fully managed file share service, and examines whether it has all the required features you’ll need in a cloud file share service, such as multi-zone availability, point-in-time snapshots and data protection. The article also takes a look at NetApp’s Cloud Volumes ONTAP, now available on Google Cloud file storage.
How to Add and Manage Lifecycle Rules in Google Cloud Storage Buckets
Managing your data lifecycle can be very challenging when dealing with unpredictable amounts of data. Luckily, Google Cloud Storage comes with a built-in Object Lifecycle Management feature which makes things much easier. Although the pricing for this feature is low, if improperly configured, expenses can rise significantly when using big data volumes or multiple API operations.
This post introduces Google Cloud’s lifecycle management feature and shows you how to create automated lifecycle rules, taking full advantage of Google Cloud’s out-of-the-box capabilities. The article also gives you a step-by-step walkthrough on how to create a Google Cloud Storage bucket and set up lifecycle rules to automatically delete data as well as change your storage class.
Google Cloud Website Hosting with Google Cloud Storage
There are many benefits to hosting a static website on Google Cloud Storage that often go overlooked. Hosting a website on Google Cloud is actually incredibly simple and offers multiple advanced capabilities and services, for those who know how to take advantage of them.
In this post we explain from A to Z how to host your website on Google Cloud Storage. This will include instructions on hosting the website files in Google Storage buckets, enabling and configuring the Google Cloud Storage static website as well as validation in the browser and advanced settings.
Cloud Storage Encryption: Key Management in Google Cloud
Client-side encryption, in which clients locally encrypt data with their own keys, provides users with more control over encryption but also necessitates careful key management, auditing, and frequent rotations. One way to bypass these challenges is to use one of Google Cloud’s SSE options. Each option has its unique set of advantages and disadvantages.
This post introduces the different SSE options - GMEK, CSEK and CMEK, discussing which is the easiest to use, which is designed for sensitive data, and other considerations such as auditing, rotation and storage. It then provides a step-by-step guide on how to use CSEK on Google Cloud Platform storage.
How To Resize Google Cloud Persistent Disk Attached to a Linux Instance
Google Cloud Persistent Disks function as reliable and high-performance block storage for your Google Compute Virtual Machine instances. Getting to know their functionality, as well as learning to perform online resizing, will provide you with a greater degree of flexibility over your deployment and ensure business continuity.
In this post we introduce Google Cloud Persistent Disks, take a look at some of its built-in features and explain how to resize them. We’ll also show you how NetApp Cloud Volumes ONTAP can offer you even more capabilities for maximum flexibility.
How to Switch Between Classes in Google Storage Service
Google Cloud Storage comes with several distinct storage classes and availability types, each tailored to fit your data’s usage patterns. Leveraging these options properly can have a huge impact on cost and performance.
This post gives an overview of Google Storage Service, focusing on the various storage classes and their respective availability types. It also provides a step by step guide to create a Google Cloud Storage bucket, add and remove data, and switching between storage classes.
How to Use the gsutil Command-Line Tool for Google Cloud Storage
Object storage is very easy to use, operates in a pay-as-you-go pricing model, and has low operational overhead, all of which make it the popular choice in cloud computing. To get started with this type of storage, it’s important to get to know your options for interacting with Google Cloud Storage. This will also be helpful for anyone setting up NetApp’s Cloud Volumes ONTAP for Google Cloud.
This post explains different ways to communicate with Google Cloud Storage: Google Cloud Console, SDK’s and the Command-line tool. The article explains, step by step, how to use the gsutil command-line to install and authenticate Google Cloud using your machine terminal, how to create or destroy a bucket, and how to add and remove objects from that bucket. The gsutil tool is especially handy when transferring large volumes of data and for automating tasks without recurring to the UI.
Google Cloud Persistent Disk: How to Create a Google Cloud Virtual Image
Knowing how to create a custom virtual image is a significant step in making the most of what the public cloud has to offer. If you are using Google Cloud Compute Engine Instances, this feature is a definite must-have to leverage when designing your system.
This post provides an explanation of virtual images and details how to create them from Google Cloud Persistent Disks. This will accelerate the bootstrap process in applications running in Google Cloud Engine and can significantly augment your deployment of Cloud Volumes ONTAP on Google Cloud.
Google Cloud Persistent Disks: Using Multiple Persistent Disks with the Same Cloud Compute Instance
This introduction to Google Cloud Compute Engine and Persistent Disks helps you get acquainted with the primary Google Cloud storage and compute options, and shows you step by step how to attach Persistent Disks to instances.
The how-to starts by referring to the instance creation process for creating one or more Google Cloud Persistent Disks, then details how to attach those storage disks to the cloud instance, and how to put it all in use with a new disk file system.
How to Use GCP Persistent Storage with Provisioned IOPS
Storage performance is a key part of a cloud storage deployment. Any failure to perform at acceptable levels will seriously impact your system’s operation. On Google Cloud, the performant block storage option is Persistent Disk. Maintaining a high level of IOPS is key to this but that means finding the right mix of disk size, disk type, and instance to use, all while making sure that costs stay under control.
In this post we show you how to provision performant disk storage with provisioned IOPS on Google Cloud so that you can see how this fine tuning process works for yourself.
Google Cloud Storage Pricing: Get the Best Bang for Your Buckets
Google Cloud Storage offers four storage classes with progressively lower storage costs, but higher retrieval costs: Standard (frequently used data), Coldline Storage (rarely used data), Nearline Storage (infrequently used data), and Archive Storage (long term archival).
Storage pricing is based on the data storage you consume, network usage and operations you perform on Google Cloud.
This post provides best practices for reducing your Google Cloud Storage costs.
GCP Persistent Disk Deep Dive: Tips, and Tricks
For the most demanding cloud workloads, block storage is key. On Google Cloud, block storage is offered through Google Cloud Persistent Disk. But many GCP Persistent Disk users may not be getting everything they can out of the service.
This blog takes a deep dive into the capabilities of GCP Persistent Disk and give you some inside tips on getting the most out of your block storage usage on Google Cloud, including how better performance can be obtained, the possibilities and advantages of using multiple disks in the same instance, growing and resizing disks in running instances, and leveraging snapshot and disk images.
Find out more in GCP Persistent Disk Deep Dive: Tips, and Tricks.
High Availability Architecture on GCP with Cloud Volumes ONTAP
Ensuring that your Google Cloud storage is protected from unexpected failures at the regional level is crucial to maintaining enterprise-level business continuity. However, the native Google Cloud storage high availability architecture options are not able to guarantee that data will not be lost in a failure scenario. This is what separates the high availability architecture used by Cloud Volumes ONTAP for Google Cloud.
The Cloud Volumes ONTAP HA configuration for Google Cloud storage provides out-of-the-box multi-region availability that guarantees data is never lost (RPO=0) and recovery time is immediate (RTO under 60 seconds). It’s a ready-to-go, and easily managed solution that has worked for years on AWS and Azure and now extends to Google Cloud.
How to Use Google Filestore with Microservices
Even stateless microservice-based workloads that run on Google Kubernetes Engine (GKE) can require persistent storage, and Google Filestore can provide it. This blog will dive into the details of attaching persistent storage to containers and provide a step-by-step guide on how to set up Google Filestore-based persistent volumes for use with GKE containers.
Read more in How to Use Google Filestore with Microservices
Understanding Google Cloud Storage Costs
Google Cloud Storage is designed to group objects into buckets. A bucket is a cloud-based container that can be assigned one of several storage tiers. Learn about Google Cloud Storage tiers, how they affect your costs, and other elements of storage costs including networking, data operations, and retrieval costs.
Read more: Understanding Google Cloud Storage Costs
Google Cloud Containers: Top 3 Options for Containers on GCP
Google provides several technologies you can use to run containers in Google Cloud environments. Learn about the top three services that can help you run containerized applications on the Google Cloud Platform - Google Kubernetes Engine (GKE), Google Anthos, and Google Cloud Run.
See Our Additional Guides on Key Cloud Storage Topics
We have authored in-depth guides on several other topics that can also be useful as you explore the world of cloud storage.
Multicloud strategies are becoming more popular as organizations seek to optimize their cloud services and deployments. These strategies can help you prevent vendor lock-in, increase your flexibility, and help you optimize costs.
This guide explains what multicloud storage is, how it works, what it’s used for, the core requirements for this storage, and how Cloud Volumes ONTAP supports it.
See top articles in our multicloud storage guide:
- One Cloud Out of Many: Why Enterprises Are Turning to Multicloud and Hybrid Cloud Architectures
- Multicloud Architecture: Partitioned, Cloud Burst and DR
- Multicloud Deployment: Creating a Plan With Cloud Volumes ONTAP
AWS offers a range of database services and support to try and meet all its clients needs. Many of these services are fully managed to help reduce your IT workload and enable you to store and use data as simply as possible.
This guide explains what AWS database support is available, what database services are available, and how you can migrate your databases to AWS.
See top articles in our AWS database services guide:
- AWS Database as a Service: DBaaS Types and Case Studies
- SQL Server in AWS: Managed Service vs Managed Storage
- AWS Oracle RDS: Running Your First Oracle Database on Amazon
Snapshots are a common method for natively backing up cloud data and services. This method enables you to save point in time backups which can be restored when needed.
This guide explains what types of storage snapshots are available, what AWS snapshots are, and how to use AWS snapshots.
See top articles in our AWS snapshots guide:
- Azure and AWS Snapshots Deep Dive: Cloud Volumes Snapshots
- Snapshots Deep Dive: AWS Snapshots and Azure Snapshots
- Understanding AWS Snapshot Pricing: Data Transfer and Storage Costs
Azure provides a wide variety of services to its users to help you manage your cloud data and services reliably. Azure Backup is one such service that can help provide data loss protection and peace of mind.
This guide explains what Azure Backup is and how to use it to backup your Azure data.
See top articles in our Azure Backup guide:
- Storage Options for Lower Azure Storage Costs and Azure Backup Costs
- The 5 Enterprise-Grade Azure Features You Need to Know About: Azure Backup, Security, and More
- Using Azure Backup Server to Backup Workloads and Files to Azure
Azure File Storage
Storing file data in Azure is simple through Azure File Storage service. This service enables you to store files across cloud and on-premises resources, enabling you to flexibly and securely share data and workflows.
This guide explains what Azure File Storage is, common use cases for Files, management concepts and components of the service, how data is accessed and the architecture of the service, and some best practices for securing your data.
See top articles in our Azure file storage guide:
Azure Files is one of several storage services available to users in Azure. It is a service designed to replicate file shares like those commonly used on premises. With this service, you can smoothly transition your files to the cloud and allow file sharing across your teams.
This guide explains what Azure Files is, how it complements other storage services, pricing and use cases for Files, and pros and cons you should be aware of.
See top articles in our Azure Files guide:
- Azure NetApp Files Register
- SMB File Sharing
- NFS and SMB - A Simple File Service Environment in Azure
Google Cloud offers a variety of storage options for you to choose from. These services form the base of many other services in the cloud and understanding what your options are can help you manage your cloud more efficiently.
This guide explains what Google Cloud Storage options exist and their common uses.
See top articles in our Google Cloud storage guide:
- Cloud File Sharing Services: Google Cloud Filestore
- Google Cloud Storage Encryption: Key Management in Google Cloud
- Google Cloud Storage Pricing: Get the Best Bang for Your Buckets
Google Cloud’s specialty is flexibility and integration of services and this extends to its database services. In Google Cloud you have a wide variety of database deployments, models, and support to choose from.
This guide explains your options for deploying databases in the cloud, what Google Cloud database services are available, and how to choose the right service for you.
See top articles in our Google Cloud database guide:
- Google Cloud SQL: MySQL, Postgres and MS SQL on Google Cloud
- SQL Server on Google Cloud: Managed Service Vs. Managed Storage
- Google Cloud SQL Pricing, Quotas, and Limits: A Cheatsheet for Cost Optimization
Software developers and DevOps engineers are packaging applications into lightweight units called containers. Kubernetes helps manage and scale containers across clusters of physical machines.
In this environment, Kubernetes storage becomes a significant challenge. By default, containers are ephemeral, meaning that any transient data on the container is lost when it shuts down. However, Kubernetes provides several options for persistent storage.
See top articles in our Kubernetes guide: