More about Google Cloud Storage
- gsutil: Command-Line Control of Google Cloud Storage
- Provisioned IOPS for Google Cloud Persistent Disk
- Google Cloud Storage Pricing: Get the Best Bang for Your Buckets
- Google Cloud Website Hosting on Google Cloud Storage
- Google Cloud Persistent Disk: How to Create a Google Cloud Virtual Image
- Google Cloud Storage Encryption with SSE
- Google Cloud Persistent Disk: How to Resize and Use
- Google Cloud Storage Bucket Lifecycle Rules & How to Change Them
- Google Storage Service How-To: Switch Google Cloud Storage Class
- Google Cloud Filestore: NFS Cloud File Storage on Google Cloud
Alongside compute and network, storage is one of the fundamental resources required in today’s technological systems and software development. Regardless if your systems are on-premises, in the cloud, or both, you always rely on storage components to persist your data. However, if you are used to the world of data centers—with SAN, NAS, and local hard drives—and are starting to venture into public cloud, the concept of cloud-based storage and the different services available for it might prove tricky.
In this article, we are going to provide some clarity about the different storage options in the Google Cloud Platform.
Google Cloud provides three main services for different types of storage: Persistent Disks for block storage, Filestore for network file storage, and Cloud Storage for object storage. These services are at the core of the platform and act as building blocks for the majority of the Google Cloud services and, by extension, to the systems you build on top of it.
Since NetApp Cloud Volumes ONTAP is now available for use in the Google Cloud, NetApp users can easily take advantage of this growing cloud platform. Let’s take a closer look at each of these Google Cloud storage services, what they were designed for, and what use cases they are each best suited to handle.
Google Cloud Persistent Disks (Block Storage)
Block storage is the traditional storage type, both in the cloud and in on-premises systems. A Google Cloud Persistent Disk provides block storage and it is used by all virtual machines in Google Cloud (Google Cloud Compute Engine). The easiest way to understand it is by imagining those Persistent Disks as mere USB drives. They can be attached or detached from virtual machines and enable you to build, as the name suggests, data persistence for your services whenever virtual machines are started, stopped, or terminated.
In addition to Google Cloud Compute Engine virtual machines, these Persistent Disks are also used to power the Google Kubernetes Engine service.
Very much like a virtual disk in your local machine, a Google Cloud Persistent Disk can either be HDD or SSD, the latter for high I/O performance. In addition, there is also the ability to choose where they are located and what type of availability is needed: they can be Regional, Zonal, or Local. While local disks are not, in theory, part of the Google Cloud Persistent Disk service, it is important to mention them. These local disks are only available in the hardware where the virtual machine is running and, while providing the best I/O performance, they are not often recommended due to the low availability and redundancy. On the other hand, if you require high availability, Regional disks will offer you that out-of-the-box, with your disks being replicated behind the scenes in different zones within a region. A more moderate (and less-expensive) approach are Zonal disks, which are storage disks that are also highly available, but only within a single zone.
Other lesser known, yet great features of Google Cloud Persistent Disks are automatic encryption, flexibility to resize while-in-use, and a snapshot capability which can be used for both backup and virtual machine image creation.
Google Cloud Filestore (Network File Storage)
Filestore is the fully managed Google Cloud service that provides network file storage. Network file storage is not a new cloud concept and very much like block storage it also exists in your typical on-premises data center. If you’re used to working with NAS (Network Attached Storage) the concept should be familiar to you.
While you could of course argue that network file storage is technically block storage (it is!), there is a very clear distinction here. A network file storage, as the name suggests, provides a disk storage over the network. This enables the development of systems with multiple parallel services that can read and write files from the same disk storage mounted over the network.
However, the advantages this gives require some caution in its use. Compared with the usual block or object storage, the performance of file storage is, as you might expect, substantially inferior. This might lead to issues with concurrency and file permissions. Therefore, while designing cloud-native systems, you should only use this solution after careful evaluation to solve these challenges.
Google Cloud Storage (Object Storage)
What is Google Cloud Storage?
Google Cloud Storage is the object storage service offered by Google Cloud. It provides some very interesting out-of-the-box features such as object versioning or fine-grain permissions (per object or bucket), that can make development easy and help reduce operational overheads. Google Cloud Storage serves as the foundation of several different services.
What kind of benefits does this storage type have? The concept of object storage is not that easy to grasp. In typical on-premises systems where capacity is more limited and connectivity fast and exclusive, having this type of storage is not at all common. The way that object storage works, however, is beautifully simple to the end user. In simple terms, its value-proposition is that you can get and put any file you want via a REST API—and this can expand indefinitely with each object growing up to the terabyte scale. Interesting, right? In Cloud Storage, different objects are grouped in unique “namespaces” called buckets. A bucket can hold multiple objects yet, a single object will belong to only one bucket.
This model for storage is widely popular in cloud-native systems due to its low cost (cents per GB) combined with the serverless approach and simplicity. The heavy work of data replication, availability, integrity, capacity planning, etc. is then left to the cloud provider. The drawback of object storage is that there is no other way to access the data besides the REST API; therefore, the typical approach for designing systems, managing data, and structuring a filesystem-type of access doesn’t work.
Google Cloud Storage Classes, Archival Storage, and Lifecycle Management Rules
Perhaps some of the most underrated functionalities of Google Cloud Storage are the different storage classes and putting Lifecycle Management Rules into use for the data buckets. Using these features can make a huge difference in terms of cost and the running operational expenses.
In Google Cloud Storage, you are required to select one storage class for your bucket: Standard (which can be either Regional or Multi-Regional), Nearline, or Coldline. The usual approach is to select Standard, where you can opt to have your bucket in a specific single Google Cloud Region or stored across multiple Regions. This works really well in different scenarios since you get a highly performant and highly available storage.
However, there are several cases where the data is not meant to be accessed frequently and having reduced availability is perfectly ok. In these cases, the Nearline and Coldline storage classes are options that can and should be explored. They can easily reduce the cost in >50% compared with the standard storage class.
The Nearline storage class is designed for data that is accessed less than once per month. One example use case is data that will only be used to produce an aggregate monthly report. Coldline storage, on the other hand, is designed for data that is accessed even less frequently—think, once per year or less. This storage class is therefore particularly useful for archival storage. One use case is to use the Coldline storage class to keep a copy of data that the business requires to be retained for a long period of time (e.g. 10 years) due to compliance with different regulatory requirements.
Lifecycle Management Rules
One of the challenges of properly leveraging storage classes (and other Cloud Storage features) is that the same type of data (i.e., data in the same bucket) might require different treatment over the course of its lifetime. For example, if you use Google Cloud Storage buckets to store your application logs, you might require high availability for the data during the first month (including versioning each object as a safeguard for data tampering), and perhaps less availability (without versioning) for the next six months, and eventually retain a copy of those logs for the following five plus years due to compliance obligations.
For these types of scenarios, you can enable Google Cloud Storage Lifecycle Management rules. This is a really great built-in feature of Google Cloud Storage that enables you to define business logic rules per bucket without much effort. With these rules you can define actions such as automatically transitioning objects between different storage classes, disable versioning, or even delete objects after certain defined periods of time. Leveraging these features and having lifecycle rules in place can translate into real and tangible cost savings at the end of the month.
In this article, we explored the different storage building blocks in the cloud and provided a good overview of the different storage services available in Google Cloud. They are quite straightforward to grasp and provide a huge amount of flexibility to develop different types of systems in the Cloud.
While designing and architecting your system is then important to understand the pros and cons of each storage type and the costs associated with them. What will likely appeal to current NetApp users trying to find out how to use Google Cloud storage is that Cloud Volumes ONTAP is now available for use with Google Cloud. NetApp’s groundbreaking cloud data management platform that has been a huge success for enterprises in AWS and Azure, can finally start to leverage the benefits of Google Cloud.
Learn More About Google Cloud Storage
Cloud File Sharing Services: Google Cloud Filestore
When it comes to using a cloud file service, cloud admins have a lot of challenges on their plate. These include ensuring data is accessible to all clients no matter the type of operating systems they use, managing backup and archive data, and making sure costs don’t go through the roof even when storing large volumes of data. Google Filestore is the newest file service on the market. How does this new offering stack up to some of the more mature options out there?
This post takes a deep dive into Cloud Filestore, Google Cloud’s fully managed file share service, and examines whether it has all the required features you’ll need in a cloud file share service, such as multi-zone availability, point-in-time snapshots and data protection. The article also takes a look at NetApp’s Cloud Volumes ONTAP, now available on Google Cloud file storage.
How to Add and Manage Lifecycle Rules in Google Cloud Storage Buckets
Managing your data lifecycle can be very challenging when dealing with unpredictable amounts of data. Luckily, Google Cloud Storage comes with a built-in Object Lifecycle Management feature which makes things much easier. Although the pricing for this feature is low, if improperly configured, expenses can rise significantly when using big data volumes or multiple API operations.
This post introduces Google Cloud’s lifecycle management feature and shows you how to create automated lifecycle rules, taking full advantage of Google Cloud’s out-of-the-box capabilities. The article also gives you a step-by-step walkthrough on how to create a Google Cloud Storage bucket and set up lifecycle rules to automatically delete data as well as change your storage class.
Google Cloud Website Hosting with Google Cloud Storage
There are many benefits to hosting a static website on Google Cloud Storage that often go overlooked. Hosting a website on Google Cloud is actually incredibly simple and offers multiple advanced capabilities and services, for those who know how to take advantage of them.
In this post we explain from A to Z how to host your website on Google Cloud Storage. This will include instructions on hosting the website files in Google Storage buckets, enabling and configuring the Google Cloud Storage static website as well as validation in the browser and advanced settings.
Cloud Storage Encryption: Key Management in Google Cloud
Client-side encryption, in which clients locally encrypt data with their own keys, provides users with more control over encryption but also necessitates careful key management, auditing, and frequent rotations. One way to bypass these challenges is to use one of Google Cloud’s SSE options. Each option has its unique set of advantages and disadvantages.
This post introduces the different SSE options - GMEK, CSEK and CMEK, discussing which is the easiest to use, which is designed for sensitive data, and other considerations such as auditing, rotation and storage. It then provides a step-by-step guide on how to use CSEK on Google Cloud Platform storage.
How To Resize Google Cloud Persistent Disk Attached to a Linux Instance
Google Cloud Persistent Disks function as reliable and high-performance block storage for your Google Compute Virtual Machine instances. Getting to know their functionality, as well as learning to perform online resizing, will provide you with a greater degree of flexibility over your deployment and ensure business continuity.
In this post we introduce Google Cloud Persistent Disks, take a look at some of its built-in features and explain how to resize them. We’ll also show you how NetApp Cloud Volumes ONTAP can offer you even more capabilities for maximum flexibility.
How to Switch Between Classes in Google Storage Service
Google Cloud Storage comes with several distinct storage classes and availability types, each tailored to fit your data’s usage patterns. Leveraging these options properly can have a huge impact on cost and performance.
This post gives an overview of Google Storage Service, focusing on the various storage classes and their respective availability types. It also provides a step by step guide to create a Google Cloud Storage bucket, add and remove data, and switching between storage classes.
How to Use the gsutil Command-Line Tool for Google Cloud Storage
Object storage is very easy to use, operates in a pay-as-you-go pricing model, and has low operational overhead, all of which make it the popular choice in cloud computing. To get started with this type of storage, it’s important to get to know your options for interacting with Google Cloud Storage. This will also be helpful for anyone setting up NetApp’s Cloud Volumes ONTAP for Google Cloud.
This post explains different ways to communicate with Google Cloud Storage: Google Cloud Console, SDK’s and the Command-line tool. The article explains, step by step, how to use the gsutil command-line to install and authenticate Google Cloud using your machine terminal, how to create or destroy a bucket, and how to add and remove objects from that bucket. The gsutil tool is especially handy when transferring large volumes of data and for automating tasks without recurring to the UI.
Google Cloud Persistent Disk: How to Create a Google Cloud Virtual Image
Knowing how to create a custom virtual image is a significant step in making the most of what the public cloud has to offer. If you are using Google Cloud Compute Engine Instances, this feature is a definite must-have to leverage when designing your system.
This post provides an explanation of virtual images and details how to create them from Google Cloud Persistent Disks. This will accelerate the bootstrap process in applications running in Google Cloud Engine and can significantly augment your deployment of Cloud Volumes ONTAP on Google Cloud.
Google Cloud Persistent Disks: Using Multiple Persistent Disks with the Same Cloud Compute Instance
This introduction to Google Cloud Compute Engine and Persistent Disks helps you get acquainted with the primary Google Cloud storage and compute options, and shows you step by step how to attach Persistent Disks to instances.
The how-to starts by referring to the instance creation process for creating one or more Google Cloud Persistent Disks, then details how to attach those storage disks to the cloud instance, and how to put it all in use with a new disk file system.
How to Use GCP Persistent Storage with Provisioned IOPS
Storage performance is a key part of a cloud storage deployment. Any failure to perform at acceptable levels will seriously impact your system’s operation. On Google Cloud, the performant block storage option is Persistent Disk. Maintaining a high level of IOPS is key to this, but that means finding the right mix of disk size, disk type, and instance to use, all while making sure that costs stay under control.
In this post we show you how to provision performant disk storage with provisioned IOPS on Google Cloud so that you can see how this fine tuning process works for yourself.
Google Cloud Storage Pricing: Get the Best Bang for Your Buckets
Google Cloud Storage offers four storage classes with progressively lower storage costs, but higher retrieval costs: Standard (frequently used data), Coldline Storage (rarely used data), Nearline Storage (infrequently used data), and Archive Storage (long term archival).
Storage pricing is based on the data storage you consume, network usage and operations you perform on Google Cloud.
This post provides best practices for reducing your Google Cloud Storage costs.