More about Kubernetes Storage
- Azure Kubernetes Service: Configuring Persistent Volumes in AKS
- Kubernetes NFS: Two Quick Tutorials
- AWS Kubernetes Cluster: Quick Setup with EC2 and EKS
- Kubernetes Shared Storage: The Basics and a Quick Tutorial
- AWS ECS vs Kubernetes: An Unfair Comparison?
- Kubernetes Persistent Storage: Why, Where and How
- Kubernetes for Developers: A Deep Dive
- How to Provision Kubernetes Persistent Volumes for NFS Services
- Using Cloud Manager for Kubernetes Deployment
- Docker Volume Tutorial - Using Trident to Provision Storage
- How to Set Up MySQL Kubernetes Deployments
- Kubernetes Persistent Volumes Cloning
- Protection for Persistent Data Storage in Kubernetes
- Dynamic Kubernetes Persistent Volume Provisioning
- Managing Stateful Applications in Kubernetes
- Understanding Kubernetes Persistent Volume Provisioning
- An Introduction to Kubernetes
In Kubernetes, persistent volumes are used to provision storage for containerized applications and provide a separation between how storage is used and how it is allocated.
That sounds great, however, there’s a catch. Many times, large allocations of storage made for an application are never used, and in other cases the data stored within a persistent volume is never compressed, leading to extra storage space being used up unnecessarily. These types of issues can be resolved using the storage efficiency technologies that are part of Cloud Volumes ONTAP, which the NetApp Trident provisioner makes available to Kubernetes.
In this article, we examine the way in which NetApp Trident and Cloud Volumes ONTAP deliver enterprise-grade data management features that considerably improve the cost effectiveness of cloud storage.
Persistent Volumes in Kubernetes
Applications require storage for myriad purposes, from database systems storing transactional business information to caching services holding frequently accessed data and more. In a Kubernetes cluster, persistent volumes enable pods to make use of storage without having to know the exact details of how that storage is actually allocated. Each pod specification can define a list of persistent volume claims that are used by the cluster to bind the pod to suitable persistent volumes.
Persistent volumes can be either created manually ahead of time, a process known as static provisioning, or automatically created in response to incoming persistent volume claims, which is known as dynamic provisioning. In both cases, it can be a challenge to ensure storage efficiency, as application developers must prepare for storage usage peaks and can often overestimate the amount of storage they need. In a cloud environment, this leads to an unnecessary increase in cloud storage costs.
There may also be situations where a large part of the production dataset is cold, or infrequently accessed, but which cannot be moved to more cost-effective storage without introducing a lot of complexity to the ways application services rely on that data. These application services require a uniform view of their data, with fast access to hot data and on-demand access to cold data. This can again be a difficult problem to solve, resulting in large allocations of high performance and costly storage.
Another scenario where transparently-applied storage efficiency would hold great value is where the data being stored contains a high degree of redundancy, which could be compressed in order to reduce the overall amount of storage required. Moving the onus for compression to the storage layer simplifies application software development and also helps to reduce the cloud storage footprint of legacy and third-party application services, where changes to the systems cannot be made.
Though storage efficiency is always important in the cloud, it can be even more so in Kubernetes environments due to the inherent scalability of containers. Spinning up new pods to deal with an increase in workload, or to provide greater redundancy and availability, also requires allocating new persistent volumes. The storage overhead for these persistent volumes can be brought under control through efficient data storage.
Reducing Kubernetes Cloud Storage Cost with NetApp Trident and Cloud Volumes ONTAP
Kubernetes benefits from Trident by being able to directly take advantage of NetApp’s enterprise-grade data storage management platform. By way of dynamic provisioning, Trident will automatically create storage volumes in Cloud Volumes ONTAP using AWS storage or Azure storage, or any on-premises ONTAP system. Besides the added benefits of FlexClone instant, writable clones and enhanced data protection for Kubernetes deployments, one of the primary reasons for using Trident is that it opens up the storage efficiency features of Cloud Volumes ONTAP, providing Kubernetes advantages in terms of transparently reducing the storage space required for persistent volumes.
Thin provisioning makes it possible to create persistent volumes that appear to pods as having the size they requested through a persistent volume claim, but without needing to allocate all of that storage in advance. Cloud Volumes ONTAP will automatically add storage capacity to the persistent volume as and when it is required, and also return back freed up storage to the common pool when data is deleted. This ensures that storage space is only allocated when it’s actually needed, which reduce storage usage costs and drives up storage space utilization. Thin provisioning also makes it much easier to plan for the future storage requirements of the cluster.
Cloud Volumes ONTAP is able to transparently apply transformations, which help to reduce storage space usage, to the data it stores. Data deduplication collapses identical copies of a block into a single block, with reference pointers inserted into every place the block is used. This can dramatically reduce storage space requirements, with some customers reporting savings of up to 70%. A small amount of storage space is consumed in order to maintain the metadata required to support the block mappings.
Data in an ONTAP storage volume can be transparently compressed without requiring any changes to client applications and services. This compression is applied to groups of consecutive blocks, as opposed to entire files, which makes reading and updating highly optimal. Data compression can also be used in conjunction with data deduplication.
NetApp is constantly innovating and extending the capabilities of the ONTAP platform, and data compaction is one of the more recent features to be introduced. After applying inline data deduplication and compression, multiple blocks that are not completely filled are combined together, removing the unused of space that would have otherwise been left in each block.
A major advantage of using Cloud Volumes ONTAP is the ability to automatically balance data between a capacity storage tier for colder data and a performance tier for fast access. Depending on the cloud vendor being used, Cloud Volumes ONTAP will use Amazon S3 or Azure Blob for the capacity tier, which provides significant cost savings for large amounts of data that are infrequently accessed, but that must still be available on-demand. When the data is required, it is transparently moved to the performance tier, and will age back out to the capacity tier when it is no longer in active use.
Cloud Volumes ONTAP customers such as Concerto Cloud Services and Reach PLC (formerly Trinity Mirror) are two examples of leveraging storage efficiencies to drastically lower TCO. Using the features covered above, Concerto managed a 96% reduction of storage footprint and Reach achieved a 50% storage reduction.
Leveraging the flexibility of persistent volumes and dynamic provisioning, NetApp Trident acts as a gateway for Kubernetes clusters to take advantage of the mature and ever-evolving data storage management feature set of Cloud Volumes ONTAP. One of the most compelling arguments for choosing this solution is the wealth of storage efficiency technologies that are built into NetApp’s ONTAP storage services, which can be used transparently, and in combination with each other, to greatly reduce operational cloud storage costs. And remember, there’s a lot more to learn about Kubernetes in the rest of this series.
Find out more about using Cloud Volumes ONTAP transform container storage management.