Maintaining the balance between performance and cost is one of the major challenges faced by organizations when it comes to enterprise-level storage systems. What we’re talking about is unused, stale data that consumes most of the storage on hand. This heavily impacts the return on investment for new storage systems, such as NetApp’s All-Flash FAS (AFF) storage arrays.
Cloud storage offers a cheaper alternative to store unused data. But that’s the cloud, and your data is on-prem. Manually categorizing and migrating the data is going to have overheads that will cast a shadow over any of the benefits offered by the cloud. NetApp has a better way.
NetApp AFF storage systems now come integrated with Cloud Tiering, a new service from NetApp that addresses the problem of shifting cold data from on-prem data centers into the cloud. Cloud tiering uses NetApp FabricPool technology to identify infrequently accessed data stored on NetApp AFF systems and tiers that data automatically to object storage in the cloud, including Amazon S3.
Amazon S3 uses a pay-as-you-go model and offers cheaper storage option for unused data, enabling organizations to leverage the all-flash, high-performance benefits of their AFF storage system for more critical live application data that needs it. When the data moved to Amazon S3 is accessed by any application, it is tiered back on premises, with virtually no impact on performance.
Configuring Cloud Tiering to Amazon S3
1. First off, we want to get situated on NetApp Cloud Central. Cloud Central is the entry point for all of the cloud-based services that NetApp offers, so this will be a helpful page to bookmark. You can get started by logging in with your Cloud Central account details. In case you don’t have an account, start by signing up:
2. Once you’re signed in, you’ll see a list of all the cloud services that are available on Cloud Central’s Fabric View. Locate the Cloud Tiering service and click on “Go to Cloud Tiering” to continue:
3. That click will redirect you to the Cloud Tiering configuration page. Click on “Let’s start, Discover your first cluster” to continue.
4. You’ll start by providing the details about your on-premises ONTAP cluster. The information details needed by Cloud Tiering to be able to identify your system are: your cluster management IP, username, and password. This will connect Cloud Tiering to your on-prem system.
Next, you should create a Service Connector that will communicate with your cluster by clicking “Add Service Connector.”
Note: If you already created one, select a service connector deployed in an AWS VPC that has network connectivity to your cluster. The service connector can be deployed on-prem as well. For detailed instructions see the cloud tiering documentation.
5. In the first section you need to choose the cloud provider where the service connector will be created. Click on “AWS”:
6. Next, provide the AWS Access Key and AWS Secret Key. It’s important to note that NetApp does not save this information. You can click on “Policy Requirements” to make sure your account has sufficient permission. When done, click on “Continue”:
7. In the location section, provide your AWS location information. This includes the AWS region, VPC, and the subnet where the service connector will be created. Don’t forget to ensure your subnet connectivity so it can communicate with AWS services. When you’re done, click “Continue”:
8. Next, you will define network information for the service connector instance such as key pair, public IP, and HTTP proxy, if needed. When done, click “Continue”:
9. In the last section you will need to set up your Security Group, either by assigning an existing Security Group or creating a new one. A Security Group acts as a virtual firewall for your instance to control inbound and outbound traffic. When done, click “Go”:
This will begin the service connector creation process. This process takes about seven minutes from start to finish. If the process encounters any errors, they will be shown here. It is important that you do not close this page. When the deployment is complete you will be redirected back to the “Discover ONTAP Cluster” page:
10. Back on the “Discover ONTAP Cluster” page, verify that the cluster information provided is correct and choose the appropriate service connector. When you’re done, click “Continue”:
11. Once your system information has been entered and a service connector selected, your deployment will be able to be recognized. Now we need to set up Cloud Tiering itself. In the Cluster Dashboard click on “Set up Tiering” to start the tiering configuration:
12. Next, a list of all your source volumes will be shown, including those that cannot be tiered. If you want to see only the volumes that can be tiered, click on the “Tier Status” dropdown menu on the right and select “Available for Tiering”, then click “Apply.”
13. Notice that you can also string search for every type of data that exists in the table displayed. For example, you can search for volumes in a specific SVM that you want to select by entering the SVM name:
14. Next, it’s time to select the volumes that you want to tier cold data and the tiering policy that will be assigned to them.
Note: By default, the Auto tiering policy is selected. This policy tiers both cold user data and snapshot data from the selected volumes to Amazon S3. If you wanted to choose another policy, such as tiering snapshots only, this would be where to make that selection.
15. When all the volumes you want to tier are selected, and the desired tiering policy is set, click on “Continue”:
16. Next, you’ll establish cluster connectivity. This involves selecting the IPspace to be used by the cluster to connect to the object storage service.
An IPspace is a logical container that is used to create administratively separate network domains for segregating different types of network traffic. Once an IPspace is selected, click “Continue”:
17. Cloud Tiering lets you tier data to object storage in a number of different clouds. This screen is where you select the cloud object storage provider you want to use. In this walkthrough, we are going to configure Cloud Tiering to Amazon S3. From the grid, select AWS as your “Object Storage Provider” and click on “Continue”:
18. Now you’re going to have to authenticate your AWS account. Click on the “Policy Requirements” link to make sure that your AWS account has the required permissions. Next, provide your AWS credentials. These include your AWS access key and secret key. It’s important to note that NetApp does not save this information. You will also choose the region where you want to deploy the Amazon S3 bucket that will store the tiered data and the storage class. Cloud Tiering will automatically create the bucket for you using the credentials you provide. Click “Continue”:
19. Review the configuration to make sure everything is correct and then click “Tier Cluster”:
20. Once the tiering configuration is completed, you can see a number of different details about your tiering projects under the Tiering health, Current Savings and Savings Opportunities tiles in the Cluster Dashboard. Click on “More Info” for a detailed view:
21. In the “Total Used Capacity” tile, you can view the storage status for volumes that you’ve enabled for tiering. In the example below, you can see that a total of 16.59 TB of cold data has been tiered to Amazon S3. The resulting cost savings of over $8000 is also displayed.
22. The “Cluster Used Capacity” tile shows the additional savings opportunity in the remaining volumes if cloud tiering is enabled. In this example, the total cold data in the remaining cluster volumes amounts to 51.16 TB which can be tiered to Amazon S3. If you decide that you want to tier that data, you can click on “Set up Tiering” to configure tiering for the remaining volumes in your system.
Benefits of Cloud Tiering
With NetApp Cloud Tiering you can choose to tier only the snapshot data of your volume or all cold data including snapshots. The default configuration uses auto tiering, which automatically tiers all the data, including snapshots, that has not been accessed for 31 days. In addition to the cost savings opportunities, Cloud Tiering offers additional benefits to organizations depending on specific use cases:
- The tiering is done transparently at the storage layer. That means no additional configuration or refactoring is required for your enterprise applications to benefit from the technology. It is essentially a zero-effort data extension to cloud.
- There is no learning curve involved as customers can continue to use their familiar processes, tools, and technology to access data.
- When cold data is accessed by applications, that data is tiered back to the on-premises all-flash storage so that the performance is not impacted.
- Using Cloud Tiering allows you to optimize your on-premises AFF storage usage so you can host more workloads on it instead of just for storing cold data. In effect, the capacity can be scaled up 20x.
- Cloud storage uses a pay-as-you-go model that drastically reduces your overall storage costs. There is no upfront investment required and, as per statistics, an average of 80% of data can be moved to low-cost cloud object storage and freeing up your on-premises storage box.
- Cloud Tiering can be managed from a simple modern user interface. You can configure tiering policies, assign them to volumes, and also monitor the cost savings all from the same interface.
Try Cloud Tiering Yourself
With this low-touch automated tiering capability, Cloud Tiering for NetApp AFF systems can act as an entry point for organizations implementing a hybrid cloud architecture. In addition to addressing datacenter space concerns, it also helps reduce your storage TCO. This feature is quite valuable when you are considering a shift from CAPEX to OPEX investment model for storage.