Stay informed Subscribe to our Blog
As public cloud adoption increases year after year and enterprises continue to migrate production data to the cloud, it’s important for stakeholders to devise a backup strategy. Data stored in the public cloud needs to be able to be restored and snapshots are key to that process.
In this article, we look at the mechanisms behind the snapshot technologies at the two leading public cloud providers, Azure and AWS. In the other parts of this series, we saw how Azure snapshot and AWS snapshot costs compare, and took a deep dive into NetApp’s Cloud Volumes ONTAP snapshots.
Azure Snapshots: Azure Backups and More
Azure Storage has two types of virtual machine disk storage: unmanaged and managed. In unmanaged disks, the tenant maintains and manages the disks which are stored in page blobs which in turn are stored in Storage Accounts. With managed disks on the other hand, Microsoft manages the service, freeing the tenant from the responsibilities of managing the storage account and its imposed limits such as disk size, IOPS, and storage bandwidth limits. Below we’ll take a look at how snapshots work in Azure managed vs. unmanaged disks.
With unmanaged disks, we want to create a snapshot of the blob object. When a snapshot of the Azure blob storage is created, the snapshot has the same URI (Uniform Resource Identifier) as the base blob with a value attached to it formatted as DateTime that indicates when the snapshot was taken.
For example, if we look at a base page blob URI https://netappaz.blob.core.windows.net/netapppb/netappsnap.vhdx, the snapshot URI will be: https://netappaz.blob.core.windows.net/netapppb/netappsnap.vhdx?snapshot=2018-06-02T04:04:51.9827557Z.
When a snapshot is created, the blob’s system properties and metadata are copied to the snapshot, excluding the blob’s lease values, as there cannot be any leases on the snapshot. You also have the option to modify the metadata of the snapshot during creation in order to uniquely identify the snapshot. To ensure an application-consistent snapshot, it is recommended to detach the disk from the VM or shut down the VM while taking the snapshot. Alternatively, for Azure backup and recovery, you can use tools like Azure Backup that will coordinate with the VSS on the VMs to ensure that the snapshots taken are application-aware while the VM is running. Although the VSS can only be used for Windows VMs, in Azure Backup a file-consistent backup is offered for Linux VMs and you can also create your own script to perform flushing or similar operations so that the snapshot is application-consistent.
As a backup strategy, snapshots are taken periodically and copied to another storage account using CopyBlob or AzCopy tools. This copied snapshot is stored in the form of page blobs. The incremental changes are copied by comparing the snapshot of the base blob and the backup blob using GetPageRanges and PutPage.
During the restore process, the point-in-time snapshot to be restored is promoted as the backup base page blob. After creating a snapshot of these newly promoted page blob, the page blob is restored as a disk in the source storage account. This restored disk is attached to the Azure VM and the old disk is detached and decommissioned.
Azure Managed Disks
With managed disks, disks are treated as first class objects, which means as far as the user is concerned, there is no storage account to maintain. It is very easy to create snapshots of managed disk VHDs by simply clicking on the “Create Snapshot” option in the Managed Disk blade. Alternatively, you can use PowerShell cmdlets like New-AzureRmSnapshotConfig and New-AzureRmSnapshot to create the snapshots. At the time of writing this article, only full snapshots are allowed, and these snapshots will be stored as VHDs based on the choice of resource group and storage type (e.g. LRS).
Creation of disks from these snapshots is also straightforward. You can use the Azure portal to create managed disks and point to the snapshot taken to create a new managed disk. Alternatively, here is an example of how you can achieve this through PowerShell.
One of the most common use cases for AWS snapshots is backing up Amazon EBS volumes to Amazon S3 by taking point-in-time snapshots. AWS snapshots are incremental and consist of changes in blocks from the most recent snapshot. On deletion, only the data unique to the snapshots are removed. These snapshots are created asynchronously, and the data is loaded lazily to the replicated volume in the background; while snapshot creation is immediate, its status will be pending until completion of the transfer. This data or block transfer process means it can take hours to create the initial replica or to update if there have been numerous block changes since the last snapshot.
As applications and processes can cache data in RAM and flush to the disk on their own schedule, it is pertinent to either stop the Amazon EC2 instance or detach the Amazon EBS volume to take an application-consistent snapshot.
By default, Amazon EBS snapshots are stored in Amazon S3 accounts maintained by AWS. Once the lazy copy has finished and the status of the snapshot is marked completed, you can copy the snapshots to another region or within the same region. Data in transit and at rest during this transfer are encrypted by AES 256-bit encryption. These copies can also be incremental where the first copy is a full replica and subsequent copies are incremental in terms of changed blocks. The copy operation is useful in use cases such as migrating an application to a new region, disaster recovery, and data retention. Amazon CloudWatch Events can provide notifications on events like completion of a snapshot creation. This in turn can trigger an AWS Lambda function to copy it to another region for DR. It is also possible to set up automated EBS snapshots creation.
Amazon EBS volumes can be restored from a snapshot by the owner or by sharing the snapshots with another account. If the snapshot is a shared snapshot, then the account must create a copy of the snapshot so that they own the resultant snapshot in order to restore the volume.
Amazon EBS volumes can be restored easily using the Amazon EC2 console, AWS CLI, or AWS Tools for Windows PowerShell. A restored volume can be attached as soon it is created. The data from the Amazon EBS snapshots are loaded lazily but if data is accessed from the restored volume that has not been loaded yet, the requested data is immediately downloaded, and the lazy copy operation continues. Unlike in Azure, the snapshots cannot be downloaded or exported outside the AWS portal.
Final Note and Cloud Volumes ONTAP Snapshots
It is important to understand the lifecycle of a snapshot and how that affects your RTO and RPO objectives. Although periodic snapshots are possible, and it can be automated, the process becomes a bit complex and cumbersome. The costs of Azure snapshots and AWS snapshots are also a consideration, as we saw in the second part of our series on snapshots, and in the first part, which covered NetApp’s Cloud Volumes snapshots.
Cloud Volumes snapshots offer ways to overcome the various limitations of cloud-native snapshots, such as the full snapshots used by Azure managed disks and the full initial replicas of Amazon EBS snapshots. In addition to this, Cloud Volumes ONTAP also provides other features such as cost-cutting storage efficiencies, data tiering to object storage on Amazon S3 or Azure Blob, instant clones with minimal storage ideal for DevOps workflows, a centralized management console in Cloud Manager, and integration with third-party applications to provide application-aware snapshots.
There are a lot more articles about AWS snapshots on NetApp Cloud Central.
The S3 Outage: Be Prepared for Unavoidable Cloud Failures with AWS Snapshots
Think the cloud can’t fail? The major AWS outage of 2017 was a sign that no matter how much confidence we have in the cloud, users still need to have contingency plans in case the entire cloud infrastructure comes crashing down, taking your sites and applications with it.
Users do have options. Turning to NetApp cloud solutions can help protect your data and make sure that you can operate and recover from the worst cloud outages.
Automating Your Disk Backup and Data Archive Part 1: AWS Database Backup with AWS Snapshots
Backups still remain an important part of making sure database systems are protected. While live system recovery is now more frequently managed through automatic failover and failback systems and high availability infrastructures, snapshot-based backups do still play a part in recovering from critical threats such as mistaken deletion, data corruption, and malicious software attacks.
This article is the first in a series that takes a deep dive into the snapshot backup methods that can be used to protect cloud-hosted databases focuses on recovering and backing up AWS databases. AWS snapshots for Amazon EBS are a key feature here for both all-cloud and hybrid cloud environments.
Automating Your Disk Backup and Data Archive Part 3: NetApp Backup with Cloud Volumes ONTAP Data Backup Solutions
In the final part of the series on cloud-based database backup we take an in-depth look at Cloud Volumes ONTAP and how it leverages NetApp Snapshots to protect database systems using AWS or Azure storage in the cloud.
NetApp Snapshots are more efficient in terms of space, costs, and time to create than any other cloud-based snapshot utility. Snapshots are also the basis of Cloud Volumes ONTAP’s instant data cloning technology, FlexClone. What do these features mean when it comes to protecting the databases that your enterprise relies on?
Cloud Snapshot Costs: AWS Snapshots, Azure Snapshots, and Cloud Volumes ONTAP
Snapshots are an important part of keeping data protected no matter which cloud you’re using. But if you’re not aware of the costs involved both in the creation of snapshots and for storing them, you might wind up straining your cloud budget, especially when there are demands for consistency, which requires constant backing up.
This post deals with managing the costs of the native snapshots capabilities that are available for Amazon EBS and for Azure storage. You’ll see examples of how the snapshots are created and the pricing formulas on each service. Plus, you’ll see how NetApp Snapshots with Cloud Volumes ONTAP Cloud give a cost-saving third option for snapshot protection that works in any cloud.
NetApp SnapMirror Data Replication with AWS
Anyone who is familiar with NetApp storage systems in the data center knows SnapMirror data replication makes it easy and fast to transfer and replicate data between storage repositories. This powerful feature is also a key part of using storage in the cloud with Cloud Volumes ONTAP. SnapMirror is integral to migrating to the cloud, managing a disaster recovery solution, and orchestrating hybrid or multicloud environments.
SnapMirror is based on NetApp’s highly efficient snapshot technology, which consumes less storage and thus costs less than using AWS snapshots via Amazon EBS. You’ll also see how SnapMirror can migrate snapshots that are application consistent though integration with NetApp SnapCenter®.
Storage Snapshots Deep-Dive: Cloud Volumes Snapshots
You’ve heard a lot in this article about NetApp Snapshots and how Cloud Volumes uses this snapshot technology to make cloud deployments with AWS and Azure less expensive and better protected. In this article, we take a deep dive into how this technology operates.
NetApp Snapshots are built on the WAFL (Write Anywhere File Layout) technology innovated by NetApp. How does WAFL work and what does it do? In this article you’ll learn that and more, such as how snapshots underpin several of the technologies used by Cloud Volumes ONTAP, from SnapMirror data replication and SnapVault data archiving, to data cloning with FlexClone® and data restoration with SnapRestore.