Storage
Mobility
Protection
Analysis and control
TOPICShare
Serverless cloud platforms such as AWS Lambda eliminate the need to manually deploy and scale the base infrastructure used by applications and services in the cloud. Instead of thinking about servers and containers, you simply write the code that should be executed and deploy it, leaving it to AWS to manage high availability and to respond appropriately to changes in user demand. It is for these reasons that the deployment of serverless microservices in cloud file sharing architectures is growing rapidly.
Despite this focus on serverless, AWS Lambda functions will still frequently require access to persistent data stores such as databases, object storage, and shared file systems.
In this article we will look at the benefits of deploying AWS file storage solutions in serverless architectures with the help of AWS Lambda, from Amazon EFS and S3 to the latest developments in FSx and NetApp Cloud Volumes ONTAP.
Use the links below to jump down to the sections on:
AWS File Storage Integration with AWS Lambda
AWS Lambda allows development teams to productionize new code within minutes, while keeping the actual business requirements front-and-center. An AWS Lambda function is, as the name implies, simply an individual function that will be called into by AWS whenever the function’s triggering criteria has been met. These functions can be written in any one of a multitude of languages, such as Java/C#, JavaScript, Python, Ruby, or Go. Different AWS Lambda functions can also be connected together and integrated with other AWS services to build complex workflows using AWS Step Functions.
There are a number of options for file storage AWS offers that can be used in serverless deployments. In the following sections, we will examine the pros and cons of some of these approaches.
Amazon S3
Amazon S3 storage is an outstanding solution for durable and cost-effective capacity storage in the cloud. It is also possible to trigger AWS Lambda functions when a new file is uploaded to Amazon S3, thereby initiating a data pipeline. In these kinds of AWS Lambda / S3 operations, in order to access files within Amazon S3, you must use the standard AWS APIs, usually via the Amazon SDK, from within your function code. The IAM role associated with each function will govern the level of access granted to Amazon S3 buckets.
As Amazon S3 provides object storage, it is most often not an ideal solution for file storage. For example, a write to an Amazon S3 object is an all-or-nothing operation, you can’t simply do a partial update. Also, environments that require high performance access to data will encounter problems if Amazon S3 is the primary data store, as this type of data access is not what this storage is designed for. But while it isn’t advisable or recommended by AWS, it is possible to use S3 as a file system solution.
Amazon EFS
AWS EFS is a managed solution for deploying file storage on AWS, allowing you to very quickly deploy scalable file systems that natively integrate with AWS Lambda. Amazon introduced access to Amazon EFS file systems from AWS Lambda to address the importance of shared files to various enterprise and data-focused applications that require a conventional file system structure.
Amazon EFS file systems that must be accessed by an AWS Lambda function must be registered in advance within the function’s configuration, along with a mount point. AWS will then ensure that the NFS file share is correctly mounted on any host that will execute the function. On execution, the function itself simply accesses the mount as a normal directory.
Although it can be very easy to get started with Amazon EFS, certain issues can present themselves after any period of real usage. Due to the system of burst credits used to determine AWS EFS performance, sustained high throughput activity can lead to a degradation in file access performance until credits have been replenished. This can be countered by either allocating more storage space, which would give access to a larger number of credits, or moving to provisioned mode. Both options means incurring higher operational costs.
AWS Lambda / EFS deployments do have one challenge to consider: Amazon EFS only supports NFS v4, which may cause issues if the files being used by AWS Lambda must also be accessed by SMB/ CIFS or NFS v3 clients.
Amazon FSx Options
To meet the wide range of different file protocol use cases for file sharing, AWS launched the FSx line of shared storage services. The benefit to AWS FSx is that it allows users to remain with trusted file storage solutions that they already use but to also take advantage of AWS and all the benefits of cloud deployment.
There are currently FSx versions for four major file storage systems: Amazon FSx for Netapp ONTAP, Amazon FSx for OpenZFS, FSx for Windows File Servers, and Amazon FSx for Lustre.
Each of these choices comes with the core capabilities inherent to FSx:
- Fully managed service
- Cost optimization
- High availability
- Hybrid architecture support
A number of the FSx services, including FSx for ONTAP, can enable SMB access for serverless architectures via Lambda.
So which of the FSx types is the right solution for you? The choice between the services largely comes down to your existing shared file storage infrastructure. ONTAP users can extend on-prem deployments to the cloud without any retooling or staff education by migrating to AWS with FSx for NetApp ONTAP. FSx for ONTAP comes with the full range of NetApp support capabilities, such as SnapMirror® data replication, FlexClone® data cloning, Global File Cache, FlexCache, and cost-cutting storage efficiencies.
Read more about using FSx for ONTAP in File Sharing with Amazon FSx for NetApp ONTAP.
NetApp Cloud Volumes ONTAP
Using Cloud Volumes ONTAP, you can fully leverage NetApp’s NAS and SAN technology in any of the major hyperscaler environments, namely AWS, Azure, and Google Cloud Platform. NetApp has spent decades building multiprotocol file storage solutions that are recognized across the industry for their performance and scalability. Cloud Volumes ONTAP is an enterprise data management solution that provides consistency across multicloud and on-premises environments.
Cloud Volumes ONTAP is built on top of the latest native compute, storage, and networking resources in the cloud. So, in AWS, Amazon EC2 and Amazon EBS are used to deploy NetApp storage services, with the additional option of using Amazon S3 as transparent capacity storage in a tiered storage architecture.
To enhance performance with Cloud Volumes ONTAP, multiple Amazon EBS volumes can be combined together to form a single primary storage pool for data to be stripped and read more efficiently. Also, local NVMe-based SSDs (on supported instances) can be leveraged for intelligent caching, which leads to dramatic read performance improvements.
On the NAS side, Cloud Volumes ONTAP supports both NFS v3 and v4, as well as SMB/ CIFS versions 2.x and 3.x. All shared file systems deployed using Cloud Volumes ONTAP benefit from:
- High Performance: Caching technology helps bring data closer to the client that needs to use the files.
- Storage Efficiency: Reduce your cloud data storage costs and operational expenditure through transparent data compression, deduplication, and thin provisioning.
- Highly Availability: Cloud Volumes ONTAP can be deployed in an AWS high availability configuration that provides zero downtime and zero data-loss across Availability Zones.
- NetApp Snapshot™: Instantly create point-in-time copies of your files, either for backup or for data versioning.
- FlexClone: Create temporary writable copies of data from your snapshots, and dramatically simplify data-oriented testing with FlexClone data cloning.
- Regional Replication: Data can be efficiently replicated between instances of Cloud Volumes ONTAP that may span AWS regions via SnapMirror.
Conclusion
AWS Lambda functions are able to integrate with various forms of AWS file storage, including file shares hosted on Cloud Volumes ONTAP.
To see how to make Cloud Volumes ONTAP and AWS Lambda part of your serverless file storage solution, check out this walkthrough on how to set up NFS storage with AWS Lambda and Cloud Volumes ONTAP.
