Blog

Cloud Automation: Why, Where and How

The original vision of cloud computing was automated, on-demand services that scale dynamically to meet demand. While this vision is now a reality, it doesn’t happen on its own. Cloud automation is complex and requires specialized tools, expertise, and hard work. Learn why cloud automation is important, where it is used, and how it works.

In this post, we’ll review the potential of cloud automation, its use cases in modern organizations, and common tools used to implement cloud automation strategies. We’ll also show how NetApp Cloud Manager and Cloud Volumes ONTAP can help with the automation of cloud data flows.

In this article, you will learn:

What is Cloud Automation?

Cloud automation enables IT teams and developers to create, modify, and tear down resources on the cloud automatically. One of the major promises of cloud computing was that services could be used on demand, if and when needed. But in reality, someone needs spin up those resources, test them, identify when they are no longer needed, and take them down, and this can represent a huge manual effort.

Cloud automation is not built into the cloud; it requires expertise and the use of specialized tools. You can leverage cloud automation tools and capabilities offered by your public cloud vendor (such as AWS or Azure), automation features in your private cloud platform (such as OpenStack or Cloud Foundry), or third party cloud tools that can perform automation in cloud environments (such as Puppet, Chef, Kubernetes, or Cloudify).

Cloud automation requires hard work, but it pays off when you get through the initial pain and gain the ability to perform complex tasks at the click of a button. Beyond the obvious benefit of reducing manual work, cloud automation has additional benefits:

  • Improved security and resilience—when sensitive tasks are automated, you do not need multiple IT people or developers logging into mission critical systems. The risk of human error, malicious insiders and account compromise is vastly reduced. In addition, you can build security best practices into automated workflows, and enforce security principles in 100% of your deployments.
  • Improved backup processes—organizations need to back up their system frequently, to guard against accidental erasure, configuration calamity, equipment failure or cyber-attack. Automating backups on the cloud, or backing up on-premise systems automatically to the cloud, dramatically improves an organization’s resilience to disaster.
  • Improved governance—when systems are set up manually or on an ad-hoc basis, administrators may have low visibility over what is actually running and may not have a centralized way to control the infrastructure. Cloud automation lets you set up resources in a standardized, controlled manner, which also means you have much more control over infrastructure running across your organization.

Cloud Automation vs Cloud Orchestration: What is the Difference?

Orchestration is one more step beyond automation. Orchestration involves scheduling and integrating automated tasks between different systems.

For example, consider a system with three nodes running the application, a central database, and a load balancer. Deploying this complex system requires a carefully coordinated series of actions—starting up the database, starting up the nodes, ensuring they connect to the database successfully, then setting up the load balancer.

While automation would involve performing each of these isolated tasks automatically, an orchestration system would be able to perform them in the correct order and timing, and ensure that the entire system is up and running. Another important use of orchestration is to manage connectivity. An orchestration system can ensure components are connected to the local network or the Internet, and are communicating with the remote services they need to function.

For example, orchestrating an app means not only deploying an application, but also connecting it to the network so it can communicate with users and other apps. In the cloud, orchestration is often key to ensuring that automated spin-up activities like auto-scaling take place in the right order, with the right security rules and permissions in place.

Cloud Automation Use Cases

Cloud automation has a variety of uses in modern enterprises and development workflows. Here are some of the most common scenarios.

Infrastructure as Code (IaC)
In the IaC paradigm, IT infrastructure should be defined in configuration files and launched automatically according to that configuration. Configuration files should be kept in version control, managed, tested and evolved, just like source code. Cloud automation makes IaC much more powerful, because public and private clouds have control over almost all aspects of the infrastructure, allowing orchestration of complex systems. Read our articles about IaC on AWS and IaC on Azure.

Workload management
The most basic form of workload management is autoscaling, a cloud automation feature that is a staple of any public cloud provider. In addition to control costs on public clouds, it is essential to track what is running, clear unused resources, and adapt the infrastructure to actual operating requirements. Cloud automation helps administrators monitor their environment and automatically adjust workloads as needed.

Application development and testing
Continuous delivery and continuous deployment, which are very widely practiced in agile development organizations, require realistic dev/test environments that are automatically provisioned as part of automated development pipelines. Cloud automation on the public cloud has been, and remains, the primary way to serve the continuous delivery pipeline and generate the environments developers need. Read our article on DevOps benefits and strategies for automating DevOps in a hybrid cloud environment.

Hybrid cloud
Hybrid cloud environments are complex and require intricate integrations between on-premise and public cloud systems. Cloud automation makes it possible to synchronize assets between local data centers and cloud resources, to perform “cloud bursting”—shifting loads to the cloud when local resources are exceeded, enable disaster recovery scenarios with a remote DR site mirroring the on-premise environment, and more. Learn more about how NetApp can help automate hybrid cloud storage.

Cloud Automation Tools

Cloud automation can be done using throwaway scripts, but the industry has realized that due to the complexity of cloud environments and the need for intricate orchestration of many day-to-day tasks, it is better to rely on a mature automation platform. Here are a few systems commonly used to automate and orchestrate tasks in the cloud.

Puppet
Puppet is a veteran configuration management tool, and one of the pioneers of Infrastructure as Code (IaC). Its commercial version, Puppet Enterprise, is especially suited to cloud workloads and supports huge sale, up to 20,000 Puppet nodes. It can be used for public, private, and hybrid clouds, and offers a GUI interface for defining configuration and automation tasks, as well as Domain Specific Language (DSL) scripts.

Ansible
Ansible is another configuration management product that is synonymous with IaC, developed by Red Hat. Its commercial version, Ansible Tower, supports full orchestration capabilities, with job scheduling, management of a resources inventory, and the concept of “playbooks” that let you automate complex series of actions. You can define configurations using its GUI, DSL, or a REST API. It provides strong support for all public cloud providers and is a robust option for private cloud as well.

Kubernetes
Kubernetes is not by definition a cloud automation solution—it doesn’t allow you to automate all your cloud resources. However, Kubernetes manages and orchestrates Docker containers, which you can use to run pretty much any workload on the cloud. It manages deployment of services within containers, resource consumption, networking within the Kubernetes cluster and with the outside world, and more. Kubernetes is widely used on the public cloud; all public cloud providers offer robust Kubernetes as a service offering, such as Amazon Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE) and Azure Kubernetes Service (AKS).

Learn more in our series of articles about managing and automating Kubernetes storage.

AWS Elastic Beanstalk
Elastic Beanstalk is an automation platform that lets you deploy applications developed with Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker, across EC2 instances. You upload your code, and Elastic Beanstalk handles provisioning of the application, autoscaling, load balancing, monitoring, and more.

AWS CloudFormation         
AWS CloudFormation is Amazon’s Infrastructure as Code platform, which allows you to define declarative templates that specify what you want to run in the cloud, and have Amazon automatically spin up the required resources. You can use CloudFormation templates to automate anything from routine deployment of an S3 bucket or EC2 instance group, to highly complex, large-scale applications combining multiple Amazon services.

Terraform      
Terraform is an open source tool by Hashicorp, which lets you build, modify and version Infrastructure as Code. It does not provide the configuration management aspect, so it works together with CM tools like Puppet or Chef. Terraform takes configuration files and generates execution plan that describes how it will reach the desired state. When configuration changes, it can create new execution plans to adapt the resources to the new desired state. It can manage anything from computer, storage and networking, to DNS and higher-level application features.

Cloud Data Automation with Cloud Volumes ONTAP

NetApp Cloud Volumes ONTAP, the leading enterprise-grade storage management solution, delivers secure, proven storage management services on AWS, Azure and Google Cloud. Cloud Volumes ONTAP supports up to a capacity of 368TB, and supports various use cases such as file services, databases, DevOps or any other enterprise workload, with a strong set of features including high availability, data protection, storage efficiencies, Kubernetes integration, and more.

In particular, Cloud Volumes ONTAP provides the Cloud Manager, a UI and APIs for management, automation and orchestration of cloud storage, supporting hybrid & multi-cloud architectures.

Learn More About Cloud Automation

There’s a lot more to learn about cloud automation. To continue your research, take a look at the rest of our blogs on this topic.

Infrastructure as Code on AWS
Amazon Web Services provides powerful Infrastructure as Code (IaC) capabilities via the CloudFormation service. Learn how CloudFormation works, the benefits of using it to build a reusable, consistent IT strategy, and quick tips to get your IaC project started.

Read: Infrastructure as Code on AWS: 5 Tips to Get You Started

Infrastructure as Code on Azure
Microsoft Azure provides the Azure Resource Manager (ARM), which lets you create JSON templates to automatically spin up any combination of IT resources. Learn how to use ARM to implement your Infrastructure as Code (IaC) strategy on Azure.

Read: Save Time and Headaches with Infrastructure as Code on Azure

How to Automate Storage Volume Provisioning with Ansible and Cloud Volumes ONTAP
Enterprises scale makes cloud automation a necessity for storage volumes provisioning. In this post we give code examples for how to combine NetApp Cloud Manager APIs and Ansible to create volumes in NetApp Cloud Volumes ONTAP without challenging manual overhead.

This cloud automation process for provisioning storage volumes helps limit the costs of manual effort required to create volumes, speeds up compliance practices, and reduces errors.

Read: How to Automate Storage Volume Provisioning with Ansible and Cloud Volumes ONTAP

Infrastructure as Code Deployment of Cloud Volumes ONTAP and Cloud Manager Using Terraform and Ansible
IaC is commonly used for compute resources, but it can be challenging to automate storage, due to the need to replicate and restore large volumes of data in a timely manner. Learn how to automate storage managed by Cloud Volumes ONTAP using declarative IaC templates, using NetApp Cloud Manager, Terraform and Ansible.

Read: Infrastructure as Code Deployment of Cloud Volumes ONTAP and Cloud Manager Using Terraform and Ansible

Cloud Automation with Cloud Volumes ONTAP and REST
NetApp Cloud Manager provides a convenient UI that lets you work with storage managed in Cloud Volumes ONTAP very straightforward. Learn how the same operations can be performed programmatically using the Cloud Manager REST API.

Read: Cloud Automation with Cloud Volumes ONTAP and REST

Deploying Cloud Volumes ONTAP and OpenShift Using Ansible
If you are using Cloud Volumes ONTAP as part of an OpenShift deployment, you can use Ansible to deploy storage services automatically. NetApp is one of six Red Hat Ansible Certified Module Vendors. There are multiple NetApp ONTAP modules you can use for common Cloud Volumes ONTAP configuration tasks. Learn how to leverage Ansible to make your setup process up to 96 times faster.

Read: Deploying Cloud Volumes ONTAP and OpenShift Using Ansible

-