hamburger icon close icon
DevOps

DevOps: The Complete Guide to Culture, Technology and Tools

Image 1-1
Yifat Perry

Product Marketing Lead

Dec 1, 2020 7:49:11 AM

What is the DevOps Methodology?

DevOps is an information technology (IT) methodology that facilitates collaboration, communication, and integration between software developers and IT operations staff. The primary purpose of DevOps is to improve the quality and speed of software delivery, enabling continuous, frequent updates that deliver value to customers.

The DevOps team works together to create a consistent development, testing and production environment, and automates the development pipeline, to make software delivery efficient, predictable, sustainable and secure.

DevOps gives developers better control over their infrastructure and a clearer understanding of the production environment, and encourages operations specialists to be involved from the onset of the development process. It creates a culture of shared ownership and responsibility for software that runs and delivers value in production.

What is a DevOps Culture?

DevOps is not just a methodology or a practice, it is also an organizational culture. DevOps culture focuses on small interdisciplinary teams that can work independently, and are jointly responsible for the user experience delivered by a software product. The DevOps team lives in production—they are primarily focused on improving live usage of the product.

A DevOps culture has the following essential elements:

  • DevOps teams adopt agile practices and integrate both development and operations into each team's responsibilities. Teams work in small iterations, striving to improve the end-to-end delivery of customer value, and removing waste and obstacles from the process. Teams are jointly responsible, eliminating silos or finger pointing.
  • DevOps teams apply a growth mindset—they use monitoring and telemetry to collect evidence in production and observe results in real time. They experiment with new features in production, using techniques like canary or blue/green deployments, to quickly collect data, test features, and use the results to drive continuous improvement.
  • DevOps teams focus on mean time to mitigate (MTTM) and mean time to remediate (MTTR) rather than mean time between failures (MTBF). In contrast to traditional waterfall teams that made major efforts to prevent problems in the field, the DevOps team recognizes that failures will happen, and emphasizes the ability to act quickly, understand the impact, and mitigate production issues.
  • DevOps teams think in terms of competencies, not roles—this includes both development and operational skills. All team members share responsibility for running services. Both developers and operations are responsible for live services, and all of them may share a rotating on-call schedule. If you built it, you’re responsible for running it.

DevOps vs Agile

What is the difference between Agile and DevOps? Both methods can facilitate collaboration and increase

 

Agile

DevOps

Process

Communication between the team and customers is continuous, and frequent changes are made to the software to ensure quality. Better suited for complex projects.

Focuses on frequent testing and delivery, but communication is primarily between developers and IT operations. Better suited for end-to-end processes.

Teams

Allows small teams to complete tasks faster. Agile methods encourage all members to share responsibilities equally instead of assigning specific responsibilities to team members. So, every agile team member should be able to handle or assign any part of the project at any time.

Suitable for large teams. Skill sets are distributed among operations and development team members: Each team member has a specific set of tasks they need to do at each stage of the SDLC.

Focus and feedback 

Typically works in sprints, each sprint lasting less than a month. The idea of a sprint is to complete the project step by step, starting each sprint right after delivery of the previous sprint’s deliverables.

Focuses on operations and business readiness. Most of the feedback comes from internal team members and metrics collected from production environments. Deadlines and goals may recur on a daily basis.

DevOps and CI/CD

DevOps closes the gap between development, operations and IT service teams. One of the primary ways it does this is by Implementing DevOps tools. These tools are implemented as pipelines, typically using continuous integration, continuous deployment, and continuous delivery (collectively known as CI/CD) workflows. CI/CD pipelines serve as structured environments that enable and promote fast delivery of high quality software.

The CI/CD implementation process uses a daily or weekly release cycle, allowing the DevOps team to get quick feedback from customers on new versions of the product.

Key CI/CD concepts and their relation to DevOps:

  • Continuous Integration—a process that ensures new code is introduced daily. This is typically done by integrating the new code into shared repositories, accessed daily by all collaborators. This is an improvement over the traditional way, in which developers would write new code and check it into the repository only at the end of the project lifecycle. The goal is to be able to detect integration errors early and correct them quickly. In a CI workflow, a new build is triggered every time new code is merged with the main branch of the code repository. Automatic tests are run for each new version to see if the latest changes broke the build.
  • Continuous Delivery—CD begins where CI ends. Continuous delivery pipelines automate the software delivery process, pushing integrated code to production without errors or delays. While implementing DevOps, CD helps developers merge new code consistently with the master branch, allowing them to automatically build new market-ready versions of the product. CD automation runs tests to ensure that each new build can be released into production.
  • Continuous Deployment—an extreme form of continuous delivery, continuous deployment means that for each major code change, the pipeline generates a new build and immediately deploys it to the production environment, with no manual intervention. This is the "Holy Grail" of the DevOps team. This is the “holy grail” of DevOps teams, because it ensures every code change is immediately running in production and the entire team can instantly see its quality and operational properties. 

Learn more in the in-depth guides to: 

The Transition to DevSecOps

The term DevSecOps stands for development, security, and operations. DevSecOps is a methodology that automates security processes across all phases of the software development lifecycle, including the initial design, testing, deployment in production, and software delivery. DevSecOps is generally considered an improvement of former software development practices, enabling teams to integrate security seamlessly into the pipeline.

Past development paradigms tacked security on at the end of the cycle, assigning this task to a separate, often siloed, security team. When done, the security team passed the product to an independent quality assurance (QA) team for testing. It was possible when teams released software updates only once or twice annually. However, as teams adopted agile and DevOps practices to reduce development cycles to weeks or days, this traditional approach to security resulted in unacceptable bottlenecks.

Here are the core benefits of DevSecOps:

  • Early remediation - DevSecOps aims to integrate infrastructure and application security seamlessly into agile and DevOps processes. It enables teams to address security issues as they emerge while it is easier, faster, and cost-effective to fix them - and teams can handle many security issues before releasing the product into production.
  • A security culture - DevSecOps ensures all teams, including development, security, and IT operations, share the responsibility over application and infrastructure security, eliminating the security silo.
  • Automation - DevSecOps pipelines enable faster release of secure software by using automation. It allows teams to automate the delivery of secure software without slowing the development cycle.

SRE and DevOps

Software reliability engineering (SRE), also called site reliability engineering, is a practice that is being adopted at many organizations alongside DevOps, and can complement and benefit DevOps teams. 

The SRE approach is to apply software engineering ideas to operations topics. Instead of treating operations as an ongoing maintenance effort, software reliability engineers build systems that make software services more reliable and easier to operate. 

SRE vs DevOps

Most practitioners agree that SRE is not a competitor to DevOps, rather it is a complementary methodology. In the Google book defining SRE, DevOps is compared to a programming language, while SRE is a class that implements it. The DevOps philosophy defines the overall behavior of the service management framework, while SRE is a way to implement DevOps and measure its success in improving product reliability.

There are four key ways SRE complements DevOps:

Accept failure as normal
Like DevOps, SRE encourages shared responsibility between IT and development. SRE makes it possible to innovate and make drastic changes to products, which may result in faults. It introduces the concept of a “risk budget”, which SREs to set measurable limits for risk, while encouraging innovation. SRE also assumes that aiming for 100% availability is not compatible with growth and ongoing development.

Implement gradual changes
SRE, like DevOps, leverage gradual change to promote continuous improvement. SRE encourages small changes and frequent deployments, meaning that negative impact will be smaller, easier to test and remediate. SREs implement automated testing of production changes, which can operate alongside CI/CD pipelines used by DevOps teams.

Use tools and automation consistently
DevOps pipelines rely heavily on automation and the adoption of new tools. SRE workflows, on the other hand, prioritize standardizing technology and information across the organization. To achieve this, SRE implementations require collaborators to use the same stacks. This can be beneficial for DevOps, because even in a full DevOps environment, the use of different technologies and tools may cause teams to unintentionally split into silos.

Measuring everything
Measurements are crucial in both DevOps and SRE. DevOps focuses primarily on process performance and achieves continuous improvement via the CI/CD feedback loop. SRE treats operations issues as software engineering issues, so it measures service level objectives (SLO) as its key indicator. By combining process goals with production SLOs, teams can achieve not only faster delivery, but also software that becomes more robust and reliable with each release.

Learn more in the in-depth guide to software documentation

DevOps and Digital Transformation

DevOps and Digital Transformation

Digital transformation is a global transition of businesses to processes and strategies using digital technology. It is widely recognized that without digital transformation, businesses will find it difficult to compete in the modern economy.

A digital transformation requires constant efforts from development and operations teams, to create and integrate new technologies supporting business processes, employees and customers. One of the key elements of a successful digital transformation is a DevOps mindset. 

DevOps ensures that organizations build technologies that are useful, easy to maintain, and capable of evolving to support changing requirements. While traditional IT is associated with legacy technology that supported old-school business, DevOps will gradually become synonymous with digital transformation. 

Learn more in the in-depth guide to digital transformation

DevOps and Cloud Cost Management

As organizations transition infrastructure to the cloud, cost management plays a pivotal role. DevOps can have a significant impact on cloud costs, by introducing efficiency, automation, and better resource management.

DevOps' inherent emphasis on automation and efficient deployment of resources plays a crucial role in keeping cloud costs under control. Automated provisioning and de-provisioning of resources in response to demand spikes, load balancing, and continuous monitoring for idle or underused resources are some of the ways DevOps can contribute to efficient use of cloud resources, ultimately leading to cost savings.

For instance, Infrastructure as Code (IaC) can help ensure efficient resource use by standardizing environments, eliminating "environmental drift", and reducing the need for over-provisioning. Similarly, continuous integration and continuous delivery (CI/CD) pipelines help reduce wastage and streamline processes by enabling teams to identify and fix issues early in the development lifecycle, which can prevent costly failures in production.

Learn more in the in-depth guides to:

The DevOps Technology Stack

The DevOps development pipeline relies on an entire technology stack that enables automation, efficiency and collaboration. Below we describe several elements of this stack, which may be used in different combinations by different teams.

Cloud Automation

Cloud automation allows IT teams and developers to automatically create, modify, and delete environments in the cloud. DevOps has leveraged cloud computing since its early days, to enable complete end-to-end automation of development and delivery pipelines.

However, automation is not built into the cloud. It requires specialized knowledge and uses specialized tools, some of them offered by public cloud providers, some as part of private cloud platforms, and some third-party tools, notably configuration management, infrastructure as code (IaC) tools, and orchestration tools like Kubernetes. These skills and tools are an essential part of any DevOps team.

Feature Flags

Feature flags, also known as feature toggles, are a powerful tool in the DevOps toolkit. They allow developers to hide, enable, or disable a feature during runtime. This means that developers can deliver new features to production even if they aren't complete or fully tested, without affecting the end users.

Feature flags can be used to perform A/B testing, canary releases, or phased rollouts. For example, a new feature can be released to a small percentage of users, and if everything works as expected, the feature can be gradually released to all users.

This approach reduces the risk associated with deploying new features and gives developers more control over the release process. Furthermore, it allows for quicker feedback, as features can be tested in the production environment early in the development process.

Learn more in the in-depth guide to feature flags

Infrastructure as Code for DevOps

Infrastructure as code (IaC) uses the same descriptive model that the DevOps team uses for code—version control—to manage infrastructure, including virtual machines, networks, and storage. Just like the same source code always produces the same binary code, an environment configuration should be able to consistently reproduce an environment every time. 

IaC solves the age old problem of environmental drift. Without IaC, IT teams had to maintain the setup of each deployment environment. Over time, each environment becomes a “snowflake”, a unique creation which cannot be replicated automatically. Inconsistencies between dev, test, and production environments can cause problems during deployment, introduce errors and involve complex manual processes.

Infrastructure as code allows DevOps teams to test applications in a production-like environment early in the development cycle. They use it to reliably set up multiple test environments as needed. Environment configuration code can be checked into version control, tested, and modified as needed, until a stable environment configuration is found.

DevOps teams implementing IaC can provision stable environments quickly and at scale. They ensure consistency by expressing the required environment state in code, without ever manually configuring the environment. Infrastructure becomes repeatable and reliable.

Learn more in our in-depth guide to infrastructure as code for DevOps.

Infrastructure as Code (IaC) on Amazon Web Services (AWS)

The primary IaC service on AWS is CloudFormation. It uses templates, which are simple configuration files using YAML or JSON syntax. Templates are easy to read and edit, and can be used to define resources for deployment. CloudFormation reads templates and then creates a set of ready-to-use resources for AWS.

CloudFormation allows DevOps teams to use templates to automatically build anything, including basic resources and complex applications using many resources. You can fine-tune configuration and repeat the process to create reliable environments, which can then be replicated for development, testing and production stages of the DevOps pipeline.

Learn more in our in-depth guides to:

Infrastructure as Code on Azure

Microsoft Azure allows DevOps teams to define and deploy infrastructure as code using Azure Resource Manager (ARM) templates. ARM templates provide declarative definitions of any cloud resources within an environment. Azure automatically sets up resources reliably and consistently from the template. 

Azure also provides "blueprints" which package ARM templates with policy and RBAC definitions—giving DevOps teams everything they need to set up cloud resources end-to-end for dev, test, and production environments.

Learn more in our in-depth guides to:

GitOps

GitOps enables you to implement continuous deployment (CD) for cloud native applications. It provides a developer-centric experience for operating infrastructure, enabling developers to use tools they are already familiar with, such as Git, to operate and automate the infrastructure.

You can implement GitOps by setting up a Git repository containing IaC templates that represent the desired production environment. GitOps creates an automated process that matches the production environment with the desired state described in the repository.

Once you have configured GitOps for your environment, you can deploy new applications or update existing ones by updating the repository. The automated process you already have in place handles the rest.

Learn more in the in-depth guide to:

Cloud Native and DevOps

“Cloud native” is a way to build and run applications that take advantage of the cloud computing delivery model. Cloud native is not about where you deploy your application, but about how applications are built and deployed. Cloud native applications can live both in the public cloud and on-premises, assuming that the local data center has cloud automation capabilities.

DevOps is one of the primary use cases for cloud native techniques. The DevOps approach naturally complements cloud native concepts like containerization, serverless, and microservices architectures:

  • Containerization and serverless frameworks make applications environment independent, eliminating conflicts between developers and operations teams, and improving collaboration between developers and testers.
  • Microservices architectures split large applications into smaller, functional elements, each of which can be iteratively developed and maintained by a DevOps team. This improves agility, reliability, and makes collaboration easier, because each element in a microservices architecture is simple and well understood.

DevOps and Kubernetes

Most if not all DevOps teams are developing and running applications in containers. Orchestration engines like Kubernetes are required to keep containers running at scale.

Kubernetes helps DevOps teams meet customer needs without worrying about the details of the infrastructure. Kubernetes replaces the old manual task of deploying, scaling, and building a resilient application. Instead, it dynamically provisions applications on available resources.

Kubernetes is essential for DevOps teams looking to automatically scale and ensure resiliency of applications, while minimizing the operations burden. For example, it allows teams to manage the scalability and elasticity of an application based on load metrics. Developers can focus on building new functionality, without worrying about whether the application can serve users during peak times.

However, Kubernetes makes certain aspects of the IT environment more complex. These include security, storage management, and management of CI/CD pipelines.

Learn more in these detailed guides:

Why is Kubernetes Essential for Modern DevOps Teams?

  1. Portability—Kubernetes makes it possible to deploy applications anywhere without tight coupling to infrastructure. Making it easier to replicate dev, test and production environments.
  2. Infrastructure as code—everything in Kubernetes works in an IaC paradigm, so both infrastructure and applications are fully declarative and can be provisioned automatically.
  3. Hybrid—Kubernetes can run locally, in any public cloud, or at the edge, providing additional flexibility for DevOps teams.
  4. Open—Kubernetes is an open source platform, supported by a large ecosystem of innovative services and tools.
  5. Deployments with no downtime—Kubernetes provides multiple deployment strategies, enabling DevOps teams to test and conduct experiments in production, for example, using blue/green or canary deployments.
  6. Immutability—containers can be stopped, deleted or re-deployed with minimal impact to the application. Servers become “cattle, not pets”.

Learn more in these detailed guides:

4 Kubernetes Deployment Strategies for DevOps Teams

Here are several ways DevOps teams can leverage Kubernetes to deploy dev, test, and production environments.

  1. Recreate deployment
    All replicas of an existing deployment are deleted and replaced with new ones. There is some downtime between shutdown and restart of each container. This method is suitable for infrequently used applications and for applications that users do not want need 24/7 availability. 
  1. Rolling update
    By default, Kubernetes is updated in stages. After running the deploy command, Kubernetes starts replacing existing containers with new updates, deploying each new container at a time. This can be used both for software updates, and for rolling back an update and reverting to a previous version.
  1. Blue-green deployment
    Blue-green deployments are not built into Kubernetes, but are easy to set up. The "blue" copy is the existing version and is replaced by the "green" copy which has a new software version.

    To achieve blue-green deployment, first create a deployment, and roll out green replicas alongside the existing blue ones. While the green replicas are being deployed, the system will use additional resources, but this is only temporary. After deploying and testing green replicas, you need to route traffic from the "blue" to the "green" replicas. You can do that by using an external load balancer. Linkerd and similar tools can help you define how much traffic you want to route to the blue vs. the green deployment.
  2. Canary Releases
    This technique enables you to deploy a basic new version of the application for a fraction of its users. For example, 10% of users get the new version while the rest continue seeing the existing one. Once the version is tested on the initial 10%, it is released to a larger subset, and eventually pushed to the entire user base.

    The main advantage of a canary release is that you can test your applications on real users in a production environment. However, to ensure the release maintains a positive user experience, you need to carefully plan in advance.

DevOps and Serverless

Serverless computing is an innovative way to deliver backend services. Serverless architecture allows users to write and deploy code without worrying about the infrastructure. There are still physical servers, but developers do not know about or interact with them.

Serverless architecture is commonly used to implement DevOps. Serverless makes it possible to build an efficient CI/CD pipeline, simply by declaring application requirements, with an on-demand pricing model. DevOps teams can implement an entire build, test, and deployment pipeline just by writing code and deploying it as serverless functions, with no hosted solutions.

Another advantage of serverless technology is that it makes updates easier. DevOps teams can introduce new versions of serverless services, while keeping existing instances running, and switch between services very easily. This makes it even easier to leverage canary and blue/green deployments, compared to a containerized approach.

Learn more in the in-depth guides:

The DevOps Toolset

DevOps teams see tooling as a means, not an end. DevOps unifies teams by automating and tracking functional processes from initial check-in of code to a repository, all the way to production deployment. To fully improve these processes, all stages of DevOps pipeline must be automated, controlled and monitored.

DevOps tools enable automation and control for planning, development, testing, deployment, operations and monitoring. In addition, some tools have a view of the entire DevOps pipeline, and can help orchestrate the entire process. 

DevOps Tools Map

A wide range of tools are available for DevOps implementations and new tools are consistently being developed. Below are the most common types of tools that are included by DevOps teams and some examples of the specific tools in use. 

Function

Examples of Tools

Automated deployment 
Automate deployment to staging or production environments.

Spinnaker
FluxCD
Flagger

Public cloud platforms
Provide scalable computing resources with rich automation capabilities.

Amazon Web Services 
Microsoft Azure 
Google Cloud Platform 
DigitalOcean 
IBM Cloud

Containerization
Enables you to deploy services in a consistent way on any platform, with orchestration tools to manage, scale, and deploy a large number of containerized resources.

Docker
Kubernetes

Collaboration
Enable teams to communicate on tasks transparently with full accountability.

Slack
JIRA

Infrastructure as code
Automates system configurations and standardizes resource provisioning.

Puppet
Chef
Ansible
Terraform
CloudFormation
Azure ARM

Logging
Enables you to collect and analyze event and operational data for troubleshooting and optimization.

Fluentd
Logstash
Filebit
Elasticsearch
Splunk 

Logging, monitoring and alerting
Enables collecting and analyzing operational data, optimizing performance, and responding to production issues. 

Logstash
Elasticsearch
Splunk 
Prometheus
Sensu
Datadog
NewRelic
Dynatrace
AppDynamics
PagerDuty

Security
Scans and secures applications and resources to remediate vulnerabilities and prevent cyber attacks. 

Snyk
WhiteSource
Snort
Veracode
Sonatype

CI/CD systems
Enables automatically building and testing code in a CI/CD pipeline.

Codefresh
Jenkins
Gitlab
CircleCI
TravisCI

Application Mapping

Application mapping is a process that involves identifying and documenting the interactions between various software applications within an organization. 

AppDynamics
Dynatrace
New Relic
Faddom
 

Documentation tools

These tools can create, manage, and update technical documentation, including system overviews, developer guides, API documentation, and more.

Confluence
ReadTheDocs
Swimm
Sphinx

Learn more about DevOps tools in our in-depth guides to:

DevOps Storage Automation with NetApp Cloud Volumes ONTAP

NetApp Cloud Volumes ONTAP, the leading enterprise-grade storage management solution, delivers secure, proven storage management services on AWS, Azure and Google Cloud. Cloud Volumes ONTAP supports up to a capacity of 2 PB, and supports various use cases such as file services, databases, DevOps or any other enterprise workload, with a strong set of features including high availability, data protection, storage efficiencies, Kubernetes integration, and more.

In particular, Cloud Volumes ONTAP provides Cloud Manager, a UI and APIs for management, automation and orchestration, supporting hybrid & multi-cloud architectures, and letting you treat pools of storage as one more element in your Infrastructure as Code setup.

Cloud Manager is completely API driven and is highly geared towards automating cloud operations. Cloud Volumes ONTAP and Cloud Manager deployment through infrastructure- as- code automation helps to address the DevOps challenges faced by organizations when it comes to configuring enterprise cloud storage solutions. When implementing infrastructure as code, Cloud Volumes ONTAP and Cloud Manager go hand in hand with Terraform to achieve the level of efficiency expected in large scale cloud storage deployments.

See Additional Guides About Key DevOps Topics

NetApp, together with several partner websites, has authored a large repository of content that can help you learn about many aspects of DevOps. Check out the articles below for objective, concise reviews of key DevOps topics.

Infrastructure as Code on AWS

Authored by NetApp


Infrastructure as Code on Azure

Authored by NetApp


kubectl Cheat Sheet

Authored by Komodor


Azure DevOps

Authored by Codefresh


Continuous Delivery

Authored by Codefresh


Continuous Integration

Authored by Cofefresh


GitOps

Authored by Cofefresh


Infrastructure as Code

Authored by Cofefresh


Software Documentation

Authored by Swimm


Configuration Management

Authored by Configu


Feature Flags

Authored by Configu


Application Mapping

Authored by Faddom


AWS Cost

Authored by Granulate


Azure Cost

Authored by Granulate


Kubernetes Architecture

Authored by Granulate


Documentation Tools

Authored by Swimm


 

Additional DevOps Resources

See additional guides on DevOps topics authored by our partner websites.