If you’re new to DevOps, check out these five steps to build your first DevOps pipeline.
DevOps has become the default way of addressing slowness, isolation, or other glitches in software development. But it doesn’t make much sense when you’re new to DevOps and not sure where to start. This article explores what a DevOps pipeline is and provides five steps to create it. Although this tutorial is not comprehensive, it will give you a foundation for getting started and expanding. First, insert a short story.
My Journey to DevOps
I used to work in Citigroup’s cloud group developing Infrastructure as a Service Web applications to manage Citi’s cloud Infrastructure, but I was often interested in studying how to make the development pipeline more efficient and how to bring a positive culture to the team. I found the answer in a book recommended by Greg Lavender. Greg Lavender is the CTO for Citi’s Cloud Architecture and infrastructure project, known as the Phoenix Project. The book reads like a novel, even though it explains DevOps.
A table at the back of the book shows how often different companies deploy to release environments:
| The company | The deployment of frequency |
|---|---|
| Amazon | 23000 times/day |
| 5500 times/day | |
| Netflix | 500 times/day |
| 1 time/day | |
| 3 times per week | |
| A typical enterprise | Once /9 months |
How do Amazon, Google and Netflix do it so often? That’s because these companies have figured out how to implement a near-perfect DevOps pipeline.
But that was not the case before DevOps at Citi. At the time, my team had environments with different build stages, but the deployment on the development server was very manual. All developers have access to only one development environment server based on IBM WebSphere Application Community Edition. The problem is that when multiple users try to deploy at the same time, the server goes down, so developers have to notify each other when they deploy, which can be painful. There are also problems with low code test coverage, cumbersome manual deployment, and the inability to track code deployment against defined tasks or user requirements.
I realised something had to be done and found a colleague who felt the same way. We decided to collaborate to build an initial DevOps pipeline — he set up a virtual machine and a Tomcat server, and I set up Jenkins, integrating Atlassian Jira, BitBucket, and code coverage testing. The side project was very successful: we almost fully automated the development pipeline and achieved almost 100% uptime on the development server, we were able to track and improve code coverage testing, and Git branches were able to be associated with deployment and JIRA tasks. In addition, most of the tools used to build DevOps are open source.
Now I realize how primitive our DevOps pipeline was because we didn’t take advantage of advanced Settings like Jenkins files or Ansible. However, this simple process works well, perhaps because of the Pareto principle (also known as the 80/20 rule).
A brief introduction to DevOps and CI/CD pipelining
If you ask someone, “What is DevOps?” “You might get a few different answers. DevOps, like Agile, has grown to encompass many different disciplines, but most people would agree on at least this: DevOps is a software development practice or software development lifecycle (SDLC), and its core principle is a cultural change — developers and non-developers breathe the same sky, and previously manual things become automated; Everyone did what they were good at; Simultaneous deployments become more frequent; Throughput improvement; Increased flexibility.
While having the right software tools isn’t the only thing you need to implement a DevOps environment, some are. One of the most critical is continuous integration and continuous deployment (CI/CD). In a pipelined environment, with different build phases (e.g. DEV, INT, TST, QA, UAT, STG, PROD), manual work can be automated and developers can achieve high-quality code with flexibility and mass deployment.
This article describes a five-step approach to building a DevOps pipeline, as shown in the figure below, using open source tools.
Without further ado, let’s get started.
Step 1: CI/CD framework
The first thing you need is a CI/CD tool. Jenkins is an open source CI/CD tool based on Java under the MIT license that serves as a tool for popularizing the DevOps movement and has become de facto standard.
So, what is Jenkins? Think of it as a magic universal remote control that works with many different servers and tools and arranges them all together. On its own, A CI/CD tool like Jenkins is useless, but becomes very powerful as you plug in different tools and servers.
Jenkins is just one of many open source CI/CD tools for building DevOps pipelining.
| The name of the | license |
|---|---|
| Jenkins | Creative Commons and MIT |
| Travis CI | MIT |
| CruiseControl | BSD |
| Buildbot | GPL |
| Apache Gump | Apache 2.0 |
| Cabie | GNU |
Here’s what DevOps looks like using the CI/CD tool.
Your CI/CD tools are running on localhost, but you can’t do anything else yet. Let’s follow the DevOps journey.
Step 2: Source control management
The best (and probably easiest) way to verify that A CI/CD tool can perform some magic is to integrate with a source control management (SCM) tool. Why do you need source control? Suppose you’re building an app. Whenever you build an application, whether you use Java, Python, C++, Go, Ruby, JavaScript, or any language, you are programming. The program code you write is called source code. In the beginning, especially if you’re working alone, it might be ok to put everything in a local folder. But as the project grows large and invites others to collaborate, you need a way to avoid merge conflicts when sharing code changes. You also need a way to restore a previous version — backup, copy and paste is outdated. You (and your team) want a better solution.
This is why SCM has become indispensable. SCM tools help with versioning and collaboration by keeping code in a repository.
Although there are many SCM tools out there, Git is the most standard and appropriate. I highly recommend using Git, but there are other open source tools out there if you like.
| The name of the | license |
|---|---|
| Git | GPLv2 & LGPL v2.1 |
| Subversion | Apache 2.0 |
| Concurrent Versions System (CVS) | GNU |
| Vesta | LGPL |
| Mercurial | GNU GPL v2+ |
With SCM, the DevOps pipeline looks something like this.
The CI/CD tool automates source check – in and – out and collaboration between members. Not bad, huh? But how do you turn it into a working application that billions of people can use and appreciate?
Step 3: Automate build tools
That’s great! Now you can check out code and commit changes to source control, and you can invite your friends to collaborate on source control. But so far you haven’t built the app. To make it a web application, it must be compiled and packaged as a deployable package or executable (note that interpreted programming languages like JavaScript or PHP don’t need to be compiled).
This leads to automated build tools. No matter which build tool you decide to use, they all have the same goal: build the source code into a desired format and automate cleaning, compiling, testing, and deploying to a location. Build tools will vary depending on your programming language, but here are some commonly used open source tools worth considering.
| The name of the | license | A programming language |
|---|---|---|
| Maven | Apache 2.0 | Java |
| Ant | Apache 2.0 | Java |
| Gradle | Apache 2.0 | Java |
| Bazel | Apache 2.0 | Java |
| Make | GNU | N/A |
| Grunt | MIT | JavaScript |
| Gulp | MIT | JavaScript |
| Buildr | Apache | Ruby |
| Rake | MIT | Ruby |
| A-A-P | GNU | Python |
| SCons | MIT | Python |
| BitBake | GPLv2 | Python |
| Cake | MIT | C# |
| ASDF | Expat (MIT) | LISP |
| Cabal | BSD | Haskell |
That’s great! Now you can put the automated build tool configuration file into source control and have your CI/CD tool build it.
Everything is so good, right? But where to deploy it?
Step 4: Web application server
So far, you have an executable or deployable package. For any truly useful application, it must provide some kind of service or interface, so you need a container to distribute your application.
For web applications, web application servers are containers. Application servers provide the environment in which programming logic in deployable packages can be detected, interfaces rendered, and web pages served to the outside world by opening sockets. In other environments you will also need an HTTP server (such as a virtual machine) to install the service application. Now, I assume you’ll learn these things on your own (although I’ll discuss containers below).
There are many open source Web application servers.
| The name of the | agreement | A programming language |
|---|---|---|
| Tomcat | Apache 2.0 | Java |
| Jetty | Apache 2.0 | Java |
| WildFly | GNU Lesser Public | Java |
| GlassFish | CDDL & GNU Less Public | Java |
| Django | 3-Clause BSD | Python |
| Tornado | Apache 2.0 | Python |
| Gunicorn | MIT | Python |
| Python Paste | MIT | Python |
| Rails | MIT | Ruby |
| Node.js | MIT | Javascript |
Now the DevOps pipeline is almost up and running. Good job!
Although you can stop here and integrate further, code quality is a very important thing for application developers.
Step 5: Code coverage test
Implementing code tests can be another troublesome requirement, but developers need to catch all bugs in their applications early and improve code quality to ensure end-user satisfaction. Fortunately, there are many open source tools available to test your code and suggest ways to improve quality. Even better, most CI/CD tools can integrate these tools and automate the testing process.
Code testing is divided into two parts: a Code Testing Framework to help write and run tests, and a Code Quality Improvement Tool to help improve code quality.
Code testing framework
| The name of the | license | A programming language |
|---|---|---|
| JUnit | Eclipse Public License | Java |
| EasyMock | Apache | Java |
| Mockito | MIT | Java |
| PowerMock | Apache 2.0 | Java |
| Pytest | MIT | Python |
| hypothesis | Mozilla | Python |
| Tox | MIT | Python |
Code quality improvement tools
| The name of the | license | A programming language |
|---|---|---|
| Cobertura | GNU | Java |
| CodeCover | Eclipse Public (EPL) | Java |
| Coverage.py | Apache 2.0 | Python |
| Emma | Common Public License | Java |
| JaCoCo | Eclipse Public License | Java |
| hypothesis | Mozilla | Python |
| Tox | MIT | Python |
| Jasmine | MIT | JavaScript |
| Karma | MIT | JavaScript |
| Mocha | MIT | JavaScript |
| Jest | MIT | JavaScript |
Note that most of the tools and frameworks mentioned earlier were written for Java, Python, JavaScript, because C++ and C# are proprietary programming languages (although GCC is open source).
Now that you’ve used the code coverage testing tool, your DevOps pipeline should look like the one shown in the beginning of the tutorial.
An optional step
The container
As I said earlier, you can publish your application on a VIRTUAL machine (VM) or server, but containers are a better solution.
What is a container? The brief introduction is that THE VM takes a lot of resources from the operating system, it increases the size of the application, and the container just needs some libraries and configuration to run the application. Obviously, VMS still have important uses, but containers are a much lighter solution for publishing applications, including application servers.
While there are other options for containers, Docker and Kubernetes are more widespread.
| The name of the | license |
|---|---|
| Docker | Apache 2.0 |
| Kubernetes | Apache 2.0 |
For more information, check out Opensource.com’s other articles on Docker and Kubernetes:
- What is a Docker?
- Docker profile
- What is Kubernetes?
- Kubernetes practice from scratch
Middleware automation tool
Most of our DevOps pipeline focuses on collaborative building and deployment of applications, but there are many other things you can do with DevOps tools. One of them is to use it to implement the Infrastructure Management Infrastructure as Code (IaC) tool, also known as the middleware automation tool. These tools help automate the installation, administration, and other tasks of the middleware. For example, automation tools can pull down applications, such as web servers, databases, and monitoring tools, with the correct configuration, and deploy them to the application server.
Here are a few open source middleware automation tools worth considering:
| The name of the | license |
|---|---|
| Ansible | GNU Public |
| SaltStack | Apache 2.0 |
| Chef | Apache 2.0 |
| Puppet | Apache or GPL |
For more middleware automation tools, check out other articles on Opensource.com:
- Ansible Quick Start guide
- Ansible automated deployment policies
- Top 5 Configuration management tools
Subsequent development
This is just the tip of the iceberg of a complete DevOps pipeline. Start with CI/CD tools and explore other things you can automate to make your team’s work easier. Also, finding open source communication tools can help your team work better together.
For more insights, here are some great articles on DevOps:
- What is the conversation
- 5 Things to Know to Become a DevOps Engineer
- DevOps for everyone
- Start using predictive analytics in DevOps
It’s also a good idea to use open source Agile tools to integrate DevOps:
- What is Agile?
- 4 Steps to becoming a Great Agile developer
Via: opensource.com/article/19/…
By Bryant Son, lujun9972
This article is originally compiled by LCTT and released in Linux China