This article tries to find similarities between automated configuration management tools like Puppet, Chef, and Ansible so that you don’t get lost in the shuffle. There will always be people creating concepts for different purposes to make the world more complicated.
This article is also suitable for people without operation and maintenance experience. Because I’m just a guy with no operational experience. Welcome to be corrected.
History with these three
I was introduced to automated operations relatively late, I only knew Puppet and automated operations a year ago (only know), and Chef and Ansible were even later. Before learning them, HOWEVER, I had no idea what operations would do. This is where my learning of Puppet, Chef and Ansible gets in the way. Because we do not know the position of these three tools in the field of operation and maintenance, which problems are solved in the process of operation and maintenance. My initial impression of these three tools is that with them, the operation of the server can be automated without me manually SSH on the server, then executing the installation software one by one, without SCP war package on the server, etc.
That first impression is my history with them. Why do you say this? It was this initial impression that made me think they had something in common. The so-called commonness is the existence of some common concepts or principles and so on. If I master these “things”, I can stand at a higher height to think. Puppet, Chef, Ansible are all tools, and for tools, commonality refers to the problems they solve in common.
But when I read a lot of articles still no results. So I decided to find out for myself what they had in common and write it down.
Create an automated operation and maintenance tool
But to find commonalities among many similar things, it seems, you need to be familiar with them all at the same time. But I don’t have that much time. So I chose another approach: visualizing myself implementing a set of automated operations tools. The problem is, I don’t even know what “automated operations tools” are supposed to do.
Let me set the bar lower and simplify: AUTOMATE my initial “impressions.” All I could think of was writing a bash script:
ssh ….
apt-get install -y java
apt-get -y nginx
scp ..
Ok, now make the problem harder: do the same thing on multiple servers. I can think of just putting all the server IP’s in an array and executing it with a for loop. The question is, what if I’ve already run a command on the server once and it might fail, and I want to run it a second time? At this point, we can add an if statement to our bash script so that if Java is installed, we don’t install it again.
Obviously, there are still many problems in reality, such as:
-
Should we write another bash script to solve the reverse configuration problem? If we take this approach, the number of bash scripts increases to the point where the new complexity of managing those scripts and their relationships becomes a new problem for us;
-
Operating system compatibility issues on the server: Our bash command varies depending on the operating system;
-
Software version upgrade or degradation issues and so on.
We could have bash every time we faced these problems, but that was never an option. Bash is too low-level for building an automated operations tool. Having said that, it should be a no-brainer to design a DSL. At this point, I think our direction is clear. Puppet, Chef, and Ansible all use different DSLS. This DSL needs to be compiled into something that the server can execute. What is executable? For now, let’s assume that this thing is a bash script.
But where does this DSL compile? On the controlled machine or on the master machine? So, I think all automated operations tools run into this problem. What are controlled machines and master machines? You understand that one machine only gives commands and the other machine only executes commands.
We just talked about designing DSLS. However, to design a complete automated operation tool, we should first consider how the master machine communicates with the controlled machine. This question made me feel powerless for a long time because I didn’t know what to do. It turns out that you can’t think about this alone. The way you communicate also depends on what DSL you design and how you compile it. At the same time, the problem of the execution results of the controlled machine also affects the design of the DSL.
We seem to forget that automated operations tools often deal with not one machine, but many machines. When faced with multiple machines, a new problem arises: how to organize them? Because different machines have different responsibilities, the corresponding configuration is also different.
I think we’ve seen the problems that automated operations tools have to solve:
1. How to communicate with controlled machines
2. How to organize hundreds of machines
3. Design and compilation of DSL
4. How to obtain the execution results
I’m not sure we’re on the same page as the writers of Puppet, Chef and Ansible. Not exactly right. But at least we have a general idea of the problems automated operations tools are trying to solve. Are they common to automated operations tools? I’m not sure.
But it doesn’t matter, let’s assume that these are the problems that all SCM tools solve, that they have in common, and let’s look at how they solve each of these problems.
The background of these three
Before we dive in, let’s take a look at their background:
Puppet
How to communicate with controlled machines
A Puppet Server is called a Puppet Master, and a Puppet Client is called a Puppet Agent. They communicate using HTTPS. Before installing Puppet, you need to set the hostname on the master machine and the master.
If you need to install the Puppet Master on your master machine, you need to install the Puppet Master on your master machine.
sudo apt-get install -y puppetmasterCopy the codeInstall Puppet Agent on a controlled machine:
sudo apt-get install -y puppetCopy the codeAfter the installation is complete, add server=
to the [main] node of the /etc/puppet-puppet. conf file and run sudo Puppet Agent -test to apply for authentication from the master machine. The main control machine performs sudo Puppet Cert Sign < Agent’s hostname> authentication.
Such manual certification is not always adopted in a production environment.
Design and compilation of DSL 1
Puppet’s DSL is called Manifest and has a.pp file extension. Like other programming languages, it also has a program’s entrance, default on it: / etc/puppet/manifests/site. The pp:
#vim /etc/puppet/manifests/site.pp
node 'mysqlserver.example.com' {
$mysql_password = 123456
package {
['mysql-common', 'mysql-client', 'mysql-server']:
ensure => present
}
service {
'mysql':
enable => true,
ensure => running,
require => Package['mysql-server']
}
exec {
'set-root-password':
path => "/usr/bin:/usr/sbin:/bin",
subscribe => [Package['mysql-common'], Package['mysql-client'], Package['mysql-server']],
refreshonly => true,
unless => "mysqladmin -uroot -p$mysql_password status",
command => "mysqladmin -uroot password $mysql_password",
}
file {
'/etc/mysql/my.cnf':
content => template('my.cnf.erb'),
require => Package['mysql-server'],
notify => Service['mysql']
}
}
Copy the code
The basic format of Manifest:
node NODENAME{ RESOURCE { NAME: ATTRIBUTE => VALUE, ... }}Copy the codeNodename is essentially the hostname of the node. Here’s a question: Why not just use IP? Resource stands for Resource, which means Puppet treats everything in a node as a Resource. We’ll get to the details later.
How do you organize hundreds of machines
If I want to control 1000 machines does that mean I need to write 1000 nodes in site.pp node NODENAME{… }? The answer is yes.
DSL design and compilation ii
Note that everything contained in the ‘node’ node essentially describes the state of the node, such as:
service { 'mysql':
enable => true,
ensure => running
}Copy the codeThe state of the ‘service mysql’ Resource is available and running. Puppet has a number of resources built into it, such as file, exec, package, etc. But what happens when Puppet doesn’t have enough resources built in? Therefore, it should support resource extensions. By analogy, all automated operations tools should support such extensions.
Puppet also provides a template mechanism. Puppet’s template pattern uses Ruby’s ERB. You can use ‘template(‘my.cnf.erb’) ‘in puppet code after placing’.erb ‘files in’ /etc/puppet.templates’ :
#vim /etc/puppet/templates/my.cnf.erb [mysql] ... .Copy the codeIf we have templates, how can we get fewer variables? $VAR_NAME = VALUE Wherever there is a variable, the concept of scope is always referenced, whether it is static or global. Puppet is static scoped. Guess what level of scope Puppet’s variables have? Puppet, Chef, Ansible variables are static scoped. How many levels of scope do they have? One more question: is this common?
We now know the Manifest and its basic format, node, resource concepts, templates, and variables. At the same time, we should not write everything in a site.pp file, just as we should not write all the logic in C’s main function. So how does Puppet’s DSL solve this problem: how does it make it easier for developers to organize code?
Now let’s look at some relative modular Puppet code: # vim/etc/Puppet/manifests/site. Pp
class mysql($mysql_password = "123456") {
package {
['mysql-common', 'mysql-client', 'mysql-server']:
ensure => present
}
service {
'mysql':
enable => true,
ensure => running,
require => Package['mysql-server']
}
exec {
'set-root-password':
path => "/usr/bin:/usr/sbin:/bin",
subscribe => [Package['mysql-common'], Package['mysql-client'], Package['mysql-server']],
refreshonly => true,
unless => "mysqladmin -uroot -p$mysql_password status",
command => "mysqladmin -uroot password $mysql_password",
}
file {
'/etc/mysql/my.cnf':
content => template('my.cnf.erb'),
require => Package['mysql-server'],
notify => Service['mysql']
}
}
node 'agent' {
include mysql
class { 'mysql':
mysql_password => '456789'
}
}Copy the codeIn this version of site.pp we use the concept of class. You can define a class and then apply the same logic to it. Finally, quote it elsewhere. However, when referring to a class, it depends on whether the class has an argument or not. This will affect how the class is used.
But even so, the maintainability of our site.pp code is not high. So Puppet also provides a module concept. In fact, puppet is just another way of saying to remove the class from site.pp. Let’s look at the third version using Module:
Mysql class = mysqlmodule;
#vim /etc/puppet/modules/mysql/manifests/init.pp
class mysql($mysql_password = "123456") {
package {
['mysql-common', 'mysql-client', 'mysql-server']:
ensure => present
}
service {
'mysql':
enable => true,
ensure => running,
require => Package['mysql-server']
}
exec {
'set-root-password':
path => "/usr/bin:/usr/sbin:/bin",
subscribe => [Package['mysql-common'], Package['mysql-client'], Package['mysql-server']],
refreshonly => true,
unless => "mysqladmin -uroot -p$mysql_password status",
command => "mysqladmin -uroot password $mysql_password",
}
file {
'/etc/mysql/my.cnf':
content => template('mysql/my.cnf.erb'),
require => Package['mysql-server'],
notify => Service['mysql']
}
}Copy the codeBefore using Module, our file structure looked like this:
| - auth. Conf | - environments | - files | - manifests | ` -- site. Pp | -- puppet. Conf | - templates | `. - my CNF. ErbCopy the codeAfter using module:
|-- auth.conf
|-- environments
|-- files
|-- manifests
| `-- site.pp
|-- modules
| `-- mysql
| |-- manifests
| | `-- init.pp
| `-- templates
| `-- my.cnf.erb
|-- puppet.conf
`-- templatesCopy the codeModule is defined, so how to use it?
#vim /etc/puppet/manifests/site.pp
node 'agent' {
include mysql
}Copy the codeAt this point, I’m sure you have a rough idea of how to write Puppet, but I don’t think it’s enough. I don’t understand why it’s designed this way. Why we are not:
Node 'agent_hostname'{install package [' mysql', 'mysqlServer'] start service 'mysql' create template mysql.cnf}Copy the codeI mean why noun-oriented descriptive language, not verb-oriented. Then I went to Configuration Management Best Practices. Disillusionment, the original configuration management is not a tool or two can be done. It is a system consisting of six core functions:
1. Source control
2. Construction engineering
3. Configure the environment
4. Change control
5. Release engineering
Deployment of 6.
So, I never understood the difference between automated operations and automated configuration management. I also read that configuration management is really state management, whether it’s server state or software state.
That makes sense. No wonder Puppet, Chef, Ansible’s DSLS are descriptive languages.
But how and where does Puppet compile our manifest? After the master machine and the controlled machine are authenticated successfully, the controlled machine will send a request to the master machine at regular intervals. This request will tell the master machine about itself (the controlled machine). The master machine takes this information, links it to the manifest and compiles it, and finally generates a catalog that the master machine (puppet client) can execute. In the process of execution, the controlled machine feeds back the execution status to the master host. This is how the master machine in Puppet gets command execution results from the master machine.
At this point, we see that Puppet has answered all four of our previous questions.
summary
1. How to communicate with controlled machines
The AGENT applies for a certificate from the Master using THE C/S architecture and HTTPS.
2. How to organize hundreds of machines
Use the ‘node’ keyword definition in the manifest.
3. Design and compilation of DSL
* How you organize your code
Puppet defines nodes (controlled nodes) in the manifest file and abstracts all node components into resource attributes. A node can contain multiple resources, which together constitute the state of the node. However, it is not possible to put all resources in one file, and a manifest file usually has more than one node. So, Puppet provides a module and class mechanism that allows you to bundle together resources that share the same responsibility. What’s the difference between class and Module? Classes can be written directly to the manifest file, while Modules must create a new directory structure. This is how Puppet organizes code.
Learn more: If you deal with relationships between resources, they are likely to have dependencies. The same applies to classes and modules. What about if else and for?
$VAR_NAME = VALUE
Learn more: Understand the scope of variables
* Templates: Use Ruby erB files
4. How to obtain the execution results
The controlled machine actively sends the execution results to the master machine.
Does it really have to have a master to work? Isn’t. Puppet provides a way to use the standalone version. For details, see Google: Puppet Apply.
Chef
Chef means Chef in Chinese. So it regards all controlled machines as “dishes”. But it won’t cook for us unless we tell it the Cookbook. What does it say on the menu? It’s a Recipe. So, we wrote recipe one by one into the Cookbook and gave it to Chef at last.
Chef is also a C/S architecture, and C and S also use HTTPS to communicate. Similarly, because of this, we might reuse the pattern of learning Puppet to learn Chef. However, the return on investment of Chef’s C/S model was too low, so I gave up after sticking to it for a while. Like Puppet, Chef offers a standalone version: chef-solo.
Ansible
Ansible says agentles (go to the client). But in practice, it requires SSH and Python on the controlled machine, with the Python-SimpleJSON package installed. In fact, most of our current machines have these installed by default. So, when using Ansible, you don’t need a special machine as a master server. Any machine can become a master machine whenever you want.
See the documentation for installing Ansible. Unlike Chef and Puppet, that part of the Ansible organization’s controlled machine logic is pulled separately, called Inventory. It is an INI file, such as hosts:
[web] 192.168.33.10 192.168.33.11 [db]Copy the codeThe file name and path are arbitrary, but it is recommended to use an indicative name and an appropriate path.
Puppet and Chef both made their own DSLS and wrote their own compilers, but Ansible uses the YAML format. I think this is a very clever design: one is that everyone is familiar with the YAML format rather than a custom DSL, two is that you don’t have to design your own DSL, and three is that you don’t have to write your own compiler. So, in my personal learning process, I found it was much easier than Puppet and Chef.
Once you understand the YAML file format, it’s time to understand the Ansible metaphor. Ansible is the director, understanding all controlled machines as actors. We developers are the writers. As long as we write the playbook, Ansible takes the script and matches it with Invenstory, the actors will only play according to the truth of the script, without any personal play.
Let’s write the first version of playbook.yml (path and name can be customized) :
---
- hosts: web
tasks:
- name: install nginx
apt: name=nginx state=latest
- hosts: db
vars:
mysql_password: '123465'
tasks:
- name: install mysql
yum: name={{item}}
with_items:
- 'mysql-common'
- 'mysql-client'
- 'mysql-server'
- name: configurate mysql-server
template: src=my.cnf.j2 dest=/etc/mysql/my.cnf
- name: start service
service: name=mysql state=startedCopy the code#vim templates/my.cnf.j2
[mysql]
...
passowrd={{mysql_password}}Copy the codeOur script includes two actors: Web and DB. Which controlled machines do they correspond to? Just look at the Invenstory documentation. So what do these actors have to do? If you look at tasks, it follows a list. Things like ‘yum’, ‘apt’, ‘template’, and ‘service’ are called modules, similar to Puppet’s resource and Chef’s recipe. Ansible itself provides a number of modules, but it’s a no-brainer and can’t meet every project’s needs, so you can develop your own.
Similarly, Ansible provides a variable and template (Jinja2) mechanism. What are the levels of scope for Ansible?
Similarly, we can’t tolerate all tasks in one file. How does Ansible organize code? Ansible introduces the concept of role. Yes, tasks with common responsibilities are grouped into the same role. Therefore, our file structure has also changed, from the original only two text files, now we need to create a new directory:
├─ all exercises, ├─ all exercises, all exercises, all exercises, all exercises, all exercises, all exercises, all exercises, all exercisesCopy the codeAt this point, playbook.yml:
---
- hosts: web
tasks:
- name: install nginx
apt: name=nginx state=latest
- hosts: db
var:
mysql_password: '123456'
roles:
- mysqlCopy the codeAnd the main. Yml:
- name: install mysql
yum: name={{item}}
with_items:
- 'mysql-common'
- 'mysql-client'
- 'mysql-server'
- name: configurate mysql-server
template: src=my.cnf.j2 dest=/etc/mysql/my.cnf
- name: start service
service: name=mysql state=startedCopy the codeThe relationship between tasks and roles. What about relationships between tasks, relationships between roles?
This is how Ansible’s DSL organizes code. The last question is how do you get the results? Ansible compiles local YML files into Python code and sends it to a controlled machine, which returns the results in Json format.
Getting started with Ansible is simple.
summary
1. How to communicate with controlled machines
As long as both the master machine and the controlled machine will have SSH
2. How to organize hundreds of machines
Managed with Invenstory
3. Design and compilation of DSL
* How you organize your code
The entry point for Ansible Language is playBook. You can add ‘Tasks’ directly to playBook. Naturally, when we think of tasks, we think of small tasks. This is true, but in Ansible, it’s called module. However, we don’t want all tasks to be in the same file, where Ansible’s role mechanism comes into play. You can put groups of people who perform the same tasks into a single role.
Further study: dependencies between modules, if-else problems
* Variable definition: Different levels have different definitions
Learn more: Understand the scope of variables
* Template: Use Jinja2
4. How to obtain the execution results
The controlled machine actively sends the execution results to the master machine.
conclusion
We programmers are constantly learning new programming languages, frameworks, and concepts. Admittedly, I assume that we all love learning, but we need to ask whether the learning is in the domain of solution or the domain of problem. So, when something new comes in, I always ask, what problem does it solve? Why does it work? On what basis? What we need is the nature of the problem and the model of how to solve it, not someone else’s medicine.
That far. In fact, there are many other automated configuration management tools out there besides the three mentioned above. In general, I think we can use the above ideas to learn. Back to our original question: What do Puppet, Chef, and Ansible have in common? Can I say that those four questions are the common ones? The answer is I don’t know.
Finally, I declare that I have not applied The Idea to any other tool than the one above. I hope you can give me feedback. thank you