On the topic of Python automation, in the last article I introduced the Invoke library, which is one of the most important components of Fabric. Fabric is also a widely used automation tool library and an automated manipulator to mention, so this article will cover it in the future.

Fabric is used to automate tasks such as application deployment and system management. It is simple and lightweight, and provides abundant SSH extension interfaces. In the Fabric 1.x version, it mixes local and remote functionality; But since Fabric 2.x, it has separated the Invoke library to handle local automation tasks, while Fabric focuses on remote and network-level tasks.

To do this, Fabric relies heavily on another core component, Paramiko, which is an SSH-based remote control module on which Fabric encapsulates a more user-friendly interface, Shell commands can be executed remotely, files can be transferred, servers can be operated in batches, identity can be authenticated, proxies can be configured and set up, and so on.

Fabric version differentiation

Python 2 was officially “retired” on New Year’s Day this year, and will only be Python 3 in the future. Many projects, including Fabric, the subject of this article, must also ship their own new versions (compatible or Python 3 only) in order to accommodate non-compatible migrations of Python versions.

Fabric itself exists in two large versions: Fabric 1 and Fabric2, and on top of this library there are two related libraries that are easily confused: Fabric2 and Fabric3 (note that the numbers here are part of the library name).

They are distinguished as follows:

  • Fabric 1.x: Python 2.5-2.7 is supported, but Python 3 is not
  • Fabric 2.x: Python 2.7 and 3.4+ are supported, but fabfiles for Fabric 1.x are not compatible
  • Fabric2: equivalent to Fabric 2.x, in order for different versions to coexist (install an old version of 1.x and then install it as the new version)
  • Fabric3: a fork (unofficial) based on Fabric1.x, compatible with Python 2&3, compatible with Fabric1.x fabfiles

In summary, we recommend using the official Fabric 2.x series version, but be aware that some outdated tutorials may be based on earlier versions (or unofficial Fabric3, also based on Fabric 1.x) and need to be identified.

For example, in the Fabric 1.x series, write an import like this: from fabric.api import run; In the new version, an error will appear: “ImportError: No module named API “(PS: the version of fabric can be determined by the presence or absence of fabric. API, just as in Python the version can be determined by the print statement or print function). In addition, the new version does not support the old version of fabfile, so error message “No idea what ‘XXX’ is!

Fabric 2 is a non-compatible version with major improvements over the previous version:

  • Support for Python 2.7 and 3.4+
  • Thread-safe, eliminates the concurrent implementation of multiple processes
  • API around the fabric. The connection. The connection is reorganized
  • Overworked the command line parser to allow regular GNU/ POSIX-style flags and options on a per-task basis (fab MyTask :weird = Custom,arg = format is no longer required)
  • You can declare pre-tasks and post-tasks
  • … (The official list of more than 10 [1], not a list in this article)

Invoke, which was introduced earlier, was separated during the development of Fabric 2 for a specific reason in this answer [2]. In summary, versioning differences should be considered when using Fabric.

Basic usage of Fabric

1, install,

The first is installation: PIP Intall Fabric. After installation, version information can be viewed in the command line window:

>>> fab -V
Fabric 2.5.0
Paramiko 2.7.1
Invoke 1.4.0Copy the code

After executing “fab-v”, you can see that I have installed Fabric 2.5.0, and you can also see the version information of its two core dependency libraries, Paramiko and Invoke.

2. A simple example

Fabric is primarily used for remote tasks, that is, to operate on a remote server. Here is a simple example:

# Fabric import Connection host_ip = '47.xx.xx.xx' # server address user_name = 'root' # server username password = '****' Con = Connection(host_ip, user_name, connect_kwargs={'password': password}) result = con.run(cmd, hide=True) print(result)Copy the code

Log in to the remote server using the account + password, and then run the date command to check the server time.

Command exited with status 0.
=== stdout ===
Fri Feb 14 15:33:05 CST 2020

(no stderr)Copy the code

In addition to the server time, the results are now printed with some extraneous information. This is because the “result” it prints is a “fabric.runners.Result” class, which we can parse out:

print(result.stdout)  # Fri Feb 14 15:33:05 CST 2020
print(result.exited)  # 0
print(result.ok)      # True
print(result.failed)  # False
print(result.command) # date
print(result.connection.host) # 47.xx.xx.xxCopy the code

The code above uses the Connection class and its run() method to run shell commands on the connected server. If administrator privileges are required, replace them with the sudo() method. If you want to execute shell commands locally, replace the local() method.

In addition, there are get(), put() and other methods, as described below.

Command line usage

The above code can be written in any.py script and then run, or slightly wrapped and imported into another script.

In addition, Fabric is a command-line tool that performs tasks through fab commands. Let’s modify the above example’s code slightly:

# file name: Fabfile.py from fabric import Connection from Fabric Import Task host_ip = '47.xx.xx.xx' # server address user_name = 'root' # CMD = 'date' # CMD = 'date' # shell @task def test(c): """ Get date from remote host. """ con = Connection(host_ip, user_name, connect_kwargs={'password': Password}) result = con.run(CMD, hide=True) print(result.stdout) #Copy the code

To explain, the main changes are:

  • Fabfile.py Filename: The script name of the entry code must use this name
  • @Task decorator: This decorator needs to be imported from fabric. It is a wrapper around the @Task decorator for Invoke and is used in the same way as Invoke (note: It also requires the context parameter “c”, but it is not actually used in the code block, instead an instance of the Connection class is used.)

You can then view and execute the corresponding tasks in the command line window of the same directory as the script:

>>> fab -l
Available tasks:
  test   Get date from remote host.

>>> fab test
Fri Feb 14 16:10:24 CST 2020Copy the code

Fab is an extended implementation of Invoke that inherits a lot of functionality, so when you execute “Fab –help”, you’ll see that many of the arguments and explanations are exactly the same compared to “inv –help” introduced earlier.

Fab adds several command line options (highlighted in blue) for the remote service scenario, including:

  • –prompt — for-login-password: prompts the program to enter the SSH login password on the command line.
  • –prompt-for-passphrase: prompts the program to enter the path of the SSH private key encryption file on the command line
  • -h or –hosts: specifies the host name to connect to
  • -i or –identity: specifies the private key file used for SSH connection
  • -s or –ssh-config: Specifies the SSH configuration file to load at run time

See the documentation [3] for more information about the command line interface to Fabric.

4. Interactive operation

An interactive prompt on a remote server asking for information such as a password or “yes” requires Fabric to listen and respond.

Here is a simple example. The Responder of invoke was introduced, initialized with a regular string and response information, and finally assigned to the Watchers parameter:

from invoke import Responder
from fabric import Connection
c = Connection('host')
sudopass = Responder(
     pattern=r'\[sudo\] password:',
     response='mypassword\n')
c.run('sudo whoami', pty=True, watchers=[sudopass])Copy the code

5. Transfer files

File transfers between local and server are common. Fabric encapsulates this nicely, with two methods available in the Connection class:

  • get(args, *kwargs) : pull remote files to local file systems or file-like objects
  • put(args, *kwargs) : push local files or file-like objects to remote file systems

In the case of an established connection, the example:

# (slightly) con. Get ('/opt / 123. TXT ', '123. TXT) con. The put (' test. TXT', '/ opt/test. TXT)Copy the code

The first argument refers to the source file to be transferred, and the second argument refers to the destination to be transferred, which can be specified as a filename or folder (the default path is used when empty or None is used) :

# (slightly) con. Get ('/opt / 123. TXT ', ') # is empty, use the default path con. The put (' test. TXT ', '/ opt/') # / opt/specified pathCopy the code

The default storage path for the get() method is os.getcwd and the default storage path for the put() method is the home directory.

6. Perform batch operations on servers

For batch operations on server clusters, the simplest way to do this is to use a for loop and then establish connections and perform operations one by one, like this:

for host in ('web1', 'web2', 'mac1'):
    result = Connection(host).run('uname -s')Copy the code

But sometimes, such schemes can be problematic:

  • If you have multiple clusters of different servers that need to perform different operations, you need to write many for loops
  • If you want to aggregate the results of each set of operations (e.g. dictionary form, key-host, value-result), you have to add additional operations outside of the for loop
  • The for loop executes sequentially and synchronously, which is inefficient and lacks exception handling (exceptions in the middle cause subsequent operations to skip).

Fabric addresses these issues by introducing the concept of a Group, which defines a Group of hosts as a Group, using the same API method as Connection, that is, a Group can be treated simply as a Connection.

Then, the developer simply manipulates the Group and ends up with a result set, reducing his own work on exception handling and execution order.

Fabric provides a base class, fabric.group. group, from which two subclasses are derived, the differences being:

  • SerialGroup(hosts, *kwargs) : Perform operations serially
  • ThreadingGroup(hosts, *kwargs) : Perform operations concurrently

The type of Group determines how the host cluster operates, and we just need to make a choice. The result of their execution is then a class fabric.groupresult, which is a dict subclass that stores the correspondence between each host connection and its execution result.

>>> from fabric import SerialGroup
>>> results = SerialGroup('web1', 'web2', 'mac1').run('uname -s')
>>> print(results)
<GroupResult: {
    <Connection 'web1'>: <CommandResult 'uname -s'>,
    <Connection 'web2'>: <CommandResult 'uname -s'>,
    <Connection 'mac1'>: <CommandResult 'uname -s'>,
}>Copy the code

In addition, GroupResult provides failed and Succeeded properties to extract a subset of failures/successes. As a result, secondary operations can also be carried out in batches easily. The original

Advanced usage of Fabric

1. Identity authentication

Fabric uses the SSH protocol to establish remote sessions. It is a relatively secure application-layer encrypted transport protocol.

Basically, there are two levels of security authentication:

  • Password-based authentication: Using an account and password to log in to a remote host is less secure and vulnerable to man-in-the-middle attacks
  • Key-based authentication: Key pair authentication (public key for the server and private key for the client) protects against man-in-the-middle attacks but takes a long time to log in

In the previous example, we used the first method, which is to use a password to log in by specifying the connect_kwargs.password parameter.

Fabric, of course, also supports the second approach. There are three ways to specify the path to the private key file, with the following priorities:

  • Find connect firstkwargs.keyFilename argument, used as a private key if found
  • Next, look for the –identify option used by the command line
  • By default, the value of ‘IdentityFile’ in the operating system ssh_config file is used

If the private key file itself is encrypted, the connect_kwargs. Passphrase parameter is required.

2. Configuration files

Fabric supports the separation of some parameter items from the business code by managing them through configuration files, such as the previously mentioned password and private key files, which can be written in the configuration file to avoid coupling with the code.

Fabric basically follows the Invoke configuration file system (nine layers are listed in the official documentation) and adds some SSH related configuration items. The supported file formats are. Yaml,. Yml,. Json, and. Py (in this order). Yaml (yML for short) is recommended.

Common configuration files are as follows:

  • System-level configuration file: /etc/fabric.yml
  • User-level configuration file: ~/.fabric.yml (Windows under C:Usersxxx)
  • Project level configuration file: /myproject/fabric.yml

The above files have a decreasing priority. Since MY machine is Windows, for convenience, I created a “.fabric.yml” file in the user directory with the following contents:

Yml user: root connect_kwargs: password: XXXX # Key_filename: # -your_key_fileCopy the code

We’ve removed the username and password from the fabfile, so we can delete these:

# file name: Fabfile.py from fabric import Connection from Fabric Import Task host_ip = '47.xx.xx.xx' # server address CMD = 'date' # shell @task def test(c): """ Get date from remote host. """ con = Connection(host_ip) result = con.run(cmd, hide=True) print(result.stdout)Copy the code

Then, on the command line, execute:

>>> fab test
Tue Feb 18 10:33:38 CST 2020Copy the code

Many parameters can also be set in the configuration file, as detailed in the documentation [4].

3. Network gateway

If the remote service is network isolated and cannot be directly accessed (on a different LAN), there is a need for a gateway/proxy/tunnel. This middle layer of machines is often referred to as jumpers or fortresses.

There are two gateway solutions in Fabric, corresponding to two options for the OpenSSH client:

  • ProxyJump: simple, low overhead, nested
  • ProxyCommand: Expensive, non-nested, more flexible

When creating a Connection object for a Fabric, you can apply both scenarios by specifying the gateway parameter:

The ProxyJump method is to nest a Connection in a Connection as the gateway of the former, the latter uses direct tcpIP of SSH protocol to open the Connection with the actual remote host for the former, and the latter can continue to nest its own gateway.

from fabric import Connection

c = Connection('internalhost', gateway=Connection('gatewayhost'))Copy the code

ProxyCommand means that the client uses SSH commands (similar to “SSH -w %h:%p gatewayhost”) locally to create a subprocess, which communicates with the server and can read the standard input and output.

This part of the implementation details in paramiko. Channel. The channel and paramiko. Proxy. ProxyCommand, in addition to the specified in the parameters, can also be defined in the Fabric support in the configuration file. For more details, please refer to the documentation [5].

Four, summary

Incompatible versions of Fabric have caused some division in the community, no doubt due to the introduction of Python 3, but there is reason to believe that the new version is better than the old one.

Much of what is written about Fabric online is out of date. This article provides a comprehensive overview of the latest official documentation to help you get started with Fabric.

After reading this article, I believe readers will be able to use it easily in a few minutes. If you have any questions, please contact me in the following ways.

————–

Public id: Python Cat

Headline number: Python Cat

Zhihu: Pea flower under the cat

Nuggets: Pea flower under the cat

————–

Related links:

Invoke the tutorial: https://mp.weixin.qq.com/s/up8lxuRhJQAXRzxkirF3nA

1, http://www.fabfile.org/upgrading.html#upgrading

2, http://www.pyinvoke.org/faq.html#invoke-split-from-fabric

3, http://docs.fabfile.org/en/2.5/cli.html

4, http://docs.fabfile.org/en/2.5/concepts/configuration.html

5, http://docs.fabfile.org/en/2.5/concepts/networking.html

The public account “Python Cat”, this serial quality articles, cat philosophy series, Python advanced series, good books recommended series, technical writing, quality English recommended and translation, etc., welcome to pay attention to oh.