Go plug-in system via RPC

Go-plugin is an RPC-based go (Golang) plug-in system. It is a plug-in system that has been used in the HashiCorp tool for over four years. Although originally created for Packer, it has also been used by Terraform, Nomad, Vault and Boundary.

Although the plug-in system is over RPC, it is currently only designed to work on local [reliable] networks. Plugins on the real network are not supported and can cause unexpected behavior.

The plug-in system has been used on millions of machines in many different projects and is proven to be battle-hardened and ready for production.

The characteristics of

HashiCorp’s plug-in system supports a range of functions.

A plug-in is an implementation of the Go interface. This makes writing and using plug-ins feel natural. For a plugin author: you just implement an interface as if it were running in the same process. For a plug-in user: you simply use and call the functions of an interface as if it were in the same process. This plug-in system handles the intermediate communication.

Cross-language support. Plug-ins can be written (and consumed) for almost all major languages. This library supports serving plug-ins through gRPC. Grpc-based plug-ins allow plug-ins to be written in any language.

Support for complex parameters and return values. This library provides apis for handling complex arguments and return values, such as interfaces, IO.Reader/Writer, and so on. We do this by giving you a library (MuxBroker) that creates new connections between clients/servers to serve additional interfaces or transfer raw data.

Two-way communication. Because the plug-in system supports complex parameters, the host process can send it interface implementations, and the plug-in can call back to the host process.

Built-in logging. Any plug-in that uses the log library will have log data automatically sent to the host process. The host process mirrors the output, prefixed with the path to the plug-in binary. This makes debugging the plug-in easy. If the host system uses HCLOG, the log data will be structured. If the plug-in also uses HCLOG, the logs from the plug-in are sent to the host HCLOG and structured.

Protocol version. Support for a very basic “protocol version” that can be incremented to invalidate any previous plug-in. This is useful when interface signatures change, protocol level changes are necessary, and so on. If the protocol version is incompatible, a user-friendly error message is displayed to the end user.

* * Stdout/Stderr synchronization. ** When plug-ins are child processes, they can continue to use stdout/stderr as usual, and the output will be mirrored to the main process. The main process can control these flows to io.writer to prevent this from happening.

Save TTY. The plug-in child process is connected to the same STDIN file descriptor as the host process, enabling software that requires TTY to work. For example, a plug-in can perform SSH, even if multiple child processes and RPCS occur, and it looks perfect to the end user.

** Upgrade the host while the plug-in is running. ** plug-ins can be “reconnected” so that the host process can upgrade while the plug-in is still running. ReattachConfig This requires the host/plug-in to know that this is possible and daemonize correctly. NewClient to determine if and how to reconnect.

Cryptography security plug-ins. Plug-ins can be authenticated with expected validation, and RPC communication can be configured to use TLS. The host process must have appropriate security to protect this configuration.

architecture

HashiCorp’s plug-in system works by starting child processes and communicating via RPC (using standard NET/RPC or gRPC). There is a single connection between any plug-in and the host process. For net/ RPC-based plug-ins, we use a connection reuse library to reuse any of the other connections above. For grPC-based plug-ins, the HTTP2 protocol processes reuse.

There are many benefits to this architecture.

  • Plug-ins cannot crash your host process. Panic in plug-ins does not scare users of plug-ins.

  • Plug-ins are easy to write: just write a Go program and a Go build. Or writing a gRPC server in any other language requires only a small number of templates to support the Go-plugin.

  • The plug-in is very easy to install: just put the binaries where the host can find them (it depends on the host, but the library also provides helpers) and let the plug-in host handle the rest.

  • Plug-ins can be relatively secure. A plug-in can only access its interface and ARGS, not the entire memory space of a process. In addition, the Go-Plugin can communicate with the plug-in over TLS.

Method of use

To use the plug-in system, you must take the following steps. These are the advanced steps that must be done. Examples can be found in the examples/ directory.

  1. Select the interface you want to expose for your plug-in.

  2. For each interface, implement an implementation of that interface, communicating either through a NET/RPC connection or through a gRPC connection or both. You have to implement both a client-side and a server-side implementation.

  3. Create a Plugin implementation that knows how to create an RPC client/server for a given plug-in type.

  4. The plug-in author calls plugin.serve to provide a plug-in from the main function.

  5. The plug-in user uses plugin.client to start a child process and request an interface implementation through RPC.

Here it is! In practice, step 2 is the most tedious and time-consuming step. Even so, it’s not too difficult, and you can see some examples in the examples/ directory as well as in our various open source projects.

For complete API documentation, see GoDoc.

The roadmap

Our plugin system is constantly evolving. When we use the plug-in system for new projects or new features on existing projects, we constantly find improvements we can make.

At this point in time, the roadmap for the plug-in system is.

** Semantic versioning. ** Plug-ins will be able to implement semantic versions. This plug-in system will provide a constrained version of the system to the host process. This complements the protocol versioning that already exists, which is more for larger underlying changes.

What about shared libraries?

When we started using plug-ins (late 2012, early 2013), plug-ins via RPC were the only option, as Go did not support dynamic library loading. Today, Go supports the plug-in standard library, but with some limitations. Our plug-in system has stabilized from tens of millions of users since 2012 and has many benefits that we value very much.

For example, we use this plug-in system in Vault, where dynamic library loading is not acceptable for security reasons. This is an extreme example, but we believe our library system has more advantages than disadvantages compared to dynamic library loading, and since we’ve built and tested it for years, we’ll continue to use it.

Shared libraries have one major advantage over our system, and that is higher performance. In the practical application of our various tools, we have never asked for higher performance from our plug-in system, and its throughput is very high, so this is not an issue for us right now.