I recently read a 2016 Google paper called “Design Patterns for Container-Based Distributed Systems,” which describes several ways containers can be used in container orcharding systems. That is, how to combine containers of different services and make these patterns best practices to make the entire distributed system more reliable and maintainable. Although it has been five years, the ideas in this article are still influencing the development of container technology and distributed systems.

The article divides design patterns into three broad categories:

  • How should a single container be implemented for container management
  • How can multiple containers coexist in a single node
  • Distributed algorithm in multi – node situation

If you are interested, please come to my official account: packy technology grocery store

Single container management mode

This pattern simply explains how a single container should interact with the upstream and downstream, and how the boundaries should be defined. More than once in this article, containers have been compared to objects in object-oriented thinking. A container should have the same boundary as an object. Provide interfaces to the upper layer to expose application information, such as monitored metrics (QPS, health information, etc.); A native life cycle should be defined for the underlying layer to make it easier for the management system to control the container and develop related components.

It’s easy to understand the upper level unified interface, as the readness probe and liveness probe monitoring services have been used in K8S. For the lower life cycle, the article gives an example: a cluster with multiple tasks executing, all of which have different priorities, some with higher priority, some with lower priority. When the cluster is running out of resources, it is necessary to let the higher-priority task execute first, but the lower-priority task is already executing. The system can’t kill the lower-priority task directly. At this point the lifecycle is required to specify how the container is stopped. In other words, the lifecycle makes it easier for the orchestration system to manage the container without mistakes. The lifecycle is a contract between the developer, the management system, and the container. In the above example, kiss deletion and ejection of low-priority tasks were performed in K8s. K8s first sent a SIGTERM signal to the container, indicating that you should stop operation immediately, save the saved items, close the files that should be closed, The SIGKILL signal is sent after a certain amount of time to terminate the task.

Single node, multi-container application pattern

Application for multiple containers, that is, multiple containers need to work together to complete the foreign service, therefore more containers is regarded as a whole operation on a node, through the Pod in K8s mechanism to deal with this scenario, which is more containers in the same Pod, there is a primary containers provide business services, to assist other container. In this scenario, the article summarizes three design modes: Sidecar, Ambassador, and Adapter.

Sidecars mode

The word “sidecar” will be familiar to those who have known about Istio, but this is not the concept of sidecar in Istio. Sidecar here is justExtend and enhanceThe main container. Sidecar can share disk space with the main container, so using log collection as a Sidecar container is a good choice (for example, Ari’s open source Log-Pilot).

Separating the enhancements from the main business functionality via the container has several advantages:

  1. As a unit of resource allocation, the container can better estimate/divide the resources consumed.
  2. Different containers can be maintained independently by different teams, dividing development responsibilities.
  3. The Sidecar container is easy to reuse
  4. The container provides a fault containment boundary so that if the Sidecar container fails, the business services of the host container will not be affected
  5. It can be maintained independently, including upgrade and rollback operations.

Ambassador mode

This pattern is actually the basis of ISTIO, hijacking the traffic of interactions between the proxy and the main container. As shown in figure

The benefits of this are obvious. The host container does not need to worry about whether the environment is complex or simple. It only needs to know the identity of the service provider and how to connect. Whether the service provider is a cluster or a single point; It doesn’t even matter what protocol you use to connect, which is done by the Ambassador container. When Istio uses an Envoy to delegate traffic, it provides more complex operations to help Istio form the entire control surface, including routing, fusing, service registration discovery, and so on.

Adapter pattern

Adapter mode is the opposite of Ambassador mode, which simplifies the host container’s perception of the external environment, while Adapter mode simplifies the presentation of the application to the outside world. For example, service monitoring is now a mandatory option, while different service applications use a variety of monitoring solutions, including Prometheus and JMX. If we need to build a unified monitoring platform, metrics with so many different technology stacks are a real headache. At this point, you need a container that translates the master container metrics to a consistent standard, and the monitoring platform only needs to align to one standard.

Multi-node application pattern

In addition to collaborating containers on a single machine, modular containers make it easier to build collaborative, multi-node distributed applications. The following design patterns are also based on the concept of POD.

Leader election

This is a common problem in distributed systems, where the implementation of the leader election is usually implemented in a different programming language. In this paper, a mode is presented. In POD, the master container runs the application service that needs to conduct the leader election. In addition, a separate container only executes the leader election algorithm (temporarily called the leader election container). The host container knows its current identity by calling the localhost interface. As shown in figure

Doing so makes it easier to develop and maintain the main container by placing the complex implementation in a separate container.

The work queue

The tasks performed by a work queue are similar to those performed by a task scheduling system (such as Celery in Python). Similar to the pattern of leader election, the design pattern proposed in this paper is still to separate the common work queue logic in distributed systems from the specific business logic, and make it platform-based and language-independent. Containers implement the run() and mount() interfaces to build a generic work queue, that is, developers only need to develop for a specific stage of the work queue logic, the final code packaging and other actions can be handled by this framework.

The architecture design of the recent CNCF hatched project ArgoProj/Argo-Workflows: Workflow Engine for Kubernetes (github.com) was also borrowed from this.

Scatter/gather model

This pattern can be colloquially understood as cloud-native MapReduce. A root server requests a number of partitioning servers to process the data in parallel. Each partitioning server returns part of the data and the root server aggregates the data back. This process can be boiled down to a sample: 1. decentralize requests; 2. Collect response data; 3. Interaction with the client. Therefore, only two containers are required for this pattern: one for partitioned data processing; One container does data aggregation (as shown in the figure below).

conclusion

In summary, this article describes how containers should be implemented and how multiple container services from a single application can co-exist in a distributed situation. Compared to SOA architecture systems, container-oriented and SOA both focus on the reuse of components through interfaces that communicate over a network, but SOA splits components into larger granularity and are more loosely coupled than the symbiotic containers mentioned above. While SOA, on the other hand, is more business-focused, the container-oriented thinking described in this article is more focused on building general-purpose middleware that makes it easier to build distributed systems.

In recent years, in fact, many of the landing schemes are developed from the design pattern in the article. For example, Istio uses Sidecar, which, like Ambassador mode here, manages traffic through proxy hijacking, separating data from control; Use of ali before open source tool for collecting container log log – pilot (AliyunContainerService/log – pilot: Collect logs for docker containers (github.com)), the Sidecar mode is used; And standard interfaces such as CRI CNI for defining container-related lifecycles. There will certainly be more applications that need to think about how distributed container services should be implemented, and this article will serve as a guide and inspiration. As summarized in the paper

We believe that the set of container patterns will only grow, and that in the coming years they will revolutionize distributed systems programming much as object-oriented programming did in earlier decades, in this case by enabling a standardization and regularization of distributed system development.

We believe that the set of container patterns will only grow, and that in the coming years they will revolutionize distributed systems programming, much as object-oriented programming has done in previous decades, standardizing and standardizing the development of distributed systems.