The background,

The tide of cloud native technology has come, and technological change is imminent. In this technological trend, netease launched The Qingzhou Micro-service cloud platform, which integrates micro-services, Servicemesh, container cloud, DevOps, etc., and has been widely used in the company group, as well as supporting the transformation and migration of many external customers.

Logging is a part of it that is often overlooked, but it is an important part of microservices and DevOps. Without logs, it is impossible to troubleshoot service problems. In addition, the unified collection of logs is also the basis for analyzing, processing, and auditing many service data. However, in the cloud native containerized environment, log collection becomes a little different.

2. Pain points of container log collection

Traditional host mode

For services deployed on traditional physical machines or VMS, log collection is clear. Service logs are directly output to the host, and the service runs on a fixed node. Deploy the log collection Agent on the node manually or with an automatic tool, add the configuration of the agent, and then you can start collecting logs. In addition, a configuration center can be added to deliver agent configurations to facilitate subsequent log configuration modification.

Kubernetes environment

In the Kubernetes environment, things are not so simple. A Kubernetes node has many different service containers running, the container log storage mode has many different types, such as stdout, hostPath, emptyDir, PV, etc. Due to frequent Pod active or passive migration, frequent destruction and creation in Kubernetes cluster, we cannot manually deliver the log collection configuration to each service like the traditional way. In addition, log data is stored in a centralized manner after collection. Therefore, it is important to search and filter logs based on namespace, POD, Container, Node, and even the environment variables and labels of containers. The above are different from the traditional log collection configuration needs and pain points, investigate the reason, or because the traditional way out of Kubernetes, unable to perceive Kubernetes, but also unable to integrate with Kubernetes. With its rapid development in recent years, Kubernetes has become the de facto standard for container choreography and can even be considered a new generation of distributed operating systems. In this new operating system, the design idea of controller drives the operation of the whole system. An abstract interpretation of controller is shown below:

Due to the good scalability of Kubernetes, Kubernetes designed a concept of custom resource CRD, users can define a variety of resources, and with the help of some framework to develop controller, use controller to turn our expectations into reality. Based on this idea, for log collection, what kind of logs a service needs to collect and what kind of log configuration is the user’s expectation, and all these need to develop a log collection controller to achieve.

3. Exploration and architecture design

With the above solution, in addition to developing a controller, what remains is some selection analysis around this idea.

Log collection agent selection

The log collection controller is only responsible for connecting to Kubernetes and generating the collection configuration, but not for actual log collection. At present, there are many log collection agents on the market, such as Logstash of traditional ELK technology stack, Fluentd, a graduate project of CNCF, Loki, and Beats series of Filebeat. Here’s a simple analysis.

  • Based on the JVM, Logstash takes up hundreds of MB or even more than GB of memory per minute. It is a bit heavy and we excluded it first.

  • Fluentd looks good on the back of CNCF and has a lot of plug-ins, but written in Ruby and C, it’s still a wait and see for our team’s technology stack.

    Although Fluentd also launched a pure C-based Fluentd-bit project, which has a small memory footprint and looks very tempting, the use of C language and the lack of dynamic reload configuration is not very close.

  • Loki has been released for a short time, and currently has limited functions, and some pressure test data shows that the performance is not very good, so wait and see.

  • Filebeat and Logstash, Kibana and Elasticsearch belong to Elastic. They are lightweight log capture agents designed to replace Logstash. They are based on Golang and fit perfectly with our stack. After measurement, the performance and resource occupancy rate are excellent, so it becomes the first choice of our log collection agent.

Agent Integration mode

Generally, there are two deployment modes for log collection agent in Kubernetes.

  1. The sidecar mode is deployed in the same Pod as a service Container. In this mode, Filebeat collects only the logs of the service Container and only needs to configure the log configuration of the container. This mode is simple and isolated. Every service needs to have a Filebeat to collect, usually there are many PODS on a node, the total memory and other overhead is not optimistic.

  2. The other, and most common, is to deploy a Filebeat container on each Node, which generally has a much smaller memory footprint, is non-invasive to pods, and is more in line with our usual usage.

    At the same time, Kubernetes DaemonSet deployment is generally used, which eliminates traditional automatic operation and maintenance tools like Ansible, greatly improving the deployment and maintenance efficiency.

    So we preferred to deploy Filebeat using Daemonset.

    The overall architecture

After selecting Filebeat as the log collection agent and integrating the self-developed log Controller, we can see the architecture as follows from the node perspective:

  1. The logging platform delivers specific CRD instances to the Kubernetes cluster, and Log Controller Ripple takes care of the List&Watch Pod and CRD instances from Kubernetes.
  2. After Ripple filtering and aggregation, a Filebeat input profile is generated. The profile describes the service collection Path, multi-line log matching, and PodName and Hostname in the log meta information by default.
  3. Filebeat automatically reloads and collects logs on nodes based on Ripple configurations and sends the logs to Kafka or Elasticsearch.

Because Ripple listens for Kubernetes events, it can sense the Pod lifecycle and automatically generate the corresponding Filebeat configuration regardless of whether the Pod is destroyed or dispatched to any node without human intervention. Ripple can sense the Volume of logs mounted by Pod. Whether docker Stdout logs are stored or HostPath, EmptyDir, or Pv is used to store logs, Ripple can generate log paths on nodes and inform Filebeat to collect logs. Ripple can obtain CRD and Pod information at the same time. Therefore, in addition to adding meta information, such as PodName, to log configurations by default, Ripple can also mark logs with container environment variables, Pod label, and Pod Annotation to facilitate subsequent log filtering and query. In addition, we have added the function of periodic log clearing to ensure that logs are not lost to further enhance log collection and stability.

4. Practice based on Filebeat

Function extension

In general, Filebeat can meet most of the log collection requirements, but it still needs to be customized in some special scenarios. Of course, The design of Filebeat also provides good scalability. Filebeat currently only provides several types of output clients like ElasticSearch, Kafka, LogStash, etc. If we want Filebeat to send directly to other backend, we need to customize our own output. Similarly, if you need to filter logs or add meta information, you can also customize the Processor plug-in. Whether it’s adding output or writing a processor, Filebeat provides the same general idea. Generally speaking, there are three ways:

  1. Fork Filebeat directly, developing on existing source code.

    Output or Processor provide interfaces like Run, Stop, etc. You just need to implement these interfaces and register the corresponding plug-in initialization methods in the init method.

    Of course, since the init method in Golang is called when the package is imported, you need to manually import Filebeat in the code that initializes it.

  2. Copy Filebeat’s main.go, import our own plugin library, and recompile.

    It’s essentially not that different from method 1.

  3. Filebeat also provides a plugin mechanism based on Golang Plugin. You need to compile your own plugin into a.so shared link library, and then specify the path of the library through -plugin in Filebeat startup parameter.

    Golang plugin is not yet mature and stable, and a plugin needs to rely on the same libbeat library and compile with the same Golang version.

In order to support the docking of various business parties, we have expanded the development of GRPC output, support multiple Kafka cluster output and so on.

Three-dimensional monitoring

However, the real difficulty is that after the actual use of the business side, various logs cannot be collected, multi-line log configuration or collection of large binary files, resulting in problems such as Filebeat OOM. We spent more time monitoring the full range of Filebeat and log collection, such as:

  1. Access to the light boat monitoring platform, with disk IO, network traffic transmission, memory occupancy, CPU usage, POD event alarm, to ensure the improvement of basic monitoring.
  2. Added log platform data link delay monitoring.
  3. Collect the logs of Filebeat, which log files are reported to start the collection and when the collection ends, so that you do not need to SSH to each node to check the log configuration and troubleshoot problems.
  4. I developed Filebeat Exporter, connected to Prometheus, and collected and reported metrics data.

Through the enhancement of three-dimensional monitoring, we greatly facilitate the investigation of problems, reduce the operation and peacekeeping labor costs, but also ensure the stability of the service.

5. Golang performance optimization and tuning

From Docker to Kubernetes, from Istio to Knative, golang-based open source projects have become the main force of the cloud native ecosystem, and Golang’s simplicity and efficiency are constantly attracting new projects to adopt it as a development language. In addition to using Golang to write Filebeat plug-in and develop controller for log collection, there are also many components based on Golang on our Micro-service platform. Among them, we have stepped in many holes and accumulated some experience in Golang optimization. But a lot of times, we’ve seen so much about GC principles, memory optimization, performance optimization, and we just don’t know what to do when we’re done with the code or the project. Practice is the sole criterion for testing truth. Therefore, it is a shortcut to improve posture and find the key problem. For performance optimization, Golang provides us with three keys:

  • go benchmark
  • go pprof
  • go trace

Here’s a simple example. Sync. Pool, for example, is used to save and reuse temporary objects to reduce memory allocation and reduce GC pressure. There are many application scenarios, such as FastHttp, which claims to be 10 times faster than Golang’s official Http, which uses sync.Pool heavily, and Filebeat, which uses sync.Pool to aggregate Batch log data into batches. Sync.pool is also used to optimize rendering efficiency when nginx-ingress-controller renders to generate Nginx configurations. Our log Controller Ripple also uses sync.pool to optimize the performance of rendering Filebeat configurations.

First, use go Benchmark to press the Filebeat configuration with go Template without sync.Pool.

You can see the results showing the time of each execution of the method, and the memory allocated.

Then take a look at the profile generated by Go Benchmark and use go Pprof to see the overall performance data.

Go Pprof actually has a lot of data for us to observe, but here we just show the memory allocation information. As you can see, Benchmark has applied for up to five gigabytes of memory during this time.

Next, we use Go Trace to check the goroutine, heap memory, GC, etc.

Only the time period from 600ms to 700ms is captured here, and it is clear that 170 GC events occurred in this 100ms.

Use the same method and procedure to pressure the sync.Pool.

The total amount of memory allocated was reduced to 160MB, and the number of GC cycles was reduced to 5 in the same time period. The gap is stark.

Summary and Prospect

In the era of cloud native, log, as a part of observability, is the basis for us to investigate and solve problems, and also the beginning of subsequent big data analysis and processing. Although there are many open source projects in this area, there is still no strong and unified log collection agent, and perhaps the blooming of a thousand flowers will continue. Therefore, the design of our own log Agent Ripple also proposes more abstractions and retains the ability to interconnect with other log collecting agents. In the future, we plan to support more log collection agents and build a richer and more robust cloud native log collection system.