background

In order to solve the problems of K8S log collection, such as multiple collection targets, difficult elastic scaling, high operation and maintenance cost, high intrusion and low collection performance, ali Cloud Kubernetes log solution was released together with the log service and container service team in February 2018. The entire cluster deployment can be completed within one minute, realizing one-stop collection of all data sources such as host logs, container logs, and container STDout on the node. And the subsequent cluster dynamic scaling does not require any secondary deployment of the collection.

Today ali Cloud Kubernetes logging solution is upgraded again, bringing you the following improvements:

  • Extreme deployment experience: You only need one command and one parameter to deploy the log solution for the entire K8S cluster.
  • Supports more configuration modes: In addition to native console and SDK configuration modes, supports CRD configuration (Kubectl, console, K8S OpenAPI).
  • K8S seamless integration: The collection configuration supports YAML deployment and is compatible with various K8S integration modes.

Log Service Introduction

Ali Cloud’s log Service is a one-stop service for log data. It was developed in 2013 and has more than 5 years of online operation experience. It has experienced the tests of Double 11 and New Year red envelopes. Log collection Agent Logtail runs on a 100W+ machine and provides services for 10,000 level applications. The main features are as follows:

Log service mainly includes real-time collection and consumption, data delivery, query and real-time analysis and other functions. Next, we introduce how to use log service for Kubernetes log collection.

This section describes the Kubernetes log collection scheme

Project introduction

Aliyun Kubernetes log collection scheme is shown in the figure above:

  • Each worker node of K8S runs a Logtail container that collects logs (including standard output and log files) from the host and other containers on the host.
  • Logtail runs in daemon set mode to ensure that each node has a Logtail container running
  • You can customize machine groups and dynamically shrink or expand clusters
  • All collection configurations support filtering specified containers by Docker lable and environment variables
  • K8S internally registers the custom resource (CRD, CustomResourceDefinition) AliyunLogConfig and deploys Alibaba-log-Controller
  • Users can manage the collection configuration through the CRD or log service console

Run the process

Take CRD configuration as an example. The internal workflow is as follows:

  1. The user to usekubectlOr other tools to apply aliyunLogConfigs CRD configuration.
  2. Alibaba-log-controller listens for configuration updates.
  3. Alibaba-log-controller automatically submits logStore creation, configuration creation, and application machine group requests to the logging service based on CRD content and server status.
  4. Logtail, running in DaemonSet mode, periodically requests the configuration server to fetch new or updated configurations and hot load them.
  5. Logtail collects standard output or log files from each POD based on configuration information.
  6. Finally, Logtail sends the processed, aggregated data to the logging service.

Deployment method

Aliyun Kubernetes users only need one command to complete the log collection deployment, and only need to enter one parameter in the command.

  1. Open ali cloud log service, log service link.
  2. Log in to the Master node of Alibaba Cloud container service Kubernetes. For details about how to log in, see SSH access cluster.
  3. In the following command${your_k8s_cluster_id}Run this command instead of your Kubernetes cluster ID.
 wget http://logtail-release.oss-cn-hangzhou.aliyuncs.com/linux64/alicloud-log-k8s-install.sh -O alicloud-log-k8s-install.sh; chmod 744 ./alicloud-log-k8s-install.sh; sh ./alicloud-log-k8s-install.sh ${your_k8s_cluster_id}Copy the code
  • Non-aliyun Kubernetes users also support, please refer to Kubernetes log Collection

Configuration mode

The log collection configuration supports console configuration by default. For Kubernetes microservices development mode, we also provide CRD configuration. You can use Kubectl directly to manage the configuration or integrate it into other orchestration services. The two configuration modes have the following features:

CRD method Console mode
Operational complexity low general
feature Support advanced configuration except console mode general
To fit the difficulty general low
The network connection Connect to the Kubernetes cluster Connect to the Internet
Integration with component/application deployment support Does not support
Authentication way Kubernetes authentication Cloud account authentication

If you’re new to logging, the console configuration is recommended, as what you see is what you get, and it’s easy to get started.

If you need to integrate log collection with service or component publishing, you are advised to use the CRD configuration. Collection configuration and service configuration can be deployed and managed directly in the same YAML file.

Plan advantage

Compared with other collection solutions, the Log service Kubernetes collection solution has the following advantages:

Introduction to core technology

In the last article of Aliyun Kubernetes log solution, we have made relevant introduction to container data collection, custom identification machine group and other technologies. This time we mainly bring you log collection configuration and K8S seamless integration of technology.

K8S is seamlessly integrated

The problem background

Unlike other open source log collection agents, Logtail has been designed with configuration management in mind from the beginning. So Logtail has supported centralized configuration management since its first release. You can centrally manage all collection configurations on the log service console or SDK, greatly reducing the log collection management burden.

But in K8S cluster environment, business application/service/component of continuous integration and automatic publishing has become the norm, use the console or the SDK operating collection configuration way is difficult and all kinds of CI, choreography framework integration, after the release of the user can only lead to business application through the console manually configure deployment and the corresponding log collection of configuration.

So the logging service has been extended specifically for K8S to support raw configuration management.

implementation

As shown in the figure above, the logging service has added a CustomResourceDefinition extension called AliyunLogConfig for K8S. Alibaba – log-Controller is also developed to monitor AliyunLogConfig events.

When users create/delete/modify AliyunLogConfig resources, Alibaba-log-controller will listen to the resource changes and create/delete/modify the corresponding collection configuration on the log service. In this way, the association between K8S internal AliyunLogConfig and the collection configuration in the log service is realized.

Alibaba -log-controller internal implementation

Alibaba-log-controller is mainly composed of 6 modules, and the functions and dependencies of each module are shown in the figure above:

  • EventListener: Listens for the CRD resource of AliyunLogConfig. This EventListener is a listener in a broad sense

    • List all of them when we initialize themAliyunLogConfigresources
    • registeredAliyunLogConfigListen for changing events
    • And then scan the whole thing periodicallyAliyunLogConfigResources prevent events from missing or processing failures
    • Package the event and hand it over to EventHandler for processing

  • EventHandler: processes the corresponding Create/Update/Delete events. As the core module of the Controller, it has the following functions:

    • Check checkpoint corresponding to ConfigMapManager. If the event has been processed (the version number is the same and the status is 200), the event is skipped
    • To prevent historical events from interfering with processing results, pull the latest resource status from the server and check whether the resource versions are the same. If the versions are different, replace the resource status with the server version
    • The event is preprocessed to meet the basic format requirements of LogSDK
    • Call LogSDKWrapper to Create log service Logstore with Create/Update/Delete configuration
    • According to the above processing results, update the correspondingAliyunLogConfigResource status

  • ConfigMapManager: Implements checkpoint management of controllers based on the K8S ConfigMap mechanism, including:

    • Maintain the mapping between checkpoint and ConfigMap
    • Provides basic checkpoint add, delete, modify, and check interfaces

  • LogSDKWrapper: A secondary wrapper based on ali Cloud’s LOG Golang SDK, including:

    • Initialize log service resources, including Project, MachineGroup, and Operation Logstore
    • The operation of converting CRD resources to log service resources is 1-to-many
    • Package SDK interface, automatically handle network exceptions, server exceptions, permission exceptions
    • Manage permissions, including automatically obtaining roles and updating STS tokens

  • ScheduledSyner: periodically synchronizes modules in the background to prevent missing events due to configuration changes during process/node failures and ensure final consistency of configuration management:

    • Periodically refresh all checkpoint andAliyunLogConfig
    • Check the checkpoint andAliyunLogConfigResource mapping. If a non-existent configuration is displayed in checkpoint, the resource is deleted

  • Monitor: Alibaba-log-controller not only outputs local run logs to STdout, but also collects logs directly to the log service for remote troubleshooting. Log collection types are as follows:

    • K8s API internal exception logs
    • alibaba-log-controllerRun log
    • alibaba-log-controllerInternal abnormal data (automatic aggregation)

Quick start

conclusion

The improvement brought by Ali Cloud log service further simplifies the threshold of K8S log collection and integration experience. Let the majority of users really experience a word: cool, from the log operation and maintenance personnel’s quality of life greatly improved.

Currently, Logtail supports host file collection, container file collection, container STdout collection, and the following collection methods (all of which are supported in K8S) :

  1. Syslog acquisition
  2. Mysql binlog acquisition
  3. JDBC acquisition
  4. HTTP acquisition