Recently, Loggie project (github.com/loggie-io/loggie/), a cloud native log system jointly initiated by NetEase and INDUSTRIAL and Commercial Bank of China, has officially opened source. This is a charge launched by NetEase Jufan to the pain points of cloud native log, and another action taken by the team and partners to practice the technical concept of “open architecture, open kernel” and hand over control to customers.
Loggie Project: Cracking the pain of cloud native logging
In the wave of enterprise digital transformation, it has become the mainstream choice to adopt cloud native technology to solve the new challenges of digital software development and operation and maintenance. However, in the cloud native environment, the large-scale and frequent dynamic migration of ** containers, log storage diversity, Kubernetes meta information query and other characteristics force the log management mode to change. With the deepening of business practice, problems such as more and more human operation and maintenance in log, difficulty in expanding function development and difficulty in supporting a larger scale gradually surfaced. Existing open source solutions in the industry failed to meet the needs, such as:
• Performance is not up to par, or extension features are not developed efficiently and well balanced
• Limited support for containerized scenarios
• Most open source projects do not provide a complete logging solution
In this context, Loggie for cloud native scenarios came into being. **Loggie is a lightweight, high-performance, cloud-native log collection Agent and relay processing Aggregator based on Golang, ** supports multiple Pipeline and component hot plug, providing:
• One-stack log solution: simultaneously supports log transfer, filtering, parsing, segmentation, log alarm, etc
• Cloud native log format: fast and convenient container log collection mode, native Kubernetes CRD dynamic configuration delivery
• Production-level characteristics: Based on long-term large-scale operation and maintenance experience, it has formed a full range of observability, fast obstacle removal, abnormal warning, automatic operation and maintenance capabilities
Based on the long-term business practice of NetEase Shufan, Loggie has formed the following characteristics:
• Extensible, hot pluggable: Different Source/Interceptor/Sink configurations, with forwarding, filtering, parsing, segmentation, log alarm and other capabilities, you can use Golang quickly self-developed plug-in
• Strong isolation: Multiple Pipeline design, reduce mutual interference, can simultaneously send multiple different data sources
• Lightweight, high performance: Based on Golang, very few resources, strong throughput performance
• Reliability: Complete log observability, native Prometheus Metrics support, limited stream and other interceptors
• Cloud native: The configuration center integrates Kubernetes, and creates CRD instances to collect container logs
At the same time, based on Loggie’s design of streaming data transmission, we can flexibly use Loggie Pipeline.
From the use form can be divided into:
• Agent mode: One Agent for each node or Pod to collect logs or other data
• Aggregator: used for forwarding, forwarding, and processing and can be independently deployed as a cluster
Can be applied in:
• Data collection: Collects container logs, node logs, Prometheus Metrics, and Kubernetes Events
• Data transfer: perform data aggregation, forwarding and diversion as the intermediary
• Data processing: cutting, converting and processing streaming data
• Log alarm: detect and alarm abnormal logs
.
In addition, using Loggie’s design, users can quickly develop a Source, Sink, or Interceptor component, reuse Loggie’s capabilities, and avoid a lot of repetitive development efforts, such as:
• In Kubernetes cluster can be convenient, direct use of CRD configuration, and support automatic reload, support to specify Loggie cluster, no need to consider deployment, configuration updates and other issues
• Loggie provides stability and reliability during transmission, ensures at-least-once and retry mechanisms, and avoids data loss and risks caused by excessive or excessive data
• Quick access to Prometheus using Loggie’s set of monitoring metrics, such as queue length, transmission latency, QPS, etc., as well as some of the system’s built-in interfaces and capabilities for fast troubleshooting
• Using the Interceptor can be used for custom data processing, format conversion, etc., avoiding too much custom code development
Loggie Ecology: Work with partners to strengthen system capabilities
**Loggie grew out of the actual needs of NetEase Yanxuan business, grew out of the long-term cooperation between Yanxuan and Sufan, and continued to develop from the close cooperation between NetEase Sufan, NetEase News and INDUSTRIAL and Commercial Bank of China. The extensive ecosystem allows projects to evolve and mature based on business needs.
Before Loggie, NetEase Yan Xuan used open source logging solutions. Due to multiple log collection scenarios, such as virtual machines, physical machines, and containers, ** needs to deal with heavy traffic that is 10 times more than that in the normal peak period. Strict selection encounters great pain points in collection performance, resource isolation, observability, anomaly discovery, and automatic operation and maintenance. ** Therefore, in 2019, Yan Xuan and Shufan began to build a new Loggie system oriented to cloud native.
Chen E, General manager of NetEase Shufan Light Boat products, said:
Loggie is an important achievement of our cloud native practice. The project follows the standard construction of cloud native community, is compatible with traditional technology architecture, and integrates the experience of industrial and Commercial Bank of China and other different industry application scenarios. By adopting Loggie, enterprises can unify log management and guarantee the smooth landing of cloud native applications. The maturity of Loggie is due to the co-construction of NetEase Sufan and business. We believe that open source will enable the project to adapt to more industrial scenarios, enrich the open source technology ecosystem and promote the development of cloud native technology.
Wang Guoyun, senior technical expert of NetEase and person in charge of Yan Zhaopai, said:
Loggie has been fully proven in NetEase yan Selected hybrid production environments for over two years. Now, with the support of the group, we open source Loggie. On the one hand, we hope to fill the gap in the community’s one-stop logging solution and help more enterprises with similar problems. On the other hand, we also hope that the Loggie project will continue to grow and develop in a more sustainable and healthy way with the help of the community.
In the future, Yan Xuan will continue to participate in the development and management of Loggie project based on its own business scenarios and technology accumulation in cloud native practices. We also look forward to more like-minded partners participating in the project.
Industrial and Commercial Bank of China has built the largest enterprise-level cloud platform in the industry, with 100% core applications in the cloud, over 100 container clusters and over 300,000 operating containers. In 2017, ICBC began to build an enterprise-level log center. The log center can collect, clean, format and store all kinds of application logs, and support users to search and download. The average daily log capacity reaches 45TB. Create lightweight operation service based on standardized log data, which can quickly and effectively support real-time acquisition of basic operation indicators such as transaction rate and time consumption, and also provide convenient user view on mobile terminal and front desk.
Wang Xin, director of cloud Computing Laboratory at ICBC Software Development Center, said:
In terms of community contribution, ICBC, as a Loggie project participant, combined with the log platform construction experience, carried out in-depth customized development in core modules such as collection target discovery, log interception and processing, and component self-monitoring. In the future, ICBC will continue to participate in the work of Loggie community, and plan to continue in-depth research in modules such as log output expansion and streaming processing to feed back the open source community.
Loggie’s future: More observability needs
Loggie has been widely used within NetEase, industrial and Commercial Bank of China, and NetEase Light Boat’s numerous commercial customer environments. In the actual measurement of Yan Selection service of NetEase, Loggie consumes only 1/4 of the CPU of the previous scheme, and the sending throughput is 1.6 ~ 2.6 times of the latter, and even reaches 200MB/s+ limit throughput under multi-pipeline acquisition.
After the release of Loggie open source, the number of self-developed cloud native projects of ** NetEase Shufan open source has reached 6. ** Also includes cloud native software-defined storage system Curve, multi-tenant visual Kubernetes management platform KubeCube, cloud native operation and maintenance diagnostic system KubeDiag, high-performance cloud native API gateway Hango, smart grid manager Slime, etc. Chen Ge previously stated that NetEase’s series of open source work in the field of cloud native technology aims to facilitate the introduction of cloud native technology stack for customers, and help customers solve problems such as control, maintenance and stability of cloud native infrastructure by enhancing the open source of basic components at the enterprise level.
Loggie’s open source means that NetEase Zipan also falls behind in the observability field of cloud native open source map, forming a more complete layout.
In the future, Loggie** will provide more components and functionality extensions, introduce lightweight streaming capabilities, and continue to address more pain points of cloud native logging and meet more needs in the observability direction. ** If you are interested in this field, please learn about the Loggie project and get involved in building the Loggie community!
To learn more
Loggie project address: github.com/loggie-io/l…
Loggie-io.github. IO /docs/
Open source overview: sf.163.com/opensource