The ordinary path of the evolution of China Minsheng Banking Eye Log platform architecture

Build an integrated log platform based on ELK (Elasticsearch+Logstash+Kibana) to provide log collection, processing, storage, search, and display. By collecting, transmitting, and storing logs, you can centrally manage massive system logs and perform quasi-real-time search and analysis, helping O&M personnel perform quasi-real-time service monitoring, fault location and rectification, service trend analysis, and security and compliance audit, and excavating the big data value of logs in depth.

At present, logs connected to the Tianye log platform cover the log data of applications, operating systems, databases, middleware, storage and management ports, and collect and store some indicators of each module.

Early platform architecture

The early Visualize logging platform used the original ELK three-tier architecture pattern: Multiple independent Logstash Agents (shippers) collect data from different sources, a central agent(Indexer) aggregates and analyzes data, and brokers in front of the central Agent use Redis as buffers. Elasticsearch (ES for short) after the central Agent is used to store and search data. The application can use customized Kibana to provide rich chart display, or it can be customized according to the RESTful API lines of ES.

• Collection layer: The Shipper represents log collection, uses Logstash to collect data from various sources, and randomly selects a server from the List of Redis servers to connect to the Broker at startup.

• Buffer layer: The Broker acts as a buffer between the remote Shipper and the central Indexer. It is implemented using Redis to improve system performance and reliability. When the central Indexer fails to extract data, the data is stored in Redis and not lost.

• Processing layer: The central Indexer also uses Logstash to extract data from the Broker and perform related analysis and processing (filter); An Indexer polls multiple Redis for data retrieval, which prevents the corresponding Broker from being unprocessed if an Indexer goes down.

• Storage layer: Elasticsearch stores the final data and provides search capabilities.

• Display layer: Kibana provides a simple, rich Web interface with data from Elasticsearch that supports all kinds of queries, statistics, and displays.

After more than a year of time, with the development of logging platform, access log volume level geometric growth, log write request to the server great performance pressure, at the same time, application of platform operations staff needs more and more complex and diverse, the early stage of the architecture design of all levels of a series of problems are gradually exposed:

1. The Logstash based Agent platform of the collection layer is limited in category and cannot support Unix operating systems such as AIX and HP_UNIX. At the same time, the common open source product Flume has a single function and cannot meet our regular log collection requirements.

2. Log parsing of the Agent consumes a lot of resources. Therefore, log analysis and processing must be promoted to the back-end processing layer.

3. The buffer layer cannot provide distributed message queue service, and the capacity and efficiency need to be improved urgently.

4. The components of the original version of the storage layer have defects, and the version iteration is fast. Therefore, centralized upgrade is required.

5. The display layer lacks unified scene management entrance, and each application scene is independent from each other without universality. Based on the above problems, we designed a new architecture.

Tianyan log platform has been connected to 58 application systems, including 44 key core applications of A and B, covering more than 500 servers with an average daily data write volume of 5T, which can well support operation and maintenance application personnel to conduct intelligent analysis and processing of log files. The platform has reached the log collection, processing, storage, search, display and other comprehensive functional requirements. Below we share some of the work we did in this architecture.

At the Agent layer, in order to better adapt to more diverse operating systems, we mainly use Logstash and Flume to collect logs and indicators. In this process, we customize Flume more and give feedback to the community.

First of all, we use Logstash to collect and analyze logs on Linux operating system. However, because the Logstash JVM environment is not a standard JDK, Logstash cannot run on HP_UNIX and AIX operating systems, so these two operating systems use Flume for log collection. All agent parsing files are common, including Logstash and Flume. If the common log system (mainly middleware) needs to collect application logs at the same time, the Logstash agent needs to be configured to include the collection logic of both application logs and system logs. The principle is that only one process can be started for the Logstash/Flume Agent of one machine to collect logs. The syslog of the operating system and the logs of the storage device are collected in a centralized manner using a separate Logstash Agent for parsing and sending.

Flume we initially used Apache Flume 1.6. While using Taildir Source component and core component, we found that it could not fully meet our requirements. For example:

1. If the FileGroup path contains a regular expression, the complete file path cannot be obtained and the log path cannot be located after the log file is uploaded to Elasticsearch.

2. Taildir Source does not support merging multiple lines into one event and can only read the file line by line;

3. Filegroup configuration does not support directory contains regular expression, inconvenience configuration directory contains more than one date and date automatically growth, for example/app/logs/yyyymmdd/appLog log;

4. When the Host Interceptor is used, only the Host name or IP address can be retained.

After studying Flume source code, we expand development on the source code. So far, we have contributed the following 4 patches to the open source community, among which FluME-2955 has been merged by the community and released in version 1.7. Please refer to embracing Open Source and Giving Back to The Community: Minsheng Bank Flume Source Code Contribution Practice for detailed introduction of the four patches.

In addition, we have opened a version on Github, which merges three patches of FluME-2960/2961/3187 into FLUME 1.7. Welcome to try it. Address:

Github.com/tinawenqiao… , the branch name is trunk-CMbc.

Flume configuration example:

With the continuous progress of log access, the shortcoming of log parsing is revealed in the agent. Since Logstash is written in Java and the plug-in is written in JRuby, and the JVM environment is required to run, the resource requirements of the server are relatively high. At the same time, when Logstash plug-in filter/ Grok is used to process complex log field extraction and preprocessing, regular resolution will consume a lot of CPU resources, and will occasionally break the CPU monitoring alarm threshold at the initial start, which may affect the production server. Although there is no actual negative impact on the application (agent has a mechanism to monitor script suicide), the application operation and maintenance staff will be very nervous. In the later evolution of the architecture, we are gradually canceling the source parsing and replacing it with background parsing.

As the Elastic technology stack grew, Filebeat emerged. Filebeat is one of the Beat members. It is written in Go and has no dependencies. It is lighter than Logstash and is ideal for installation on production machines without excessive resource usage. In the comparison test, we created 1 GIGAByte of log data without regular parsing. Under the condition of allocating the same resources, the CPU soared to 80% at the start of single-thread logtash, and then basically around 60%. The processing was completed in 63s. When Filebeat is started, the CPU is about 40% and then about 20%, and the processing is finished in 15s. As a result, Filebeat is much better than Logstash in both performance and resource ratio. In the later evolution, we gradually put log parsing in the back end of the new architecture, and only need Filebeat to collect logs and send them as they are.

The agent deployed on the original architecture platform of the previous generation lacks unified management function. Therefore, the related configuration information needs to be manually implemented, which is less automated and cannot be associated with other systems. We under the new logging platform architecture dependent on big data control platform of complete automation requirements, big data control platform is our data base product team developed a set of big data clustering and eye log platform agent unified operational control of the project, with intelligent service discovery and server management functions. The launch of the big data management and control platform realizes the automatic deployment of the log platform Agent, click trigger start and stop, and monitoring visualization. Monitoring visualization sends heartbeat information to Kafka through Filebeat and Flume. Develop topological program on Storm cluster to consume Kafka heartbeat information and store it in MySQL database of big data control platform for display. In addition, the big data control platform is linked with the work order system, service catalog and CMDB system. The log platform itself only acts as the basic framework, and the unified authentication permission addition, metadata information delivery and work order data flow processing are all realized through the cluster modular input and uploading of agent configuration files on the management control platform page.

In terms of Agent resource consumption, in addition to necessary means of optimization, Agent process monitoring is also configured when Agent is deployed. Centralized monitoring platform is used for unified deployment, and a crontab script is deployed on Agent. Immediately kill the Agent processes that occupy high resources (CPU, memory, and storage) when a monitoring alarm is generated to prevent agent collection from affecting application system performance.

In the application log specification, we stipulate that application logs must contain a timestamp, which will be used as the default @timestamp when agent collects data. Logstash Agent needs to add an English abbreviation for its application. Flume Agent needs to use avro serialization method to add appName, hostname and Path to form an array and send it to Indexer for deserialization. The self-collected operating system indicators carry fields of the operating system type. The logs of the operating system and centralized storage must be identified by host names or IP addresses. At present, index data is stored in ES in lower case. Later, relevant logs can be optimized and analyzed according to the specific needs of system personnel.

In this layer we mainly work with Kafka instead of Redis. Although Redis was technically recommended in Elastic’s early official guidelines, Redis is not a professional message queue from a product perspective. The biggest problem with Redis queues is that capacity is limited by memory and the persistence of large single-node memory is too long. In the case of no replication, data stored in Redis is easy to be lost when the whole machine fails (replication is not used at first because copies are too wasteful). Kafka has a better throughput and clustering mode than Redis, with an average daily data volume of T levels and hundreds of millions of documents accessed per second. Kafka has a well-developed high availability mechanism, and a single Broker offline does not affect the operation of the entire cluster. The official Elastic blog post also includes a series of posts using Kafka as an ELK broker. Kafka, as a distributed message queue, can make full use of cluster resources. Each application uploads logs and allocates them to a topic. Different system logs use their own topics. A topic can have multiple partitions, which can be properly allocated to each node, and logs collected are evenly distributed to the partitions.

At the same time, we have developed corresponding management and control system for Kafka Broker, Topic, ConsumerGroup and their Offset to provide configuration services, capacity performance index collection, display and start/stop maintenance operations and other functions.

We also developed configuration management and performance monitoring capabilities for Zookeeper for Kafka:

Kafka and Zookeeper’s core microservices are open source: github.com/gnuhpc/Kafk…

Processing Layer (Indexer)

As mentioned in the previous article, we have started to gradually carry out the work of source side lightweight, so we specially apply for a batch of machines with strong CPU capacity to be used as Logstash Indexer log parsing service. The front-end plan of the Agent is to use Filebeat/Rsyslog instead of Logstash to collect logs. Filebeat only collects logs and matches multiple lines. Log parsing is concentrated in the Logstash Indexer after Kafka. The preliminary architecture diagram is as follows:

The related rules are as follows: Operating system, standard middleware, database running logs and indicators Due to the log specification, a Logstash Indexer can process all data sent by the agent. However, because of the different log formats of application logs, data collected by Flume and Logstash are processed by one or more Logstash Indexers for each system at the Indexer layer. To achieve the goal that log field parsing is performed on the back end. In ES, indexes are created by application and split by day. Application logs of the same application are stored in the same Index mode.

At the same time, we are actively investigating Hangout due to the log parsing efficiency of Logstash as a back-end Indexer and the complexity of managing many Logstash processes

(github.com/childe/hang…).

Ctrip open source, my team is the second author of the project) on Docker and the possibility of StreamSets replacing back-end Logstash for more efficient and flexible log handling. Streamsets is a data processing platform that is optimized for data in transmission. It provides a visual data stream creation model and is released in an open source way. The life cycle of the data collector can be controlled through the administrative console, providing a rich monitoring and management interface. However, it does not support the cluster mode itself, and there are few cases of actual application in production environment at home and abroad, resulting in high learning costs.

We upgraded Logstash from 2.x to 5.x. According to some test results, the performance of the new Logstash version has been greatly improved. In addition, the new version of Logstash provides a monitoring API, and Kibana’s X-Pack Montior is also free. This is very helpful for managing and viewing the Logstash state. This is especially true with our current multi-logStash Indexer consuming Kafka messages architecture. The Logstash upgrade is relatively simple, and the software layer can be replaced directly. However, since many parameters and configuration files have changed in the latest version, the directory needs to be replanned and reconfigured as follows:

1. The Logstash 5.X resolution configuration files need to be placed in separate directories, because JVM parameters and related configurations are written in separate configuration files rather than passed in as parameters. This makes it easier to allocate resources differently for different Indexers. Such as CPU, memory and so on. At the same time, because the new version of the Logstash provides a monitoring API, HTTP ports need to be allocated to access the Logstash. The Logstash cannot be started due to port conflicts in the same directory configuration. You need to add –path. Settings to a different configuration directory.

2. The Logstash version of Kafka Input no longer supports zk_connect, white_list and other old Kafka API parameters. The new parameters must be used to connect to port 9092 instead of the ZooKeeper port to consume data.

3. Options to allocate memory and Logstash. Yml to set CPU resources and Batch parameters.

4. Using the upgraded Logstash, we found that the production of Chinese logs with non-UTF-8 encoding had the problem of garble after the deserialization of Avro sequence. Finally, we modified the source code of logstuck-input-Kafka plug-in

   Vendor/bundle/app / 1.9 / gems/Logstash – input – Kafka – 5.1.8 / lib/Logstash/inputs/Kafka. Rb

   The change provides two new options:

(1) Charset specifies the encoding of the original log. The default value is not UTF-8

(2) Charset_field is an array that specifies which fields to transcode. The default value is null, that is, none of the fields to convert.

See the code

Github.com/logstash-pl…

In the next part, we will introduce the technical architecture of our Storage layer and Presentation layer and our work. Finally, we will show the current application scenarios and make a summary. In addition, the data of China Minsheng Bank is growing rapidly, and there is an urgent shortage of technical personnel related to ELK, Hadoop and Spark big data. Our official website is sincerely looking for partners who are interested in banking big data industry and focus on technology. You are welcome to contact us.

Zhao Meng works in the Big data basic technology Platform and Product Group of the Information Technology Department of the Head Office of China Minsheng Bank, and is in charge of The Tianyan log platform, focusing on the construction of the distributed log platform of the whole bank and the implementation of Elasticsearch application scheme in the bank.

Huang Pengcheng is working in the Big data basic technology platform and product Group of the Information Technology Department of the Head Office of China Minsheng Banking Corporation. He is the team leader, responsible for the planning, construction and maintenance of Hadoop platform, participating in the Tianyan log platform and big data control platform, and wechat GnuhPC.

Wen Qiao works in the Big data basic Technology Platform and Product Group of the Information Technology Department of the Head Office of China Minsheng Bank, responsible for the design and development of the bank’s big data control platform and participating in Elasticsearch technology work. She also went deep on Flume.

The Ordinary Road of The Evolution of China Minsheng Bank’s Tianyan Log Platform Architecture (Part II)

Storage Layer

Our storage layer uses Elasticsearch. Hot data is stored in SSDS, warm data is stored in SATA disks, and cold data is stored in HDFS. Elasticsearch and Logstash used version 2.3.5, based on Kibana4 for visualization. Due to the very active ES community, ELK releases have been particularly frequent, jumping from 2.x to 5.x in less than a year. 5. The X version of Lucene has been updated to 6.2, and there should be a significant performance improvement in search. At the same time, the official recommendation of the new ES performance is more stable, ELK technology stack products have great changes and function expansion. At the time of the upgrade, the official documentation showed that ES 5.x was stable. We looked at it for a while and ended up using 5.5.0. We also looked forward to indexing 5.

The official offline upgrade mode is used to directly replace the old version with the new version and ensure that the original data is available. Use the ES Migration plug-in to check for potential problems before the upgrade

A. the node properties

Some node property names have been changed, such as node.box_type to node.attr.box_type. There are no client nodes in the new version. There are master node, Data node, Ingest node, coordinating only node.

B. index settings

ES starting from 5.0 index parameters are not set in config files and need to be set at the cluster index level (except for instance levels such as index.codec which can be set at node level). Note that the following statement is an error because none of these parameters can be changed dynamically.

C. Rename Settings

Mlockall is changed to bootstrap.memory_lock. Discovery.zen. initial_ping_timeout is changed to discovery.zen.ping_timeout, default is 3s, The new version has removed discovery. Zen. Initial_ping_timeout and discovery. Zen. Ping. Two set the timeout.

D. Parameter changes

Discovery. Zen. Ping. Multicast parameters has been abandoned, in 5. Remove the multicast in X, using unicast or cloud discovery plugin. Destructive_requires_name: disable_delete_all_indices: destructive_REQUIres_name:

5.X does not have the path.work: configuration, ES 5.X translog flush parameter only

The index. The translog. Flush_threshold_size.

The ES upgrade procedure is as follows:

1. Pause writing to the Logstash Indexer log data

2. Disable cluster Shard Allocation

3. Manually perform POST /_flush? Wait_for_ongoing, returns until the operation is finished

4. Manually run the POST /_flush/synced command

5. Shut down all ES cluster nodes

6. Perform the Elasticsearch software replacement of the latest version

7. Start all ES cluster nodes

8. Re-enable shard Allocation for the cluster

9. After recovery is complete, the cluster health status changes to green

10. Restart the Logstash Indexer log data writing function

The head plug-in needs to be upgraded to 5.x. The original kopF visual monitoring plug-in is no longer available. We use Cerebro instead of KOPF, and cerebro and kopF plug-in pages are almost the same. Function can be completely replaced. At the same time, we plan to use cross-cluster, which is introduced in the latest version, to achieve cross-center and cross-cluster access.

Of course, the upgrade process can not be smooth sailing, encountered a lot of problems, now share a more representative “pit” to share with you.

After the new version of ES program replacement is restarted, the state is always red. Check the log, there are a lot of errors about IK, can not find the analyzer for IK, as shown below:

Prior to the upgrade, we learned from Github’s IK project that ik has been renamed after 5.0, as shown in the screenshot below:

For example, the index L-Kibana-2017.08 was created before the upgrade, and the index _all was set to IK. After the upgrade, the index was changed to IK_SMART. Therefore, the ik analyzer error was not found. If the index is closed and the analyzer is changed after the index is closed, the analyzer name can be changed. If the index is closed, the analyzer name can be changed. Enables the latest VERSION of the IK plug-in to support a profiler named IK.

(https://github.com/medcl/Elasticsearch-analysis-ik/pull/411/commits/63326ca322ccb8c1bd3662ab7c0bfd38a86a53cb)

After the upgrade of each component of the storage layer, the effect is obvious. The most intuitive feeling is that the problem that the heap usage of the master node continues to increase over time is no longer. In fact, the storage layer is not only upgraded to ELK, but also adjusted the underlying index structure, cleared redundant data, developed common tool scripts, small-scale resource expansion and a series of work, so after testing and actual use evaluation, the upgraded platform is more stable, more efficient query.

In addition, SSDS are introduced in the new architecture to improve THE READ and write performance of ES. An ES storage device has two SD disks and several SATA disks. Therefore, each ES Server starts three ES nodes, two Hot nodes, and one warm node. Indexer is configured with hot node ports. The template definition in ES ensures that real-time data is only written to the Hot node.

Periodically move data from the hot node to the Cold node using the official ES recommended curator tool. The SSD data retention period is one week. The action configuration file for Exhibit Exhibit is as follows:

Life cycle management The Hadoop Distributed File System (HDFS) policy is implemented to import log data to the big data platform periodically. Log data generated one month ago is periodically imported to the big data platform based on scheduled tasks, but is not saved in the ELK. Logs and indicator data of the big data platform are saved for one year. Note: The curator used in hot and cold data separation must use the latest version and will need to be re-installed after the cluster is upgraded.

Presentation Layer

Kibana5 visualization is more flexible and rich, providing more components and monitoring visualizations, more functionality and a better user experience that appeals to platform users. The Kibana upgrade is relatively simple, but the DEFAULT RAW field has been changed to a keyword field, so the Kibana indices property needs to be refreshed, and the visualize field needs to be modified. The new version of Kibana does add rich views and presentation capabilities, as well as a more aesthetically pleasing display.

Since the X-Pack Security module is charged, We use by means of self-developed Kibana proxy address (https://github.com/gnuhpc/Kibana-multitenant-proxy) has realized the Kibana access control and data shows desensitization. Currently, permissions are controlled to the Index layer, that is, a user can only access the specified Index, if access to other indexes will not be displayed on Kibana. At the same time, due to the sensitive data of the banking industry, we also provide the function of setting desensitization keyword in the configuration file, not desensitization when logging into ES, and encryption display when querying on Kibana. The results of Kibana-Proxy desensitization display are as follows:

Application scenarios

Can use keywords and simple symbols to solve the search problem, avoid the use of complex re, have a better user search experience. When multiple servers output logs, you need to quickly find out which server sends the request, what kind of error is reported, and why no error is reported on other servers. Sometimes, you need to query some server logs in parallel to determine the problem. For example, you need to query the logs of the primary and secondary databases to determine whether the databases are synchronized at a certain point.

Mining log data and extracting valuable data to form charts for display. Kibana provides rich visual chart display, convenient from the application point of view for the overall daily traffic of the business system, access to important functions statistics; Display the overall trading volume, transaction success rate and delay of the system from the perspective of transaction, and support business operation in multiple dimensions.

Periodically analyze alarm data by category to provide data support for application performance estimation, process monitoring, task flow detection, and promotion and deployment. Analyze the statistics and trends of the coverage of related technical components in the production environment, and understand the actual use of various technical components in the production from a multi-dimensional perspective.

After the upgrade of Kibana, we encapsulated A layer of general operation overview on dashboard to display the access conditions of class A, B, C and D systems in A unified manner, forming an application-oriented view analysis template. New access applications only need to apply the template directly, and there is no need for separate configuration. The template contains unified log information about service transactions, operating systems, middleware, and databases, and is expanded into an analysis view of application systems.

Summary and Prospect

After less than two years of construction, through different architecture adjustment and design and development, our elK-based log platform achieves the following functional objectives as expected:

• Accurate data location: Fine-granularity field resolution of logs meets service requirements for centralized log storage and management in different scenarios, facilitating query.

• Write and query efficiency: Through ELK platform upgrade, cluster memory tuning, reasonable fragment configuration, hot and cold data separation to maximize log write and query efficiency.

• Ha deployment: The ELK platform cluster ha deployment enables the log platform system to be available during failover without service interruption.

• Safe and reliable: the security and reliability of platform log data are guaranteed through independent development permission control and data desensitization.

Architecture continues to evolve and technology is always on the move. Objectively speaking, each architectural change of the platform is not the most correct choice or the best solution. Our big data basic technology platform and product team of MINsheng Bank has been exploring tirelessly how to make the “ordinary road” extraordinary. Our ultimate goal, as the name of the platform, development and operations engineers can make active at any time through the “third eye” view real-time system status, know all of the system, the accident hidden trouble sensing, the performance capacity of procedures, prompted by the log eye platform logging access and management become an important part of the production operations. In addition, the data of China Minsheng Bank is growing rapidly, and there is an urgent shortage of technical personnel related to ELK, Hadoop and Spark big data. We are sincerely looking for partners who are interested in banking big data industry and focus on technology on our official website. Please contact the second author on wechat to exchange your attention.

About the author:

Zhao Meng works in the Big data basic technology Platform and Product Group of the Information Technology Department of the Head Office of China Minsheng Bank, and is in charge of The Tianyan log platform, focusing on the construction of the distributed log platform of the whole bank and the implementation of Elasticsearch application scheme in the bank.

Huang Pengcheng is working in the Big data basic technology platform and product Group of the Information Technology Department of the Head Office of China Minsheng Banking Corporation. He is the team leader, responsible for the planning, construction and maintenance of Hadoop platform, participating in the Tianyan log platform and big data control platform, and wechat GnuhPC.

Wen Qiao works in the Big data basic Technology Platform and Product Group of the Information Technology Department of the Head Office of China Minsheng Bank, responsible for the design and development of the bank’s big data control platform and participating in Elasticsearch technology work. She also went deep on Flume.

Follow our wechat account “AI Front “and reply to “AI” in the background to obtain the SERIES of “AI Front “PDF e-books