preface


It is too troublesome to check the log of the new project. It is checked between multiple machines, and I still don’t know whether it is the same request. When printing logs, use MDC to add a traceId to the logs. How does the traceId be transmitted across systems?


[liuzhirichard] Record technology, development and source code notes in work and study. From time to time, share what you’ve seen and heard in your life. Welcome to guide!

background

Also notes for the development of new projects, because the distributed architecture is used and involves the interaction between various systems

Here’s a very common problem:

  1. A single system is deployed in a cluster, and logs are distributed on multiple servers.
  2. Multiple systems log in multiple machines, but a request, check the log is more difficult.

The solution

  1. SkyWalking tracEID is used for link tracing.
  2. Use trace.id of Elastic APM for link tracing;
  3. Generate traceId and put the traceId to the MDC node.

MDC

MDC (Mapped Diagnostic Context) is a map used to store Context data for the specific thread running the Context. Therefore, if log4j is used for logging, each thread can have its own MDC that is global to the entire thread. Any code belonging to that thread can easily access values that exist in the thread’s MDC.

How to Use MDC

  1. inlog4j2-spring.xmlLog format%X{traceId}Configuration.
<Property name="LOG_PATTERN">
    [%d{yyyy-MM-dd HH:mm:ss.SSS}]-[%t]-[%X{traceId}]-[%-5level]-[%c{36}:%L]-[%m]%n
</Property>
<Property name="LOG_PATTERN_ERROR">
    [%d{yyyy-MM-dd HH:mm:ss.SSS}]-[%t]-[%X{traceId}]-[%-5level]-[%l:%M]-[%m]%n
</Property>

<! - omitted - >

<! -- Configuration of the output console -->
<Console name="Console" target="SYSTEM_OUT" follow="true">
    <! -- Output log format -->
    <PatternLayout charset="UTF-8"  pattern="${LOG_PATTERN}"/>
</Console>
Copy the code
  1. Adding interceptors

Intercept all requests, get the traceId from the header and place it in the MDC, or generate one with the UUID if it does not get one.

@Slf4j
@Component
public class LogInterceptor implements HandlerInterceptor {
    
    private static final String TRACE_ID = "traceId";

    @Override
    public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception arg3) throws Exception {}@Override
    public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView arg3) throws Exception {}@Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {

        String traceId = request.getHeader(TRACE_ID);
        if (StringUtils.isEmpty(traceId)) {
            MDC.put(TRACE_ID, UUID.randomUUID().toString());
        } else {
            MDC.put(TRACE_ID, traceId);
        }


        return true; }}Copy the code
  1. Configuring interceptors
@Configuration
public class WebConfig implements WebMvcConfigurer {
    @Resource
    private LogInterceptor logInterceptor;

    @Override
    public void addInterceptors(InterceptorRegistry registry) {
        registry.addInterceptor(logInterceptor)
                .addPathPatterns("/ * *"); }}Copy the code

How to pass traceId across services

  • FeignClient

Since FeignClient is used to make calls between services, you only need to add a request interceptor

@Configuration
public class FeignInterceptor implements RequestInterceptor {

    private static final String TRACE_ID = "traceId";

    @Override
    public void apply(RequestTemplate requestTemplate) { requestTemplate.header(TRACE_ID, MDC.get(TRACE_ID)); }}Copy the code
  • Dubbo

If Dubbo is used, traceId can be transmitted by extending Filter

  1. Write the filter
@Activate(group = {"provider", "consumer"})
public class TraceIdFilter implements Filter {

    @Override
    public Result invoke(Invoker
        invoker, Invocation invocation) throws RpcException {


        RpcContext rpcContext = RpcContext.getContext();


        String traceId;

        if (rpcContext.isConsumerSide()) {

            traceId = MDC.get("traceId");

            if (traceId == null) {
                traceId = UUID.randomUUID().toString();
            }

            rpcContext.setAttachment("traceId", traceId);

        }

        if (rpcContext.isProviderSide()) {
            traceId = rpcContext.getAttachment("traceId");
            MDC.put("traceId", traceId);
        }

        returninvoker.invoke(invocation); }}Copy the code
  1. Specify a filter
SRC | - the main | - Java. | | - com | - XXX - XxxFilter Java (implement the Filter interface) | - resources | - meta-inf | - dubbo | - org. Apache. Dubbo. RPC. The Filter (Plain text file, XXX = com.xxx.xxxfilter)Copy the code

Screenshot below:

The test results are as follows:

Dubbo filter source code address at the end of the article can also pay attention to the public number, send traceid to obtain

The other way

If you’re using SkyWalking or Elastic APM, you can inject in the following way:

  1. SkyWalking
<dependency>
    <groupId>org.apache.skywalking</groupId>
    <artifactId>apm-toolkit-log4j-2.x</artifactId>
    <version>{project.release.version}</version>
</dependency
Copy the code

Then set [%traceId] to pattern in the log4j2. XML file

  1. Elastic APM

    1. Specify enable_log_correlation to true at startup
    2. will%X{trace.id}The configuration is in pattern in the log4j2. XML file

extension

Unified Log Collection

With traceId, logs can be traced and queried on multiple servers. To improve query efficiency, you can summarize logs together.

The common usage method is the elK-based logging system:

  1. Use FileBeat to collect logs and submit them to the Logstash
  2. Logstash performs word segmentation filtering and output to Elasticsearch
  3. Query logs from Elasticsearch using Kinbana or my own visualization tool

conclusion

This article mainly records the recent development process encountered a point of problem, hope to help small partners. Any deficiencies are welcome to be corrected. If you have other suggestions or opinions, please leave a comment and make progress together.

The relevant data

  1. Log4j 2 API:logging.apache.org/log4j/2.x/m…
  2. SkyWalking:github.com/apache/skyw…
  3. Elastic APM: www.elastic.co/guide/en/ap…
  4. Dubbo filter:dubbo.apache.org/zh-cn/docs/…
  5. Dubbo Filter Demo: github.com/liuzhihangs…