With the popularity of microservice architectures, multiple services are often involved in a single request, making service performance monitoring and troubleshooting more complex. APM was born.

Currently, there are many implementations of APM server on the market. Zipkin, Jaeger, SkyWalking, Elastic APM, etc. (Pinpoint doesn’t support OpenTracing, so we don’t cover it.)

This tutorial provides only a few development ideas and is technically demanding, as much of it involves understanding the source code of the corresponding component. The full source code is available on Github.

This time, Jaeger is chosen as the server example. The main content is as follows.

---

The first part

In the first part, we’ll take a look at key apM information and some conceptual things to prepare for the examples that follow. Here, we briefly describe how the Jaeger was installed and some of the preparations.

The second part

Use a simple Java program to show how to generate traces using the OpenTracing API and demonstrate a call chain generation with two nodes.

The third part

Use OkHttp3 and SpringBoot to demonstrate how to implement a distributed call. This article is still built using the underlying API, so feel good about development should be able to handle a variety of scenarios.

The fourth part

Take SpringCloud as an example to illustrate the call chain generation logic of microservices. Because of the variety of calls, we will only use the popular Feign call to illustrate how it works. The implementation of SpringCloud already has a better wheel, considered also more comprehensive, can refer to its source code for analysis.

---

If you are developing your own middleware, or doing some integration work, this tutorial will allow you to quickly add APM functionality to your components. As long as your API is compatible with the OpenTracing protocol.

---

Overall introduction

For monitoring system and APM link, little sister taste public account has a more detailed description, it is recommended to read it first.

Out of all the monitoring components, there’s always one for you

The following is a brief overview of the main elements of the call chain.

Call chain main factor

Data Collection section

It is mainly used for diversified data collection and preparation for data analysis. It is required to be easy to use with minimal intrusion (development effort) and in extreme cases (such as collection components not available) without any impact on the business. As you can see, the amount of development in this section is enormous, especially with the need to integrate upstream and downstream of Nginx, diverse base components, and diverse technology stacks.

Data Analysis section

There are real-time analysis and offline analysis. In general, real-time analysis is more valuable, with key outputs such as second-level call volume, average response time, TP values, etc. In addition, the call chain (Trace) needs to store the full amount of data, and some requests with high concurrency and large buried points will have performance problems.

Monitoring alarm

This part uses the output of data analysis to inform subscribers of their attention through SMS emails and other forms. Monitoring and alarm platforms should be as close as possible to DevOPS platforms, including autonomous service platforms.

Why jaeger

Jaeger’s development is active, and its modular division is flexible. In the case of very large data volumes, the data can be buffered in Kafka first (as well as in preparation for accessing various flow analysis platforms). These are explained in passing in the jaeger installation section.

scaling

This is handy if your project uses SpringBoot. The following content has been mentioned more than once, please ignore those you have read.

Here’s what we’ll do: Any method annotated with @owlTrace will generate a call chain message.

First, we define an annotation:

import java.lang.annotation.*;
@Target({ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface OwlTrace {
}
Copy the code

Then, implement its processing class. The code intercepts the Spring-managed beans using AOP, and very simply implements the construction of Trace information. The code is as follows:

import io.opentracing.Span;
import io.opentracing.Tracer;
import io.opentracing.tag.Tags;
import lombok.extern.slf4j.Slf4j;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Lazy;

import java.util.LinkedHashMap;
import java.util.Map;


@Configuration
@Slf4j
public class OwlTraceAutoConfiguration {
    static final String TAG_COMPONENT = "java";

    @Autowired
    @Lazy
    Tracer tracer;

    @Bean
    public TracingAspect pxTracingAspect(a) {
        return new TracingAspect();
    }

    @Aspect
    class TracingAspect {
        @Around("@annotation(com.sayhiai.arch.trace.annotation.OwlTrace)")
        public Object pxTraceProcess(ProceedingJoinPoint pjp) throws Throwable {
            Span span = null;
            if(tracer ! =null) {
                final String cls = pjp.getTarget().getClass().getName();
                final String mName = pjp.getSignature().getName();
                span = tracer.buildSpan(cls + "." + mName)
                        .withTag(Tags.COMPONENT.getKey(), TAG_COMPONENT)
                        .withTag("class", cls)
                        .withTag("method", mName)
                        .startActive(false)
                        .span();
            }
            try {
                return pjp.proceed();
            } catch (Throwable t) {
                Map<String, Object> exceptionLogs = new LinkedHashMap<>(2);
                exceptionLogs.put("event", Tags.ERROR.getKey());
                exceptionLogs.put("error.object", t);
                span.log(exceptionLogs);
                Tags.ERROR.set(span, true);
                throw t;
            } finally {
                if(tracer ! =null&& span ! =null) {
                    span.finish();
                }
            }
        }
    }
}

Copy the code

Finally, add the path to SRC /main/resources/ meta-INF /spring.factories according to how Spring loads:

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
arch.trace.core.owl.OwlTraceAutoConfiguration,\
arch.trace.core.log.LoggingAutoConfiguration
Copy the code

A Spring Boot starter is implemented by packing components into jars.

use

application.properties

Make sure you have AOP on

# AOP
spring.aop.auto=true
spring.aop.proxy-target-class=true

opentracing.jaeger.log-spans=trueOpentracing. Jaeger. Udp - sender. Host = 192.168.3.237 opentracing. Jaeger. Udp - sender. Port = 5775Copy the code

End

Next, we will step into the development of the client API.