In a microservice architecture, the call chain can be long, from HTTP to RPC and from RPC to HTTP. And developers want to know the call situation and performance of each link, the best solution is full link tracing.

The way to trace is to generate a spanID of your own at the start of a request and pass it along the entire request link. We use this spanID to look at the overall link health and performance issues.

Let’s look at the link implementation of Go-Zero.

The code structure

  • Spancontext: Saves link context information “traceid, SPANID, or whatever else you want to pass”
  • Span: an operation, storage time, and some information in a link
  • propagator:tracePropagate downstream operations “extract, inject”
  • noop: implemented emptytracerimplementation

concept

SpanContext

Before WE introduce span, let’s introduce context. SpanContext holds context information for distributed tracing, including Trace ids, Span ids, and anything else that needs to be passed downstream. The Implementation of OpenTracing needs to pass the SpanContext through some protocol to associate spans from different processes to the same Trace. For HTTP requests, SpanContext is typically passed in HTTP headers.

Here is the default spanContext implementation of Go-Zero

SpanContext struct {traceId string // traceId indicates the global unique ID of tracer. SpanId string // spanId indicates the unique ID of a span in a single trace. Unique in trace}Copy the code

Developers can also implement the interface methods provided by SpanContext to pass their own context information:

type SpanContext interface { TraceId() string // get TraceId SpanId() string // get SpanId Visit(fn func(key, Val string) bool) // Custom operation TraceId, SpanId}Copy the code

Span

A REST call, database operation, etc., can be a span. Span is the smallest Trace unit for distributed tracing. A Trace consists of multiple spans. The tracing information contains the following information:

type Span struct {
    ctx           spanContext       // The context of the pass
    serviceName   string            / / service name
    operationName string            / / operation
    startTime     time.Time         // Start time stamp
    flag          string            // flag whether the trace is a server or a client
    children      int               // This span fork comes out childsnums
}
Copy the code

From the definition structure of span: in microservices, this is a complete child call process, including the call start startTime, spanContext, which marks its own unique attribute, and the number of children of fork.

Application instance

HTTP is already integrated as built-in middleware in Go-Zero and RPC. Let’s look at how to use HTTP, RPC, tracing:

HTTP

func TracingHandler(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        / / * * 1 * *
        carrier, err := trace.Extract(trace.HttpFormat, r.Header)
        // ErrInvalidCarrier means no trace id was set in http header
        iferr ! =nil&& err ! = trace.ErrInvalidCarrier { logx.Error(err) }/ / 2 * * * *
        ctx, span := trace.StartServerSpan(r.Context(), carrier, sysx.Hostname(), r.RequestURI)
        defer span.Finish()
        / / * * * * 5
        r = r.WithContext(ctx)

        next.ServeHTTP(w, r)
    })
}

func StartServerSpan(ctx context.Context, carrier Carrier, serviceName, operationName string) ( context.Context, tracespec.Trace) {
    span := newServerSpan(carrier, serviceName, operationName)
    / / * * * * 4
    return context.WithValue(ctx, tracespec.TracingKey, span), span
}

func newServerSpan(carrier Carrier, serviceName, operationName string) tracespec.Trace {
    3 / / * * * *
    traceId := stringx.TakeWithPriority(func(a) string {
        ifcarrier ! =nil {
            return carrier.Get(traceIdKey)
        }
        return ""
    }, func(a) string {
        return stringx.RandId()
    })
    spanId := stringx.TakeWithPriority(func(a) string {
        ifcarrier ! =nil {
            return carrier.Get(spanIdKey)
        }
        return ""
    }, func(a) string {
        return initSpanId
    })

    return &Span{
        ctx: spanContext{
            traceId: traceId,
            spanId:  spanId,
        },
        serviceName:   serviceName,
        operationName: operationName,
        startTime:     timex.Time(),
        // mark it as server
        flag:          serverFlag,
    }
}
Copy the code
  1. Run the header -> carrier command to obtain information about the traceId in the header
  2. Open a new span and wrap “traceId, spanId” in the context
  3. Get the traceId, spanId, from the carrier “i.e., header” above.
    • Check whether the header is set
    • If not set, random generation returns
  4. fromrequestGenerates a new CTX, encapsulates the corresponding information in the CTX, and returns
  5. Copy the current context from the above contextrequest

This enables the span information to be passed to the downstream service along with the Request.

RPC

There are clients and servers in RPC, so there is also clientTracing and serverTracing from TRACING. The logic of serveTracing is basically the same as that of HTTP.

func TracingInterceptor(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ... grpc.CallOption) error {
    // open clientSpan
    ctx, span := trace.StartClientSpan(ctx, cc.Target(), method)
    defer span.Finish()

    var pairs []string
    span.Visit(func(key, val string) bool {
        pairs = append(pairs, key, val)
        return true
    })
    // **3** Add data in the pair to CTX as a map
    ctx = metadata.AppendToOutgoingContext(ctx, pairs...)

    return invoker(ctx, method, req, reply, cc, opts...)
}

func StartClientSpan(ctx context.Context, serviceName, operationName string) (context.Context, tracespec.Trace) {
    / / * * 1 * *
    if span, ok := ctx.Value(tracespec.TracingKey).(*Span); ok {
        / / 2 * * * *
        return span.Fork(ctx, serviceName, operationName)
    }

    return ctx, emptyNoopSpan
}
Copy the code
  1. Gets the SPAN context information brought down upstream
  2. Create a new CTX from the acquired span, span “inherits traceId from parent span”
  3. Span-generated data is added to CTX and passed to the next middleware to flow downstream

conclusion

Go-zero intercepts the request to obtain the link traceID, and allocates a root Span to the entry of the middleware function. In subsequent operations, sub-spans are split, and each Span has its own specific identifier. After Finsh, it is collected in the link tracking system.

The ELK tool allows developers to trace traceID and see the entire invocation chain. At the same time, Go-Zero does not provide a whole set of trace link solutions. Developers can encapsulate the span structure of Go-Zero, build their own reporting system, and access link tracking tools such as Jaeger and Zipkin.

reference

  • go-zero trace
  • Introduction to Open Distributed Tracing with Jaeger implementation

The project address

Github.com/tal-tech/go…

If you like the article, please click on github star🤝