preface

As mentioned in the previous article, observability construction under cloud native includes three aspects: Metrics, Logging events and link Tracing. Peter Bourgon briefly described the relationship between the above three observability signals in the article Metrics -tracing-and logging. That is, these three signals all have their own roles in the observability system, but they are related to each other to some extent. The following figure

​

When we receive alarms (usually Metrics and Tracing), we need to view abnormal indicators on the monitoring platform, and then query logs on the log platform based on a unique ID (for example, order number). Finally, we can locate abnormal services on the Trace platform through TraceId. It can be seen that in traditional monitoring platforms, the above three signals are often isolated, and we often need to cross multiple platforms to solve a problem. Observant observation platform aims to associate the above three signals to reduce maintenance cost and improve r&d efficiency.

In the observability platform, we can quickly associate one type of signals to another related type of signals, as shown in the figure below

The goal of Grafana is to make Grafana a complete observational platform that not only shows Metrics, Logs, and Traces to users, but also provides the ability to correlate them with each other and quickly solve problems using all three pillars.

This article will use Opentelemetry+Loki+Temp+Granafa to build an end-to-end observable platform.

architecture

The deployment of

This case code I have all uploaded to Github, please refer to opentElemetry – Tutorial.

  1. Install the loki-docker-driver log driver plug-in on the host node. Loki-docker-driver is used to collect container STDout logs.

    docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
Copy the code

  2. One-stop deployment of Loki, Tempo, OpenTelemetry Collector, Prometheus, Granafa.

    1. In/opentelemetry – tutorial/opentelemetry/collector – contrib/docker – compose directory

      docker-compose up -d
Copy the code

    2. The following containers should be started at this point

      docker-compose ps
Copy the code
      Name Command State Ports ------------------------------------------------------------------------------------------------------------------------ -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - docker compose_grafana_1 / run. The sh Up 0.0.0.0:3000 - > 3000 / TCP, : : : 3000 - > 3000 / TCP docker-compose_loki_1 /usr/bin/loki -config.file ... Up 0.0.0.0:3100->3100/ TCP,:::3100->3100/ TCP docker-compose_minio_1 /usr/bin/docker-entrypoint... Up 9000/tcp, 0.0.0.00:9001 ->9001/ TCP,:::9001->9001/ TCP docker-compose_mysql_1 docker-entrypoint.sh mysqld Up 0.0.0.0:3306->3306/ TCP,:::3306->3306/ TCP, 33060/ TCP docker-compose_otel-collector_1 /otelcontribcol --config=/... The Up 0.0.0.0:4317 - > 4317 / TCP, : : : 4317 - > 4317 / TCP, 0.0.0.0:55679 - > 55679 / TCP, : : : 55679 - > 55679 / TCP, 55680/tcp docker-compose_prometheus_1 /bin/prometheus --config.f ... Up 0.0.0.0:9090->9090/ TCP,:::9090->9090/ TCP docker-compose_tempo_1 / tempo-config. file=/etc/t... Up 0.0.0.0:3200->3200/ TCP,:::3200->3200/ TCP PUSHgateway /bin/ PUSHgateway Up 0.0.0.0:9091->9091/ TCP,:::9091->9091/ TCPCopy the code

  3. Build the Otel-Web-client image and install OpentElemetry – JavaAgent. jar inside the image (you can also mount it). This is a Web project whose sole purpose is to query user data from Mysql. I have uploaded it to Dockerhub, you can also follow these steps to create an image.

    1. In/opentelemetry – tutorial/opentelemetry/otel – spring – webmvc/client directory packaging, execution

      mvn package
Copy the code

    2. In/opentelemetry – tutorial/opentelemetry/otel – spring – webmvc/client directory to build image, is carried out

      docker build --tag otel-web-client .
Copy the code

  4. Deploy otel-web-client with Docker-compose and mount your own Appliaction.yml

    1. In/opentelemetry – tutorial/opentelemetry/otel – spring – webmvc/web directory

      docker-compose up -d
Copy the code

    2. The following containers should be started at this point

      docker-compose ps
Copy the code
      Name Command State Ports --------------------------------------------------------------------------------------------------------- web_otel-web-client_1 sh -c java -javaagent:./op ... The Up 0.0.0.0:8081 - > 8081 / TCP, : : : 8081 - > 8081 / TCPCopy the code

View Logs and Traces with Granafa

  1. Loki-docker-driver can collect log containers by clicking logBrowser in Explore. If you select container_name, loki-docker-driver can collect log containers. Next, select web_otel-web-client_1 to view the logs.

  2. In the Log, we have automatically injected trace_id and other information through opentElemetry – Java-instrumentation Logger MDC auto-instrumentation. You can then use trace_id to correlate to Tempo to query for Trace.

  3. See the Trace

The last

Logs to Logs is implemented with the above functions, and Granfa Tempo can also implement this functionality by configuring data sources (currently Loki only). Since Loki and Prometheus use lable to query data, Metrics to Logs or Logs to Metrics can be achieved by using the same Lable.

reference

How do I associate metrics, logs, and traces in Grafana