introduce

Add Prometheus monitoring middleware to microservices based on the Echo framework as a complete example.

What is a Prometheus monitoring interceptor/middleware?

The monitoring interceptor records Prometheus Metrics for each API request.

We will use RK-boot to start the Echo framework microservice.

Please visit rkdocs.net lilify. app/cn for the full tutorial

The installation

go get github.com/rookie-ninja/rk-boot
Copy the code

Quick start

Rk-boot integrates the following open source libraries by default.

  • Rk-prom serves as the Prometheus client boot library.

Attention! For the example to run smoothly, be sure to set the suffix of module to rk-demo in the go.mod file.

For example, module github.com/your-repo/rk-demo

1. Create the boot. Yaml

The boot.yaml file describes the original information about Echo framework startup. Rk-boot reads boot.yaml to start Echo.

To verify, we enabled the following options:

  • CommonService: commonService contains a set of common apis. details
  • PROM: Prometheus client.
  • Prometheus Middleware: Start Prometheus middleware.
---
echo:
  - name: greeter                     # Required
    port: 8080                        # Required
    enabled: true                     # Required
    prom:
      enabled: true                   # Optional, default: false
    commonService:
      enabled: true                   # Optional, default: false
    interceptors:
      metricsProm:
        enabled: true                 # Optional, default: false
Copy the code

2. Create a main. Go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main(a) {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}
Copy the code

3. Folder structure

$tree. └── Go. ├─ go. ├─ goCopy the code

4. Start the main. Go

go run main.go
Copy the code

5. Verify

$ curl -X GET localhost:8080/rk/v1/healthy
{"healthy":true}
Copy the code

Visit Prometheus client: http://localhost:8080/metrics

Visual monitoring

We have started Prometheus monitoring in the local process, all that remains is how to view the monitoring in a nice web page.

There are many tools out there, but we chose the easy, popular, free approach, which is Prometheus + Grafana.

Architecture diagram

Let’s see what the process looks like.

In fact, the principle is very simple, is [hijacking] API request, and record [time], [error code] and other information. After that, let Prometheus service take the initiative to pull data from the created service. Finally, let the Grafana service pull data from Prometheus and display the data table.

Quick start

1. Create Prometheus. Yml

We started by creating the Prometheus. Yml configuration file to enable the Prometheus service to pull data from localhost:8080/metrics.

In the following configuration, we did not specify /metrics because Prometheus uses /metrics by default to pull data.

Attention! We set the targets into the host. The docker. Internal: instead of localhost: 8080 to 8080, this is because the Prometheus inside the container, our service in the local.

This is a convenient way to access the port of the local machine from the container. explain

global:
  scrape_interval: 1s # Make scrape interval to 1s for testing.

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: 'rk-demo'
    scrape_interval: 1s
    static_configs:
      - targets: ['host.docker.internal:8080']
Copy the code

2. Start the Prometheus

We use Docker to start.

Prometheus uses port 9090 by default.

$ docker run \
      -p 9090:9090 \
      -v /<your path>/rk-demo/prometheus.yml:/etc/prometheus/prometheus.yml \
      prom/prometheus
Copy the code

3. Verify the Prometheus

Please start main.go as shown above and send a /rk/v1/healthy request.

Next, take a look at the data from the Prometheus service.

Go to localhost:9090 and search rk_demo_greeter_resCode, we can see that the data is already in Prometheus.

Visit: localhost:9090/targets and you can see that Prometheus has pulled data every second.

4. Start Grafana

Grafana uses port 3000 by default.

$ docker run -p 3000:3000 --name grafana grafana/grafana
Copy the code

Access: localhost: 3000

At this point, Grafana will let you log in with the default username and password as follows.

User name: admin Password: admin

5. Add Prometheus data source to Grafana

Grafana is just a Web UI tool, and in order to see the data report, we told Grafana where to look for Prometheus.

Select Prometheus as the data source.

Prometheus address, same as the above reason, because Grafana run in Docker, so, we don’t use localhost: 9090, but the host. The Docker. Internal: 9090.

6. Import the Dashboard

We can edit the Grafana Dashboard ourselves, but this is not an easy task. For rK-boot services, we provide a default [free] Grafana Dashboard template.

Note that the Dashboard imported here only matches [services created by the logic above].

Why? Because rK-boot default uses <App name >_<Entry name > _XXX as Prometheus’ metrics name.

If the user uses a different module or Entry name, the Dashboard Variable needs to be changed. We will show you how to use Grafana in a future article.

Move to the Dashboard Import page

Import Dashboard 15111, defined in: grafana.com/grafana/das…

Specify the Prometheus data source, which is the Prometheus we configured above.

Start monitoring

Attention! If the number of requests is too small, 0 is displayed. Please make more requests.

concept

Now that we can get the monitoring data from Grafana, let’s see what kind of monitoring data is added to the middleware in rK-Boot.

The monitoring interceptor logs the following monitoring by default.

Monitoring items The data type details
elapsedNano Summary RPC time-consuming
resCode Counter Counter based on RPC return code
errors Counter A counter based on RPC errors

All three monitors have the following labels.

The label details
entryName Echo entry name
entryType Echo entry type
realm Environment variables: REALM, eg: rk
region Environment variables: REGION, eg: Beijing
az Environment variables: AZ, eg: Beijing-1
domain Environment variables: DOMAIN, eg: prod
instance The local Hostname
appVersion fromAppInfoEntryTo obtain
appName fromAppInfoEntryTo obtain
restMethod The Http method. eg: GET
restPath The Http path. eg: /rk/v1/healthy
type Service type. eg: Echo
resCode Return code, eg: OK