The overall structure

SpringBoot’s actuator provides monitoring endpoints.

Prometheus is a monitoring system that obtains monitoring data from Springboot, stores it in the form of timing data, and provides monitoring data query service.

Grafana is a professional UI dashboard system that supports a wide range of data sources, including Prometheus, from which data can be easily retrieved and displayed using a dashboard.

Micrometer was introduced in SpringBoot 2 to make it easier to connect to various monitoring systems, including Prometheus.

So the overall structure is:

  • Springboot (Micrometer) generates monitoring data.
  • Prometheus obtains and stores monitoring data of SpringBoot applications, and provides data query services.
  • Grafana connects to the Prometheus data source, invokes its data query service, and presents it in a professional dashboard UI.

Practical steps

  • Create an application – as a monitoring target, generate monitoring data.

  • Integrated measurement library Micrometer — for docking with monitoring system Prometheus.

  • The deployment of Prometheus

  • Configure Prometheus — Monitor previously created SpringBoot applications and learn about Prometheus’ query services.

  • Deploy Grafana

  • Add Prometheus data source

  • Add JVM Monitoring dashboard – shows the JVM status of previous SpringBoot applications.

  • Self-defined monitoring indicators – Our own monitoring indicators are the underlying basic data, and we need to write our own codes for business-related indicators.

  • Dynamic change monitoring target – It is not appropriate for Prometheus to modify the configuration file and restart if the monitoring target changes, and dynamic configuration is required.

1. Create an application integration Micrometer

Create a minimal SpringBoot application with micrometer dependencies.

Pom. XML:

<?xml version="1.0" encoding="UTF-8"? >
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>2.2.4. RELEASE</version>
		<relativePath/> <! -- lookup parent from repository -->
	</parent>
	<groupId>com.example</groupId>
	<artifactId>springboot2demo</artifactId>
	<version>0.0.1 - the SNAPSHOT</version>
	<name>springboot2demo</name>
	<description>Demo project for Spring Boot</description>

	<properties>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-actuator</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>io.micrometer</groupId>
			<artifactId>micrometer-registry-prometheus</artifactId>
			<version>1.1.4</version>
		</dependency>

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
			<exclusions>
				<exclusion>
					<groupId>org.junit.vintage</groupId>
					<artifactId>junit-vintage-engine</artifactId>
				</exclusion>
			</exclusions>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>

</project>

Copy the code

application.properties

Spring. The application. The name = springboot2demo # open all physical service management endpoints. Web. Exposure. Include = * # name added to the meter to be used In the tag # so that Prometheus according to the application name to distinguish the different service management metrics. Tags. Application = ${spring. Application. The name}Copy the code

Add beans to the startup class to monitor JVM performance metrics:

package com.example.springboot2demo;

import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryCustomizer;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;

@SpringBootApplication
public class Springboot2demoApplication {

    public static void main(String[] args) {
        SpringApplication.run(Springboot2demoApplication.class, args);
    }

    @Bean
    MeterRegistryCustomizer<MeterRegistry> configurer(
            @Value("${spring.application.name}") String applicationName) {
        return (registry) -> registry.config().commonTags("application", applicationName); }}Copy the code

Start the service.

To view monitoring endpoint information:

2. The deployment of Prometheus

Website:

prometheus.io/

You can download the installation package to install it, but it is extremely slow and almost impossible to download.

Docker can be used for deployment, because there are Docker images in China, so the speed is very fast.

Start with Docker:

$ docker run --name prometheus -d9090-9090 - p 127.0.0.1: PROM/Prometheus
Copy the code

When this is done, take a look at the Prometheus interface.

http://localhost:9090/targets is a target monitoring list page:

http://localhost:9090/graph is a query console, also has a simple chart shows:

Now there is no docking application, you can see the details after docking.

Prometheus + Springboot application

To monitor applications, add application information in the Prometheus configuration file.

Path of the configuration file in the container: /etc/prometheus.

Take a look at the default contents of the configuration file:

$ docker execID - it [container] cat/etc/Prometheus/Prometheus yml
Copy the code

The red box is the part we want to focus on, and we can add our application in this form.

The following items need to be added:

  - job_name: 'springboot_app'
    scrape_interval: 5s
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['192.168.31.6:8080']
        "labels": {
            "instance": "springboot2-A".
            "service": "springboot2-A-service"
        }
Copy the code

Metrics_path Specifies the path to monitor the endpoint.

Targets specifies the IP port of the application. IP is used instead of localhost because Prometheus is running on a container and using localhost accesses the inside of the container.

You can copy the configuration file inside the container instead of modifying the configuration file inside the container. After modifying the configuration file, restart the container and mount the modified configuration file.

Copy the configuration file from the container:

$Docker cp/container ID: / etc/Prometheus/Prometheus yml.
Copy the code

Modify the configuration file, add the configuration, the final content:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'springboot_app'
    scrape_interval: 5s
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['192.168.31.6:8080']
        "labels": {
            "instance": "springboot2-A".
            "service": "springboot2-A-service"
        }
Copy the code

Stop the previous container and restart:

$ docker run --name prometheus -d \
    -p 9090:9090 \
    -v [PATH]/prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus
Copy the code

Access control list page http://localhost:9090/targets can see our application:

Click on the endpoint link to see monitoring data such as:

Enter the query console page, http://localhost:9090/graph, a criterion can query, for example http_server_requests_seconds_sum, effect:

4. The deployment Grafana

Docker mode running:

$ docker run -d \
    -p 3000:3000 \
    --name=grafana \
    grafana/grafana
Copy the code

After the startup, access: http://localhost:3000, default user name password admin/admin.

5. Add Prometheus data source

6. Display application JVM information

The JVM dashboard is already available in Grafana, so we can import it and use it.

The dashboard number is 4701.

At this point, the whole process of Prometheus + Grafana + Springboot has run.

However, these indicators are general indicators at the bottom, and there will be individual requirements at the business level. Here we define some monitoring indicators by ourselves.

7. Customize monitoring indicators

Requirement: Monitors the number of requests for all interfaces.

Adding dependencies to an application:

		<dependency>
			<groupId>org.aspectj</groupId>
			<artifactId>aspectjrt</artifactId>
			<version>1.9.4</version>
		</dependency>
		<dependency>
			<groupId>org.aspectj</groupId>
			<artifactId>aspectjweaver</artifactId>
			<version>1.9.4</version>
		</dependency>
		<dependency>
			<groupId>cglib</groupId>
			<artifactId>cglib</artifactId>
			<version>3.2.12</version>
		</dependency>
Copy the code

Count interface requests using AOP:

package com.example.springboot2demo;

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import org.aspectj.lang.JoinPoint;
import org.aspectj.lang.annotation.AfterReturning;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Before;
import org.aspectj.lang.annotation.Pointcut;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import javax.annotation.PostConstruct;

@Component
@Aspect
public class APICounterAop {

    @Pointcut("execution(public * com.example.springboot2demo.*.*(..) )")
    public void pointCut(a) {
    }

    ThreadLocal<Long> startTime = new ThreadLocal<>();

    @Autowired
    MeterRegistry registry;
    private Counter counter;

    @PostConstruct
    private void init(a) {
        counter = registry.counter("requests_total"."status"."success");
    }

    @Before("pointCut()")
    public void doBefore(JoinPoint joinPoint) throws Throwable {
        System.out.println("do before");
        counter.increment(); // Request count
    }

    @AfterReturning(returning = "returnVal", pointcut = "pointCut()")
    public void doAfterReturning(Object returnVal) {
        System.out.println("do after"); }}Copy the code

Create a test interface:

package com.example.springboot2demo;

import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class TestController {
    @RequestMapping("/hello")
    public String hello(a) {
        return "hello"; }}Copy the code

Restart the application, visit the test interface a few more times, and then view the Application Monitoring endpoint page in Prometheus to see the monitoring results:

And then we showed that in Grafana.

8. Dynamically change the monitoring target

The monitoring targets defined in the Prometheus configuration file above are static and require a restart after changing the configuration file.

If services change or are added, it is certainly not appropriate to restart Prometheus frequently.

Prometheus provides dynamic loading by putting service information into a separate external file specified in the Prometheus configuration, which Prometheus automatically reloads when its contents change.

Service information configuration file For example:

[{"targets": [
            "192.168.31.6:8080"]."labels": {
            "instance": "springboot2-A"."service": "springboot2-A-service"}}]Copy the code

From the Prometheus configuration file:

.
  - job_name: 'springboot_app'
    scrape_interval: 5s
    metrics_path: '/actuator/prometheus'
    file_sd_configs:
    - files:
      - /home/*.json
      refresh_interval: 1m
Copy the code

Directory to mount the service information profile when starting the Prometheus container:

$ docker run --name prometheus -d -p 9090:9090 \
	-v [PATH]/prometheus.yml:/etc/prometheus/prometheus.yml \
	-v [PATH]:/home \
	prom/prometheus
Copy the code

Recommended reading:

  • Remote live architecture
  • Canal is used to achieve data heterogeneity
  • How does MySQL Order by work?
  • Ali open source distributed transaction framework Seata
  • Distributed unique ID generation scheme