Installation and configuration of the Prometheus monitoring system

Introduction of 0.

Prometheus is an open source system monitoring and alert kit originally built on SoundCloud. Since its inception in 2012, Many companies and organizations have adopted Prometheus, which has a very active developer and user community. It is now a separate open source project and maintained independently of any company. To underscore this point and clarify the governance structure of the project, Prometheus joined the Cloud Native Computing Foundation in 2016, the second hosted project after Kubernetes.

• Powerful multi-dimensional data model.
• Time series data is distinguished by metric names and key-value pairs.
• All metrics can be set to any multi-dimensional label.
• The data model is more arbitrary and does not need to be deliberately set to a dot-delimited string.
• You can aggregate, slice, and slice the data model.
• Supports double precision floating point types, and tags can be set to full Unicode.
• Flexible and powerful query statement (PromQL) : Multiple metrics can be multiplied, added, joined, and fractional in the same query statement.
• Easy to manage: Prometheus Server is a single binary that works directly locally and does not rely on distributed storage.
• Efficiency: The average sampling point is only 3.5 bytes, and a Prometheus Server can handle millions of metrics.
• Use pull mode to collect time series data, not only for the benefit of native testing but also to avoid problematic servers pushing bad metrics.
• Time series data can be pushed to the Prometheus server by means of push gateway.
• Targets to monitor can be obtained through service discovery or static configuration.
• There are a variety of visual graphical interfaces.
• Easy to scale.

1. Basic environment

Environment/Components	version	Download address
The operating system	CentOS7.6	Archive.kernel.org/centos-vaul…
Prometheus	2.25.0	Github.com/prometheus/…
Go	1.16	Golang.org/dl/go1.16.l…
Grafana	yum install latest	Mirror.tuna.tsinghua.edu.cn/help/grafan…

2. Install the Prometheus

2.1 installation

Gz -c /opt mv /opt/ Prometheus -2.25.0. Linux-amd64 /opt/ PrometheusCopy the code

2.2 Configuring automatic startup upon Startup

vim /usr/lib/systemd/system/prometheus.service
Copy the code

[Unit]
Description=prometheus service

[Service]
User=root
ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.path=/opt/prometheus/data

TimeoutStopSec=10
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
Copy the code

systemctl daemon-reload
systemctl enable prometheus
Copy the code

2.3 Starting services

systemctl start prometheus
systemctl status prometheus
Copy the code

2.4 validation

To access the monitoring page provided by Prometheus, open IP:9090.

3. Install Grafana

Prometheus’s default page may not be as intuitive, we can install Grafana to make the monitoring look more intuitive.

3.1 Configure the yum source of Tsinghua University

Open a browser input address: https://mirror.tuna.tsinghua.edu.cn/help/grafana/, part copy CentOS/Redhat users:

vim /etc/yum.repos.d/grafana.repo
Copy the code

[grafana]
name=grafana
baseurl=https://mirrors.tuna.tsinghua.edu.cn/grafana/yum/rpm
repo_gpgcheck=0
enabled=1
gpgcheck=0
Copy the code

yum makecache
Copy the code

3.2 installation Grafana

yum install grafana -y
Copy the code

3.3 Starting Services

systemctl daemon-reload
systemctl enable grafana-server
systemctl start grafana-server
Copy the code

3.4 access Grafana

The default user name and password are admin, and you will need to change the default login password on the first login.

3.5 Adding a Prometheus Data source

Click “Add Your First Data Source” in the main interface and select Prometheus:

On the Dashboards page, select Prometheus 2.0 Stats to Import:

On the Settings page, fill in the Prometheus address and save:

Switch to “Prometheus 2.0 Stats” we just added to see the entire monitoring page:

4. Some common monitoring examples

4.1 Monitoring a Linux Machine (node_exporter)

• Download address:

https://github.com/prometheus/node_exporter/releases/download/v1.1.1/node_exporter-1.1.1.linux-amd64.tar.gz

• Install node_exporter on monitored machine:

Gz -c /opt mv /opt/ node_friend-1.1.1.linux-amd64 /opt/ node_friend-1.1.1.linux-amd64 /opt/ node_friend-1.1.1.linux-amd64 /opt/ node_friend-1.1.1.linux-amd64Copy the code

• Start the service:

Configure automatic startup upon startup:

vim /usr/lib/systemd/system/node_exporter.service
Copy the code

[Unit]
Description=node_exporter service

[Service]
User=root
ExecStart=/opt/node_exporter/node_exporter

TimeoutStopSec=10
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
Copy the code

systemctl daemon-reload
systemctl enable node_exporter
Copy the code

Start the service:

systemctl start node_exporter
systemctl status node_exporter
Copy the code

• Add monitoring items to the Prometheus configuration file:

vim /opt/prometheus/prometheus.yml
Copy the code

  - job_name: 'linux-node'
    static_configs:
    - targets: ['192.168.0.112:9100']
      labels:
        instance: linux-node1
Copy the code

Restart the Prometheus:

systemctl restart prometheus
Copy the code

• Grafana imported the painted dashboard:

Json file download address:

node-exporter_rev5.json

Change the name, select the data source we created before, and click Import:

Grafana’s default plugins directory is /var/lib/grafana/plugins. You can decompress the downloaded plugins to this directory and restart grafana

Plug-in download address:

grafana-piechart-panel

unzip -q grafana-piechart-panel-5f249d5.zip -d /var/lib/grafana/plugins/
systemctl restart grafana-server
Copy the code

View installed plug-ins:

/usr/sbin/grafana-cli plugins ls
installed plugins:
grafana-piechart-panel @ 1.3.3
Copy the code

Tips: There is another command-line way to install plug-ins:

grafana-cli plugins install grafana-piechart-panel
grafana-cli plugins install digiapulssi-breadcrumb-panel
grafana-cli plugins install grafana-polystat-panel

systemctl restart grafana-server
Copy the code

Refresh the Grafana page and you can see the node monitor we just set:

4.2 monitor MySQL (mysqld_exporter)

• Create a monitoring user on the monitored server:

GRANT REPLICATION CLIENT, PROCESS ON *.* TO 'mysqld_exporter'@'127.0.0.1' identified by 'mysqld_exporter';
GRANT SELECT ON performance_schema.* TO 'mysqld_exporter'@'127.0.0.1';
flush privileges;
Copy the code

• Download address:

https://github.com/prometheus/mysqld_exporter/releases/download/v0.12.1/mysqld_exporter-0.12.1.linux-amd64.tar.gz

• Install mysqLD_exporter on monitored machines:

Mkdir -p /opt tar ZXF mysqld_exporter-0.12.1.linux-amd64.tar.gz -c /opt mv /opt/mysqld_exporter-0.12.1.linux-amd64 /opt/mysqld_exportercd /opt/mysqld_exporter
Copy the code

• Set user to the database login user and password to the password of the user:

Vim.my.cnf [client] host=127.0.0.1 port=3306 user=mysqld_exporter password=mysqld_exporterCopy the code

• Configure automatic startup and service startup upon startup:

vim /usr/lib/systemd/system/mysqld_exporter.service
Copy the code

[Unit]
Description=mysqld_exporter service

[Service]
User=root
ExecStart=/opt/mysqld_exporter/mysqld_exporter --config.my-cnf=/opt/mysqld_exporter/.my.cnf

TimeoutStopSec=10
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
Copy the code

systemctl daemon-reload
systemctl enable mysqld_exporter
systemctl start mysqld_exporter
systemctl status mysqld_exporter
Copy the code

• Add mysql monitoring to Prometheus configuration file and restart:

vim /opt/monitor/prometheus/prometheus.yml
Copy the code

  - job_name: 'mysqld-node'
    static_configs:
    - targets: ['192.168.1.235:9104']
      labels:
        instance: mysqld-node1
Copy the code

Restart the service:

systemctl restart prometheus
Copy the code

Template Json file download address:

mysql_rev1.json

• Import the dashboard that has been drawn and select the data source that has just been createdprometheusData source:

Instead of uploading the JSON file with grafana ID, enter id 7362 and click the load button.

4.3 monitor the Redis (redis_exporter)

• Download address:

https://github.com/oliver006/redis_exporter/releases/download/v1.20.0/redis_exporter-v1.20.0.linux-amd64.tar.gz

• Install redis_exporter:

Mkdir -p /opt/redis_exporter tar ZXF redis_exporter-v1.20.0.linux-amd64.tar.gz -c /opt/redis_exporterCopy the code

• Configure automatic startup and service startup upon startup:

vim /usr/lib/systemd/system/redis_exporter.service
Copy the code

[Unit] Description=redis_exporter service [Service] User=root ExecStart=/opt/redis_exporter/redis_exporter -redis.addr Redis: / / 192.168.1.235:6379 - redis. The password 123456 TimeoutStopSec = 10 Restart = on - failure RestartSec = 5 [Install] WantedBy=multi-user.targetCopy the code

systemctl daemon-reload
systemctl enable redis_exporter
systemctl start redis_exporter
systemctl status redis_exporter
Copy the code

• Add Redis monitoring to Prometheus configuration file and restart:

vim /opt/prometheus/prometheus.yml
Copy the code

  - job_name: 'redis-node'
    static_configs:
    - targets: ['192.168.0.116:9121']
      labels:
        instance: redis-node1
Copy the code

Restart the service:

systemctl restart prometheus
Copy the code

• Grafana imported the painted dashboard:

Json file download address:

Grafana ID: 4074 或者 14091

4.4 monitor the Elasticsearch (elasticsearch_exporter)

• Download address:

https://github.com/justwatchcom/elasticsearch_exporter/releases/download/v1.1.0/elasticsearch_exporter-1.1.0.linux-amd64 .tar.gz

• Install elasticsearch_exporter:

Mkdir -p /opt tar ZXF elasticSearch_tariff-1.1.linux-amd64.tar. gz -c /opt/ mv / opt/elasticsearch_exporter - 1.1.0. Linux - amd64 / opt/elasticsearch_exporterCopy the code

• Configure automatic startup and service startup upon startup:

vim /usr/lib/systemd/system/elasticsearch_exporter.service
Copy the code

[Unit] Description=elasticsearch_exporter service [Service] User=root ExecStart = / opt/elasticsearch_exporter elasticsearch_exporter -- es. Uri = http://elastic:[email protected]:9200 TimeoutStopSec=10 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.targetCopy the code

vim /usr/lib/systemd/system/elasticsearch_exporter.service
systemctl daemon-reload
systemctl enable elasticsearch_exporter
systemctl start elasticsearch_exporter
systemctl status elasticsearch_exporter
Copy the code

• Add ElasticSearch to the Prometheus configuration file and restart:

vim /opt/prometheus/prometheus.yml
Copy the code

  - job_name: 'elasticsearch-node'
    static_configs:
    - targets: ['192.168.0.116:9114']
      labels:
        instance: elasticsearch-node1
Copy the code

• The following describes the configuration items that can be monitored over the PUBLIC network HTTPS protocol:

  - job_name: 'es-node'
    static_configs:
    - targets: ['mmbapoc.zhizhangyi.com:9070']
      labels:
        instance: es-node1
    scheme: https
    metrics_path: /es/node1
Copy the code

Here’s an explanation: Scheme specifies the protocol for pull data to be HTTPS, and metrics_path specifies the Nginx reverse proxy match path on the Exporter machine. See below.

• Nginx reverse proxy configuration:

The location/es/node1 {proxy_pass http://127.0.0.1:9114/metrics; proxy_set_header Host$http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
Copy the code

To explain, Nginx is deployed on the Intranet, Prometheus requests Nginx through the public network, and Nginx reversely proxies to the Exporter server.

• Restart the service:

systemctl restart prometheus
Copy the code

• Grafana imported the painted dashboard:

Grafana ID: 2322

4.5 monitor the Rabbitmq (rabbitmq_exporter)

• Download address:

https://github.com/kbudde/rabbitmq_exporter/releases/download/v1.0.0-RC8/rabbitmq_exporter-1.0.0-RC8.linux-amd64.tar.gz

• Install rabbitmq_exporter:

Mkdir -p /opt tar ZXF rabbitmq_tarix-1.0.0-rc8.linux-amd64.tar. gz -c /opt mv / opt/rabbitmq_exporter - 1.0.0 - RC8. Linux - amd64 / opt/rabbitmq_exporterCopy the code

• Configure automatic startup and service startup upon startup:

vim /usr/lib/systemd/system/rabbitmq_exporter.service
Copy the code

[Unit]
Description=rabbitmq_exporter service

[Service]
User=root
ExecStart=/opt/rabbitmq_exporter/rabbitmq_exporter -config-file /opt/rabbitmq_exporter/config.json

TimeoutStopSec=10
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
Copy the code

vim /opt/rabbitmq_exporter/config.json

{
    "rabbit_url": "http://127.0.0.1:15672"."rabbit_user": "rabbitadmin"."rabbit_pass": "123456"."publish_port": "9119"."publish_addr": ""."output_format": "TTY"."ca_file": "ca.pem"."cert_file": "client-cert.pem"."key_file": "client-key.pem"."insecure_skip_verify": false."exlude_metrics": []."include_queues": ". *"."skip_queues": ^ "$"."skip_vhost": ^ "$"."include_vhost": ". *"."rabbit_capabilities": "no_sort,bert"."enabled_exporters": [
            "exchange"."node"."overview"."queue"]."timeout": 30."max_queues": 0}Copy the code

vim /usr/lib/systemd/system/rabbitmq_exporter.service
systemctl daemon-reload
systemctl enable rabbitmq_exporter
systemctl start rabbitmq_exporter
systemctl status rabbitmq_exporter
Copy the code

• Add RabbitMQ monitoring to the Prometheus configuration file and restart:

vim /opt/prometheus/prometheus.yml
Copy the code

  - job_name: 'rabbitmq-node'
    static_configs:
    - targets: ['192.168.0.116:9119']
      labels:
        instance: rabbitmq-node1
Copy the code

• The following describes the configuration items that can be monitored over the PUBLIC network HTTPS protocol:

  - job_name: 'rabbitmq-node'
    static_configs:
    - targets: ['mmbapoc.zhizhangyi.com:9070']
      labels:
        instance: rabbitmq-node1
    scheme: https
    metrics_path: /rabbitmq/node1
Copy the code

Here’s an explanation: Scheme specifies the protocol for pull data to be HTTPS, and metrics_path specifies the Nginx reverse proxy match path on the Exporter machine. See below.

• Nginx reverse proxy configuration:

The location/rabbitmq/node1 {proxy_pass http://127.0.0.1:9119/metrics; proxy_set_header Host$http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
Copy the code

To explain, Nginx is deployed on the Intranet, Prometheus requests Nginx through the public network, and Nginx reversely proxies to the Exporter server.

• Restart the service:

systemctl restart prometheus
Copy the code

• Grafana imported the painted dashboard:

Grafana ID: 4371

4.6 monitor JMX (jmx_exporter)

The following configuration is closely related to the business and does not have extensive reference!

Source address | compiled download address

Upload the above JAR file to the project directory. Take MMB-service as an example, and store the JAR package in MMB-service /classes/jmx_prometheus_javaagent-0.15.0.jar.

Modify mMb-service startup parameters:

• The original:

nohup java -Dconfig.cluster=${ZOOKEEPER_URL}-Xmx1G -Xms1G -classpath \ ./:.. /lib/* com.uusafe.analyze.mmba.service.App >/dev/null 2>&1 &Copy the code

• Revised:

Nohup Java - javaagent:. / jmx_prometheus_javaagent - 0.15.0. Jar = 9501:. / config_jmx yml \ - Dcom. Sun. Management jmxremote. SSL =false \
    -Dcom.sun.management.jmxremote.authenticate=false \
    -Dcom.sun.management.jmxremote.port=5555 \
    -Dconfig.cluster=${ZOOKEEPER_URL}\ -Xmx1G -Xms1G \ -classpath ./:.. /lib/* com.uusafe.analyze.mmba.service.App >/dev/null 2>&1 &Copy the code

The main differences are:

- javaagent:. / jmx_prometheus_javaagent - 0.15.0. Jar = 9501:. / config_jmx yml \ - Dcom. Sun. Management jmxremote. SSL =false \
    -Dcom.sun.management.jmxremote.authenticate=false \
    -Dcom.sun.management.jmxremote.port=5555 \
Copy the code

Config_jmx. yml Configuration file contents:

-- startDelaySeconds: 0 hostPort: 127.0.0.1:5555 SSL:false
lowercaseOutputName: false
lowercaseOutputLabelNames: false
Copy the code

Ensure that the ports are consistent and the Service Java service is enabled.

Configure the Nginx agent:

The location/Java/mmba/service {proxy_pass http://127.0.0.1:9501/metrics; proxy_set_header Host$http_host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
    }
Copy the code

This path is a custom path that matches the metrics_path value in the configuration file for Prometheus below.

Overloading Nginx:

/opt/nginx/sbin/nginx -s reload
Copy the code

Modify the Prometheus configuration file:

sudo vim prometheus.yml
Copy the code

  - job_name: 'mmba-service'
    static_configs:
    - targets: ['mmba116.test.com:9070']
      labels:
        instance: mmba-service
    scheme: https
    metrics_path: /java/mmba/service
Copy the code

Restart the Prometheus service:

sudo systemctl restart prometheus
Copy the code

Log in to the Grafana panel to configure the dashboard:

Import the template with grafana ID 10519.

Example screenshot:

5. Common Websites

• prometheus download: https://prometheus.io/download/
• prometheus exporter: https://prometheus.io/docs/instrumenting/exporters/
• grafana dashboard: https://grafana.com/dashboards
• grafana plugins: https://grafana.com/plugins