Monitor middleware and Grafana usage

Grafana use

Adding a data source

Creating a Test Dashboard

Network adapter Test Data

A panel of operation

Set the unit
The panel name
Curve the alias
Curve of the sort
Curve to copy
Curve in silence
The panel to copy
Setting the alarm Line

Set the tables

Query using variables

Nested variable

Dashboards need to load variables when they open, which is one of the reasons they slow down

Import the Node_exporter template from the Dashboard store

Address grafana.com/grafana/das…

Two import modes
- The url to import
- Importing a JSON File
Grafana.com/grafana/das…

Add monitoring for Grafana

Grafana will also expose indicators to add collection

http://grafana.prome.me:3000/metrics
Copy the code

Added to the Prometheus collection pool

- the targets: - 172.20.70.205:3000Copy the code

Import mall dashboard

Address grafana.com/grafana/das…

Example Add the monitoring function of mysql

Deploy mysql_exporter on your mysql machine

Project address: github.com/prometheus/…

Deploy mysql_exporter using Ansible

Ansible-playbook -i host_file service_deploy.yaml -e "TGZ =mysqld_exporter-0.12.1.linux-amd64.tar.gz" -e "app=mysqld_exporter"Copy the code

I post the Ansible deployment file:

service_deploy.yaml 

- name:  install
  hosts: all
  user: root
  gather_facts:  false
  vars:
      local_path: /opt/tgzs
      app_dir: /opt/app

  tasks:
      - name: mkdir
        file: path={{ app_dir }}/{{ app }} state=directory
      - name: mkdir
        file: path={{ local_path }} state=directory


      - name: copy  config and service
        copy:
          src: '{{ item.src }}'
          dest: '{{ item.dest }}'
          owner: root
          group: root
          mode: 0644
          force: true

        with_items:
          - { src: '{{ local_path }}/{{ tgz }}', dest: '{{ local_path }}/{{ tgz }}' }
          - { src: '{{ local_path }}/{{ app }}.service', dest: '/etc/systemd/system/{{ app }}.service' }

        register: result
      - name: Show debug info 
        debug: var=result verbosity=0

      - name: tar gz
        shell: rm -rf /root/{{ app }}* ; \
          tar xf {{ local_path }}/{{ tgz }} -C /root/ ; \
          /bin/cp -far /root/{{ app }}*/* {{ app_dir }}/{{ app }}/ \

        register: result
      - name: Show debug info
        debug: var=result verbosity=0

      - name: restart service
        systemd:
          name: "{{ item }}"
          state: restarted
          daemon_reload: yes
          enabled: yes
        with_items:
          - '{{ app }}'
        register: result

      - name: Show debug info
        debug: var=result verbosity=0

Copy the code

After the deployment, the service is not started. The following error message is displayed

Mar 30 10:41:10 prome_master_01 mysqld_exporter[30089]: time="2021-03-30T10:41:10+08:00" level=fatal msg="failed reading ini file: open .my.cnf: no such file or directory" source="mysqld_exporter.go:264"

Copy the code

Create and authorize a collection user

mysql -uroot -p123123

CREATE USER 'exporter'@'%' IDENTIFIED BY '123123' ;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';
FLUSH PRIVILEGES;


Copy the code

Method 1: Use the DATA_SOURCE_NAME environment variable in the service file of mysqLD_exporter

< class = "class" style = "margin-bottom: 0pt; margin-bottom: 0pt; margin-bottom: 0ptCopy the code

Restart mysqLD_Exporter

systemctl daemon-reload
systemctl restart mysqld_exporter

Copy the code

Check the mysqLD_exporter log

[root@prome_master_01 logs]# systemctl status mysqld_vendor -l ● mysqld_vendor. Service -mysqld_vendor Loaded: loaded (/etc/systemd/system/mysqld_exporter.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2021-03-30 10:53:08 CST; 11min ago Main PID: 30158 (mysqld_exporter) CGroup: / system.slicer /mysqld_exporter/mysqld_exporter Mar 30 10:53:08 prome_master_01 mysqld_exporter[30158]: Time =" 2021-03-30t10:53:08 +08:00" level=info MSG ="Starting mysqld_exporter (version=0.12.1, branch=HEAD, revision=48667bf7c3b438b5e93b259f3d17b70a7c9aff96)" source="mysqld_exporter.go:257" Mar 30 10:53:08 prome_master_01 mysqld_exporter[30158]: Time =" 2021-03-30t10:53:08 +08:00" level=info MSG ="Build context (go=go1.12.7, user=root@0b3e56a7bc0a, date=20190729-12:35:58)" source="mysqld_exporter.go:258" Mar 30 10:53:08 prome_master_01 mysqld_exporter[30158]: time="2021-03-30T10:53:08+08:00" level=info msg="Enabled scrapers:" source="mysqld_exporter.go:269" Mar 30 10:53:08 prome_master_01 mysqld_exporter[30158]: time="2021-03-30T10:53:08+08:00" level=info msg=" --collect.global_status" source="mysqld_exporter.go:273" Mar 30 10:53:08 prome_master_01 mysqld_exporter[30158]: time="2021-03-30T10:53:08+08:00" level=info msg=" --collect.global_variables" source="mysqld_exporter.go:273" Mar 30 10:53:08 prome_master_01 mysqld_exporter[30158]: time="2021-03-30T10:53:08+08:00" level=info msg=" --collect.slave_status" source="mysqld_exporter.go:273" Mar 30 10:53:08 prome_master_01 mysqld_exporter[30158]: time="2021-03-30T10:53:08+08:00" level=info msg=" --collect.info_schema.innodb_cmp" source="mysqld_exporter.go:273" Mar 30 10:53:08 prome_master_01 mysqld_exporter[30158]: time="2021-03-30T10:53:08+08:00" level=info msg=" --collect.info_schema.innodb_cmpmem" source="mysqld_exporter.go:273" Mar 30 10:53:08 prome_master_01 mysqld_exporter[30158]: time="2021-03-30T10:53:08+08:00" level=info msg=" --collect.info_schema.query_response_time" source="mysqld_exporter.go:273" Mar 30 10:53:08 prome_master_01 mysqld_exporter[30158]: time="2021-03-30T10:53:08+08:00" level=info msg="Listening on :9104" source="mysqld_exporter.go:283"Copy the code

Method 2: Start the my.cnf service

Environment=DATA_SOURCE_NAME='exporter:123123@(localhost:3306)/'
Copy the code

Add the mysqLD_exporter collection to the collection pool

Static_configs: -targets: -172.20.70.205:9104Copy the code

Import mysqld-dashboard on grafana

Address grafana.com/grafana/das…

Start collection items as required

Add process monitor process_exporter

Experience atop

yum -y install atop
Copy the code

Deploy process-Exporter on the machine

Project address: github.com/ncabatoff/p…

Deploy Process-Exporter using Ansible

Ansible-playbook -i host_file service_deploy.yaml -e "TGZ = process-promise-0.7.5.linux-amd64.tar. gz" -e "app=process-exporter"Copy the code

Prepare the configuration file process-xue.yaml

Specify how the collection process works. The following example represents all cmDlines

cat <<EOF >/opt/app/process-exporter/process-exporter.yaml
process_names:
  - name: "{{.Comm}}"
    cmdline:
    - '.+'
EOF
Copy the code

Add the Process-Exporter collection to the collection pool

Static_configs: -targets: -172.20.70.205:9256-172.20.70.215:9256Copy the code

Import the Process-Exporter Dashboard on Grafana

Address grafana.com/grafana/das…
Optional grafana.com/grafana/das…
- Variable substitution
- label_values(namedprocess_namegroup_num_procs, instance)
- label_values(namedprocess_namegroup_cpu_seconds_total{instance=~”$host”},groupname)

Black box probe blackbox_exporter

Deploy blackbox_exporter on the machine

Project address: github.com/prometheus/…

Deploy blackbox_exporter using Ansible

Ansible-playbook -i host_file service_deploy.yaml -e "TGZ = blackbox_shipper -0.18.0.linux-amd64.tar.gz" -e "app=blackbox_exporter"Copy the code

Page to blackbox

Address: http://172.20.70.205:9115/

Page access target HTTP probe

http://172.20.70.205:9115/probe?target=https://www.baidu.com&module=http_2xx&debug=true
Copy the code

Results the interpretation

Logs for the probe: Ts = 2021-03-30t07:28:17.405299592z Caller =main.go:304 module=http_2xx target=https://www.baidu.com level=info MSG ="Beginning probe" probe= HTTP timeout_seconds= 119.5ts = 2021-03-30t07:28:17.40563586z caller=http.go:342 module=http_2xx target=https://www.baidu.com level=info msg="Resolving target address" ip_protocol=ip6 Ts = 2021-03-30t07:28:17.414113889z caller=http.go:342 module=http_2xx target=https://www.baidu.com level=info MSG ="Resolved target address" IP =110.242.68.4 ts= 2021-03-30t07:28:17.414249109z Caller =client.go:252 module=http_2xx Target =https://www.baidu.com level=info MSG ="Making HTTP request" URL =https://110.242.68.4 host=www.baidu.com Ts = 2021-03-30t07:28:17.459576352z Caller =main.go:119 module=http_2xx target=https://www.baidu.com level=info MSG ="Received HTTP response" status_code=200 ts= 2021-03-30t07:28:17.459696667z caller=main.go:119 module=http_2xx target=https://www.baidu.com level=info msg="Response timings for roundtrip" roundtrip=0 Start = 2021-03-30 T15: cast. 414370915 + 08:00 dnsDone = 2021-03-30 T15: cast. 414370915 + 08:00 ConnectDone = 2021-03-30 T15: cast. 423500145 + 08:00 gotConn = 2021-03-30 T15: cast. 449441723 + 08:00 ResponseStart = 2021-03-30 T15: cast. 459467652 + 08:00 end = 2021-03-30 T15: cast. 459684294 + 08:00 Ts = 2021-03-30t07:28:17.459886914z caller=main.go:304 module=http_2xx target=https://www.baidu.com level=info MSG ="Probe Succeeded "duration_seconds=0.054504338 Metrics that would have been returned: # HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds # TYPE Probe_dns_lookup_time_seconds Gauge PRObe_dNS_lookup_time_seconds 0.008485086 # HELP probe_duration_seconds Returns how Long the probe took to complete in seconds # TYPE probe_duration_seconds gauge Probe_duration_seconds 0.054504338 # HELP  probe_failed_due_to_regex Indicates if probe failed due to regex # TYPE probe_failed_due_to_regex gauge probe_failed_due_to_regex 0 # HELP probe_http_content_length Length of http content response # TYPE probe_http_content_length gauge probe_http_content_length 227 # HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirects # TYPE probe_http_duration_seconds gauge probe_http_duration_seconds{phase="connect"} 0.009129316 probe_http_duration_seconds 0.01002596 probe_http_duration_seconds {phase = "processing"} {phase = "resolve"} 0.008485086 probe_http_duration_seconds 0.035070878 probe_http_duration_seconds {phase = "TLS"} {phase = "transfer"} 0.000216612 # HELP probe_http_redirects The number of redirects # TYPE probe_http_redirects 0  # HELP probe_http_ssl Indicates if SSL was used for the final redirect # TYPE probe_http_ssl gauge probe_http_ssl 1 # HELP probe_http_status_code Response HTTP status code # TYPE probe_http_status_code gauge probe_http_status_code 200 # HELP probe_http_uncompressed_body_length Length of uncompressed response body # TYPE probe_http_uncompressed_body_length  gauge probe_http_uncompressed_body_length 227 # HELP probe_http_version Returns the version of HTTP of the probe Response # TYPE probe_HTTP_version Gauge Probe_HTTP_version 1.1 # HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes. # TYPE probe_ip_addr_hash gauge probe_ip_addr_hash 4.37589817e+08 # HELP probe_ip_protocol Specifies whether probe IP protocol is IP4 or IP6 # TYPE probe_ip_protocol gauge  probe_ip_protocol 4 # HELP probe_ssl_earliest_cert_expiry Returns earliest SSL cert expiry in unixtime # TYPE Probe_ssl_earliest_cert_expiry gauge probe_ssl_earliest_cert_expiry 1.627277462e+09 # HELP probe_ssl_last_chain_expiry_timestamp_seconds Returns last SSL chain expiry in timestamp seconds # TYPE Probe_ssl_last_chain_expiry_timestamp_seconds gauge probe_ssl_last_chain_expiry_timestamp_seconds = 1.627277462e+09 # HELP  probe_ssl_last_chain_info Contains SSL leaf certificate information # TYPE probe_ssl_last_chain_info gauge probe_ssl_last_chain_info{fingerprint_sha256="2ed189349f818f3414132ebea309e36f620d78a0507a2fa523305f275062d73c"} 1 # HELP probe_success Displays whether or not the probe was a success # TYPE probe_success gauge probe_success 1 # HELP probe_tls_version_info Contains the TLS version used # TYPE probe_tls_version_info gauge Probe_tls_version_info {version="TLS 1.2"} 1 Module configuration: prober: HTTP HTTP: ip_protocol_fallback: true TCP: ip_protocol_fallback: true icmp: ip_protocol_fallback: true dns: ip_protocol_fallback: trueCopy the code

Description of HTTP states in the HTTP trace

DNS resolution time: dnsdone-dnsstart

TLS handshake time: gotconn-dnsdone

TLS CONNECT Connection time: connectdone-dnsdone

Non-tls CONNECT Connection time: Gotconn-dnsdone

Processing server processing time: responseStart – gotConn
Transfer Data transfer time: end-responsestart

trace := &httptrace.ClientTrace{
    DNSStart:             tt.DNSStart,
    DNSDone:              tt.DNSDone,
    ConnectStart:         tt.ConnectStart,
    ConnectDone:          tt.ConnectDone,
    GotConn:              tt.GotConn,
    GotFirstResponseByte: tt.GotFirstResponseByte,
}
Copy the code

Blackbox_exporter needs to pass the target and Module parameters to be added to the collection pool as follows

Metrics_path: /probe metrics_path: /probe # params: module: 200 # [http_2xx] Look for a HTTP response. The target: [Prometheus. IO, www.baidu.com, 172.20.70.205:3000] static_configs: - the targets: - 172.20.70.205:9115Copy the code

You will see that the instance data will only have the blackbox_exporter address and not the target address

Probe_duration_seconds {instance = "172.20.70.205:9115", the job = "blackbox - HTTP"}Copy the code

See the description of 015 multi-instance collection

Blackbox_exporter collection is added to the collection pool

Metrics_path: /probe metrics_path: /probe [http_2xx] # Look for a HTTP 200 response. static_configs: - targets: - http://prometheus.io # Target to probe with http. - https://www.baidu.com # Target to probe with https. - http://172.20.70.205:3000 # Target to the probe with HTTP on port 3000. The relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 127.0.0.1:9115 # The Blackbox exporter's real hostname:port. Metrics_path: /probe metrics_path: /probe [ssh_banner] # Look for a HTTP 200 response. static_configs: - targets: -172.20.70.215 # Target to probe with HTTPS. Relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: Instance-target_label: __address__ replacement: 172.20.70.205:9115 # The Blackbox exporter's real hostname:port.Copy the code

Import blackbox_exporter Dashboard on Grafana

Address grafana.com/grafana/das…

SSH probes are based on TCP

Page access detection

# module USES ssh_banner detection 172.20.70.215: http://172.20.70.205:9115/probe? Module =ssh_banner&target=172.20.70.215:22 # HELP probe_dns_lookup_time_seconds Returns the time taken for probe DNS lookup in seconds # TYPE probe_dns_lookup_time_seconds Gauge Probe_dNS_lookup_time_seconds probe_duration_seconds Returns how long the probe took to complete in seconds # TYPE probe_duration_seconds gauge Probe_duration_seconds 0.02228226 # HELP probe_failed_due_to_regex Indicates IF probe failed due to regex # TYPE probe_failed_due_to_regex gauge probe_failed_due_to_regex 0 # HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes. # TYPE probe_ip_addr_hash gauge probe_ip_addr_hash 9.51584696e+08 # HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6 # TYPE probe_ip_protocol gauge probe_ip_protocol 4 # HELP probe_success Displays whether or not the probe was a success # TYPE probe_success gauge Ssh_banner: prober: TCP TCP: query_response: - expect: [root@prome_master_01 blackbox_exporter]# Telnet 172.20.70.215 22 Trying 172.20.70.215... Connected to 172.20.70.215. Escape character is '^]'. Ssh-2.0-openssh_7 foreign host.Copy the code

configuration

/metrics metrics_path: /probe metrics_path: /probe [ssh_banner] # Look for a HTTP 200 response. static_configs: - targets: -172.20.70.205:22 # Target to probe with http.relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: Instance-target_label: __address__ replacement: 172.20.70.205:9115 # The Blackbox exporter's real hostname:port.Copy the code

Blackbox – ping configuration

/metrics metrics_path: /probe metrics_path: /probe [icmp] # Look for a HTTP 200 response. static_configs: - targets: -192.168.26.112 # Target to probe with HTTPS. -192.168.26.112 # Target to probe with HTTPS. -114.114.114.114 relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 192.168.26.112:9115 # The Blackbox exporter's real hostname:port.Copy the code

Ping detection

Page access

http://172.20.70.205:9115/probe?module=icmp&target=www.baidu.com # # result interpretation HELP probe_dns_lookup_time_seconds Returns the  time taken for probe dns lookup in seconds # TYPE probe_dns_lookup_time_seconds gauge probe_dns_lookup_time_seconds 0.195704171 # HELP probe_duration_seconds Returns How long the probe took to complete in seconds # TYPE Probe_duration_seconds Gauge PRObe_duration_seconds 0.378563375 # HELP probe_ICMP_duration_seconds Duration of ICMP Request by phase # TYPE probe_ICMP_DURation_seconds gauge PRObe_ICMP_DURation_seconds {phase="resolve"} 0.195704171 Probe_icmp_duration_seconds {phase=" RTT "} 0.182456226 probe_icmp_duration_seconds{phase="setup"} 0.000145827 # HELP probe_icmp_reply_hop_limit Replied packet hop limit (TTL for ipv4) # TYPE probe_icmp_reply_hop_limit gauge probe_icmp_reply_hop_limit 49 # HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the # TYPE probe_ip_addr_hash gauge probe_ip_addr_hash 2.282787449e+09 # HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6 # TYPE probe_ip_protocol gauge probe_ip_protocol 4 # HELP probe_success Displays whether or not the probe was a success # TYPE probe_success gauge probe_success 1Copy the code

This section describes the SSH detection process

Prometheus - > blackbox_exporter using http://192.168.0.112:9115/probe? The module = ssh_banner & target = 192.168.0.127%3 a22 - > 192.168.0.127:22Copy the code

Redis_exporter Collects multiple instances

The project address

Github.com/oliver006/r…

Deploy redis_exporter using ansible

Yaml -e "TGZ =redis_exporter-v1.20.0.linux-amd64.tar.gz" -e "app=redis_exporter"Copy the code

Redis_exporter collection is added to the collection pool in the same mode as blackbox_exporter

scrape_configs: ## config for the multiple Redis targets that the exporter will scrape - job_name: 'redis_exporter_targets' static_configs: - targets: - redis: / / 172.20.70.205:6379 - redis: / / 172.20.70.205:6479 metrics_path: / scrape relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 172.20.70.215:9121Copy the code

[root@prome-master01 prometheus]# systemctl start redis_6379 [root@prome-master01 prometheus]# systemctl start redis_6479

Copy the grafana JSON project to import the large map

Address raw.githubusercontent.com/oliver006/r…

rendering

Monitor middleware and Grafana usage

Grafana use

Add monitoring for Grafana

Example Add the monitoring function of mysql

Add process monitor process_exporter

Black box probe blackbox_exporter

Redis_exporter Collects multiple instances

Related Posts

Spirng Framework source code parsing outline (2)

[Java Post] Huawei campus recruitment + Alibaba social recruitment completed in September

How did I become an Apache APISIX Committer in 146 days?