This is the third day of my participation in the First Challenge 2022

preface

Prometheus has many solutions for napping, and this article is just one of them. If you happen to use this solution, I hope it will be helpful.

This is a set of alarm monitoring tools I often use:

  1. prometheus: Installation | Prometheus
  2. alert manager: Configuration | Prometheus
  3. Prometheus – webhook – dingtalk: github.com/timonwong/p…

Monitor the topology

​

Prometheus sends alarm information to the spike group.

As for this monitoring and alarm scheme, some students in the industry should still be using it, because I also learned about this scheme on the Internet at the beginning, and then implemented it after targeted improvement in my own business scenarios.

Alerts to different robots

Webhook-dingtalk receives the alarm information sent by alert-Manager, beautifies it (I mainly use it to beautify the alarm and convert it into markdown format), and then distributes the alarm to the pinning robot.

I also wrote a previous article on how to distribute alarms to different robots: Prometheus configured alarms to be distributed to different spike groups!

The configuration is written in the configuration file. Of course, it can be hot loaded, but it is very inconvenient to distribute different alarm information to different robots.

For example, I made an alarm for RocketMQ, but different projects use different consumer groups, and the project leader only cares about the consumption backlog alarm for each consumer group. As the person responsible for the messaging platform, we want to receive all the alarm information in our own alarm group. Then, there are many consumer groups and corresponding project groups. Different alarms need to be sent to different project groups, which is very troublesome.

The solution

We recommend a tool: Webhook-dingtalk-Dispatcher

You can customize which alarms are notified to which nailing robots.

As follows:​

In the previous example, I configured three alarm rules.

  1. If the alarm contains any of the configured keywords, the robot will receive the alarm
  2. If the alarm information contains all the configured keywords, the corresponding machine can receive the alarm
  3. The robot will receive any alarms that are sent

New Alarm Topology

After adding webhook-dingtalk-dispatcher, the new alarm topology is as follows:

​

Webhook-dingtalk-dispatcher can be optimized to support beautification of alarm messages, so that webhook-dingtalk can be replaced, but it is not currently supported.

Webhook dingtalk — the dispatcher to use

download

Download: github.com/xxd76379515…

The deployment of

1. Download and decompress the installation package.

2. The configuration file is config/application.yml. If you need to change the port or token information, change it here.

3. Start/stop

# start
sh bin/start.sh
# stop
sh bin/shutdown.sh
Copy the code

4. Configure Prometheus – webhook-dingTalk and modify the config.yml file of Prometheus – Webhook-dingTalk

Modify the promethee-webhook-dingtalk configuration file, not the webhook-dingtalk-dispatcher configuration file.

Webhook-dingtalk-dispatcher = webhook-dingtalk-dispatcher = webhook-dingtalk-dispatcher = webhook-dingtalk-dispatcher

targets:
  webhook:
    url: http://localhost:7006/dispatcher?access_token=1234567890
Copy the code

The access_token value is set in the webhook-dingtalk-dispatcher configuration file. The default value is 1234567890.

test

Webhook-dingtalk-dispatcher is now available in your browser at http://localhost:7006

Then click the Add button to add a robot alarm configuration:

​

After configuration, you can also click a test link to see whether the robot has received the alarm information and check whether the configuration is correct.

Click Submit.