The original link

What is a Sentry

Sentry is an open source application monitoring and error tracking solution. This solution consists of SDKS corresponding to various languages and a set of huge data backend services. The application uses the token bound to it to access the Sentry SDK to configure data reporting. The Sentry SDK can also be configured to report version information and release environment associated with errors. In addition, the Sentry SDK automatically captures related operations before exceptions occur, facilitating subsequent exception tracing. After the abnormal data is reported to the data service, it is filtered, extracted, and displayed on the Web page of the data background.

After the connection is complete, we can check the abnormality of the application in real time from the management system, so as to actively monitor the running status of the application on the client. By configuring alarms and analyzing the occurrence trend of anomalies, we can nip anomalies in the bud more actively and affect fewer users. By analyzing the details of exceptions and tracking abnormal operations, the client can avoid applying the status of two eyes and one black eye to solve problems more efficiently.

The preparatory work

Docker

Docker is open source containerization technology that can be used to build and containerize applications. Compose is a tool for configuring and running multiple Docker apps, allowing you to configure all of your app’s services from a single profile and create and run them with one click. The recommended deployment environment is Linux

Deployment and configuration requirements:

  • The Docker 19.03.6 +
  • Compose 1.24.1 +
  • 4 CPU Cores
  • 8 GB RAM
  • 20 GB Free Disk Space

Quickly deploy Sentry

git clone https://github.com/getsentry/onpremise.git
Copy the code

I am using the latest version here. Later, I can compare the time and clone the code after checkout to the corresponding branch (Sentry updates are quite frequent. If you need to follow the blog process, it is suggested to use the corresponding version).

You are advised to configure the image acceleration service (aicomocloud and DaoCloud have free services) and then modify or generate the /etc/docker-daemon. json file after obtaining the link

{
  "registry-mirrors": ["Mirror address"]}Copy the code

The install.sh file is in the root directory of the pulled code. You can execute the script directly to complete the rapid deployment. This script will pull the image required by Sentry. The process is slow and you are advised to deploy it in a good network environment. The deployment process is as follows:

  • Environmental audit
  • Generate the service configuration
  • Docker volume data volume creation (Docker persistent storage, docker independent storage)
  • Pull and upgrade the base image
  • Build the mirror
  • Service initialization
  • Setting an administrator account (skip this step and create one manually)

Docker-compose up: docker-compose up: docker-compose up: docker-compose up: docker-compose up: docker-compose up: docker-compose You can see the startup logs of the service. The service can be started only after the internal Web, relay, Snuba, and Kafka are started and initialized

When you access the management background for the first time, the welcome page is displayed. You can officially access the management background after completing the required configuration.

  • Root URL: indicates the public Root address of the interface reporting exceptions. (During network resolution configuration, background services can be configured with two domain names on the Intranet and the Internet, and only the interface resolution rules/API /[ID]/store/ are reported to the public network to prevent data leakage.)
  • Admin Email: administrator account created during the install.sh phase
  • Outbound email: indicates the mail service configuration. You can configure it in /sentry/config.yml in the root directory

Docker data store location changed

During service operation, a large number of logs and Postgres will be generated, and these data will be hung in docker volume. Volume is hung in the /var directory by default. Usually, the /var directory has a small capacity, and the memory will be easily occupied with service operation. We can modify the docker volume mounted in the directory

Create a folder in the directory with the maximum capacity
mkdir -p /data/var/lib/
# Stop docker service
systemctl stop docker
Copy the default docker data to the new path, delete the old data and create a soft connection, even if the storage actually occupies the disk as the new path
/bin/cp -a /var/lib/docker /data/var/lib/docker && rm -rf /var/lib/docker &&  ln -s /data/var/lib/docker /var/lib/docker
Restart the Docker service
systemctl start docker
Copy the code

Email Notification Configuration

Yml /sentry/config.yml in the root directory. Before configuring email addresses, enable the SMTP service of email addresses

mail.backend: "smtp"
mail.host: "smtp.163.com"
mail.port: 25
mail.username: "Email address"
mail.password: "Password"
mail.use-tls: true  # Whether to use TLS service
#The email address to send on behalf of
mail.from: "Mail source address"
Copy the code

Git and Slack can also be configured in this configuration file. For details, see Sentry official documentation

The env file

COMPOSE_PROJECT_NAME=sentry_onpremise
SENTRY_EVENT_RETENTION_DAYS=90
SENTRY_IMAGE=getsentry/sentry:83b1380
# You can either use a port number or an IP:PORT combo for SENTRY_BIND
# See https://docs.docker.com/compose/compose-file/#ports for more
SENTRY_BIND=9000
SENTRY_IMAGE=getsentry/sentry:nightly
SNUBA_IMAGE=getsentry/snuba:nightly
RELAY_IMAGE=getsentry/relay:nightly
SYMBOLICATOR_IMAGE=getsentry/symbolicator:nightly
Copy the code

Env in the root directory, or SENTRY_EVENT_RETENTION_DAYS in the environment of docker-comemess. yml file. We can control the duration of data retention. Reduce server memory consumption If the database does not have a scheduled reclamation mechanism, manually delete the database.

Reclaim statement for referenceVacuumdb -u [user name] -d [database name] -v -f --analyzeCopy the code

The mirroring service required by Sentry and the operation mechanism

Open the docker-comemess. yml file in the root directory and run the command on the terminal

docker ps -a
Copy the code

Here we can compare the currently running container and docker-compose configuration and guess what each service is doing (personal understanding, reference only) :

  • Nginx: Network communication between services
  • Sentry_defaults: indicates the default environment
    • Worker: Handle background work, email, alarm, etc
    • Cron: processes scheduled tasks
    • Web: Sentry’s Web page service
    • post-process-forwarder
    • Ingest-consumer: Processes Kafka messages
  • Snuba-cleanup: Data cleanup
  • Relay:
    • Data reported through the Web is sent to the relay
    • Relay directly returns the response status
    • You then continue processing the data in background tasks
    • Parsing events, formatting, and enabling filtering rules
    • Data is written to Kafka
  • Postgres: default database used to store abnormal data
  • Redis: Data interception configuration
  • Kafka: Data response, establishing long connections between services
  • Zookeeper: manages kafaka services

Sentry works as follows:

  • Abnormal data is parsed by nginx to the relay service.
  • The relay uses PG to obtain the latest matching relationship between the application and the token, verifies the token in the data, returns 403 or 200, and blocks and filters the data.
  • Relay sends data to different Topics in Kafka.
  • Sentry subscribes to some of these topics and stores the parse data to Postgres for later viewing error details.
  • Snuba subscribes to other topics, tags the data, extracts key features, and stores them in ClickHouse for quick retrieval of data based on key features.