Wechat official account: Operation and maintenance development story, author: Jock

! [image. PNG] (img – blog. Csdnimg. Cn/img_convert… Object]&name=image.png&originHeight=1867&originWidth=4020&originalType=url&ratio=1&size=4386618&status=done&style=none&t askId=ude5e0690-ea57-482c-bbaf-a8bf35e60a6)

Hi, I’m Jock, a front-line operations practitioner.

Today I’m going to share with you what an error tracking system is and how to use it.

In the entire application life cycle, there is no escape from “errors”. In many cases, the log platform is used to capture and view logs to find the cause of the error. However, in many cases, the application logs are misleading and cannot directly point out the source of the problem. And also lack of timely analysis and alarm capabilities (of course, now many commercial log systems have this function, but need Money).

Is there open source software that works better?

Sentry is an excellent application bug tracking system and supports most languages, as shown below. ! [image. PNG] (img – blog. Csdnimg. Cn/img_convert… Object]&name=image.png&originHeight=553&originWidth=1349&originalType=binary&ratio=1&size=78812&status=done&style=none&t askId=u47bf3059-9cd1-4270-ad6d-e85fac721fa&width=1349)

What is an error tracking system?

Error tracking system is a software that tracks system errors and anomalies. It is designed to help development, operation and maintenance engineers track application error information.

Application errors can be tracked in real time and followed up in a unified manner to improve error management capabilities. Allow business scenarios to find bugs faster than users can report bugs.

What is the Sentry

Sentry is a cross-platform application error tracking system focused on error reporting.

How does Sentry work?

First, the application integrates with Sentry’s SDK, and then sends error information to the Sentry server when an error occurs in the application. The Sentry server is divided into web, Cron and worker. The application (client) reports the error information to The Web when it occurs, and the web processes the error information into the message queue or Redis memory queue. The worker consumes data from the queue for processing.

Its main advantages are as follows:

  • Provides a beautiful WEB UI interface
  • Support for almost all major development languages SDK, easy access
  • Provide full error details
  • Supports unified error aggregation analysis
  • Supports dashboards, monitoring, and alarm functions
  • Support team management and member management
  • Supports log audit

​

Of course, there are some disadvantages. There are a lot of middleware required for maintenance and deployment, such as Kafka, rabbitMQ, Redis, PGSQL, etc. This set of deployment costs are still relatively high. But these problems can be overcome for all the benefits.

! [image. PNG] (img – blog. Csdnimg. Cn/img_convert… Object]&name=image.png&originHeight=625&originWidth=1366&originalType=binary&ratio=1&size=163135&status=done&style=none& taskId=ue83f694e-3fd0-4606-814b-91b21a65a3e&width=1366)

Deploy the Sentry

Sentry provides services in two ways:

  • SAAS platform, there are different payment methods
  • Privatization deployment

​

Here is how to deploy in Kubernetes.

Environment introduction

  • Kubernetes： 1.17.17
  • Docker: 18.09.0
  • Helm: 3.6.3
  • Storage: Use Local PV

OpenEBS profile

OpenEBS is an open source container-based block storage software written in the Go language. OpenEBS makes it more reliable to run tasks critical and workloads that require data persistence in containers.

With OpenEBS, you can treat containers with persistent data just like any other normal container. OpenEBS itself is also deployed through containers, supporting Kubernetes, Swarm, Mesos, Rancher scheduling, storage services can be assigned to each POD, application, cluster, or container level, including:

  • Data persistence across nodes
  • Data synchronization across availability zones and cloud vendors
  • Use commercial hardware and container engines to provide highly scalable block storage
  • Integrated with the Container Orchestration engine, developers can automatically configure OpenEBS for their applications
  • Based on CloudByte’s experience in BSD containerization, this paper provides users with QoS guarantee of OpenEBS

​

OpenEBS architecture can be divided into two parts: Data Plane and Control Plane:

  • Data plane: Provides data storage for applications
  • Control plane: Manages OpenEBS volume containers, which typically uses container choreographer capabilities

The deployment environment

1. Deploy OpenEBS

OpenEBS supports Helm Chart and Operator deployment. (1) Deployment using Helm Chart

helm repo add openebs https://openebs.github.io/charts
helm repo update
helm install openebs --namespace openebs openebs/openebs --create-namespace
Copy the code

(2) Use Operator deployment

kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml
Copy the code

After the deployment is complete, the following PODS are generated.

# kubectl get pod -n openebs 
NAME                                          READY   STATUS    RESTARTS   AGE
maya-apiserver-67b5b5c858-4mstb               1/1     Running   0          5d19h
openebs-admission-server-6bdf9b76d6-r4r6b     1/1     Running   0          5d19h
openebs-localpv-provisioner-966d864cd-sf8fp   1/1     Running   0          5d19h
openebs-ndm-59lkx                             1/1     Running   0          5d19h
openebs-ndm-bphgw                             1/1     Running   0          5d19h
openebs-ndm-grxlb                             1/1     Running   0          5d19h
openebs-ndm-operator-55b8ccc64b-r7tkr         1/1     Running   0          5d19h
openebs-ndm-zwhmm                             1/1     Running   0          5d19h
openebs-provisioner-55794b6cd4-ptd85          1/1     Running   0          5d19h
openebs-snapshot-operator-5d78fcbd96-7xqzv    2/2     Running   0          5d19h
Copy the code

And StorageClass as follows.

# kubectl get sc | grep openebs
local (default)             openebs.io/local                                           Delete          WaitForFirstConsumer   false                  5d19h
openebs-device              openebs.io/local                                           Delete          WaitForFirstConsumer   false                  5d19h
openebs-hostpath            openebs.io/local                                           Delete          WaitForFirstConsumer   false                  5d19h
openebs-jiva-default        openebs.io/provisioner-iscsi                               Delete          Immediate              false                  5d19h
openebs-snapshot-promoter   volumesnapshot.external-storage.k8s.io/snapshot-promoter   Delete          Immediate              false                  5d19h

Copy the code
2. Deploy Postgresql

In fact, there is a Postgresql Chart package in Sentry’s Helm Chart. The reason for this deployment is that there are various problems in the Postgresql link through the Chart package deployment in Sentry. For example:

FATAL: password authentication failed for user "postgres"
Copy the code

To deploy Postgresql using Helm Chart, perform the following steps:

# add Helm warehouse Helm repo add bitnami https://charts.bitnami.com/bitnami Helm repo update # download Chart package Helm pull bitnami/postgresqlCopy the code

Write the configuration file (my-value.yaml) as follows:

global:
  postgresql:
    postgresqlDatabase: "sentry"
    postgresqlUsername: "postgres"
    existingSecret: ""
    postgresqlPassword: "postgres"
    servicePort: ""
    replicationPassword: ""
Copy the code

To install PGSQL, run the following command:

helm install postgresql -n sentry -f my-value.yaml bitnami/postgresql
Copy the code

After deployment, you can see the following Pod:

# kubectl get po -n sentry 
NAME                                                  READY   STATUS      RESTARTS   AGE
postgresql-postgresql-0                               1/1     Running     0          3h39m
Copy the code
3. Deploy Sentry

Sentry is also deployed using Helm Chart as follows:

# add Helm warehouse Helm repo add sentry https://sentry-kubernetes.github.io/charts Helm repo update Helm search repo sentry # Download the Chart package to view and modify Chart Helm Pull Sentry/SentryCopy the code

Write the configuration file (my-value.yaml) as follows:

externalPostgresql:
  database: sentry
  port: 5432
  username: postgres
  host: postgresql
  password: postgres
postgresql:
  enabled: false
mail:
  backend: dummy
  from: "[email protected]"
  host: "smtp"
  password: "UZKSGLFEANWGLZNT"
  port: 465
  useTls: true
  username: ""
user:
  create: true
  email: [email protected]
  password: P@ssword
Copy the code

Deploy Sentry as follows:

helm install sentry -n sentry -f my-value.yaml sentry/sentry
Copy the code

After the deployment is complete, the following Pod is generated:

# kubectl get po -n sentry NAME READY STATUS RESTARTS AGE postgresql-postgresql-0 1/1 Running 0 3h50m sentry-clickhouse-0 1/1 Running 0 3h50m sentry-clickhouse-1 1/1 Running 0 3h50m sentry-clickhouse-2 1/1 Running 0 3h50m sentry-cron-578647dd7-gk7gf 1/1 Running 0 3h50m sentry-ingest-consumer-7564f644bf-srkj2 1/1 Running 0 3h47m sentry-kafka-0 1/1 Running 2 3h50m sentry-kafka-1 1/1 Running 2 3h50m sentry-kafka-2 1/1 Running 2 3h50m sentry-nginx-675d779699-tfdr5 1/1 Running 0 3h50m sentry-post-process-forward-5f586f6cdf-lxmc4 1/1 Running 0 3h47m sentry-rabbitmq-0 1/1 Running 0 3h50m sentry-rabbitmq-1 1/1 Running 0 3h49m sentry-rabbitmq-2 1/1 Running 0 3h48m sentry-relay-75597cb98b-jm282 1/1 Running 0 3h47m sentry-sentry-redis-master-0 1/1 Running 0 3h50m sentry-sentry-redis-slave-0 1/1 Running 0 3h50m sentry-sentry-redis-slave-1 1/1 Running 0 3h49m sentry-sessions-consumer-6b499bf64d-lm7gq 1/1 Running 0 3h47m sentry-snuba-api-5586d5f9f8-tcn5s 1/1 Running 0 3h50m sentry-snuba-cleanup-errors-1630472400-xv5wh 0/1 Completed 0 134m sentry-snuba-cleanup-errors-1630476000-xd7d2 0/1 Completed 0 74m sentry-snuba-cleanup-errors-1630479600-sd59b 0/1 Completed 0 14m sentry-snuba-cleanup-transactions-1630472400-l9bcx 0/1 Completed 0 134m sentry-snuba-cleanup-transactions-1630476000-cjsbd 0/1 Completed 0 74m sentry-snuba-cleanup-transactions-1630479600-zn5dz 0/1 Completed 0 14m sentry-snuba-consumer-58b7bfd545-rnjmx 1/1 Running 0 3h47m sentry-snuba-outcomes-consumer-57c589bf6d-lxg5m 1/1 Running 0 3h47m sentry-snuba-replacer-5bf4d6b5d7-gcc7g 1/1 Running 0 3h47m sentry-snuba-transactions-consumer-57bb7f8ccb-q6gkt 1/1 Running 0 3h47m sentry-web-7c9766768b-njxjl 1/1 Running 0 3h50m sentry-worker-844fd65687-6b26p 1/1 Running 2 3h50m sentry-worker-844fd65687-k2h6z 1/1 Running 2 3h50m sentry-worker-844fd65687-rfjcl 1/1 Running 2 3h50m sentry-zookeeper-0  1/1 Running 0 3h50mCopy the code

The user name and password are [email protected] and P@ssword, which can be accessed through ingress or nodeport.

Create a project

Enter the WEB UI interface and create a project in the project column as follows: [image. PNG] (img – blog. Csdnimg. Cn/img_convert… Object]&name=image.png&originHeight=625&originWidth=1366&originalType=binary&ratio=1&size=160599&status=done&style=none& taskId=u0ba1cb2d-044b-41c4-a35b-1979721921b&width=1366)

Here I create a Go project as follows: [image. PNG] (img – blog. Csdnimg. Cn/img_convert… Object]&name=image.png&originHeight=625&originWidth=1366&originalType=binary&ratio=1&size=173529&status=done&style=none& taskId=ue71758c0-7194-4625-a059-dad1b1e225f&width=1366)

After clicking Create, a Dsn address is generated and an example configuration is given as follows:

package main

import (
	"fmt"
	"github.com/getsentry/sentry-go"
	sentrygin "github.com/getsentry/sentry-go/gin"
	"github.com/gin-gonic/gin"
	"github.com/pkg/errors"
	"net/http"
)

func main(a) {
	// To initialize Sentry's handler, you need to initialize Sentry itself beforehand
	if err := sentry.Init(sentry.ClientOptions{
		Dsn: "https://[email protected]/5942245"}); err ! =nil {
		fmt.Printf("Sentry initialization failed: %v\n", err)
	}
	sentry.CaptureMessage("It works!")
	sentry.CaptureException(errors.New("error msg"))
	// Then create your app
	app := gin.Default()

	// Once it's done, you can attach the handler as one of your middleware
	app.Use(sentrygin.New(sentrygin.Options{}))

	// Set up routes
	app.GET("/".func(ctx *gin.Context) {
		ctx.String(http.StatusOK, "Hello world!")})// And run it
	app.Run(": 3000")}Copy the code

For other types of projects, there are three steps:

  • Create a project in the Sentry WEB UI interface
  • Generate the corresponding Dsn
  • Introduce it into the actual application code

​

After an application is deployed, an application error is reported in Post mode and can be viewed on the WEB UI.

! [image. PNG] (img – blog. Csdnimg. Cn/img_convert… Object]&name=image.png&originHeight=562&originWidth=1366&originalType=binary&ratio=1&size=174385&status=done&style=none& taskId=u9d3a3b0a-e29c-4fe2-831d-8845b42b60f&width=1366)

​

Create the alarm

The alarm types that can be created are as follows:! [image. PNG] (img – blog. Csdnimg. Cn/img_convert… Object]&name=image.png&originHeight=625&originWidth=1366&originalType=binary&ratio=1&size=217240&status=done&style=none& taskId=ua3dd0d09-ce52-444f-a1b1-c187a4e1dee&width=1366)

Then create the rule as follows:! [image. PNG] (img – blog. Csdnimg. Cn/img_convert… Object]&name=image.png&originHeight=625&originWidth=1366&originalType=binary&ratio=1&size=178055&status=done&style=none& taskId=u806ee44d-8374-406b-b428-ebe3cf6040b&width=1366)

In addition, you can also carry out team, member management, log audit, etc., more functions need to practice themselves.

See [1] sentry. IO / [2] openebs. IO / [3] github.com/getsentry/s… [4] github.com/sentry-kube…