Docker Advanced - Container monitoring cAdvisor+InfluxDB+Granfana

An overview of the

In the previous article, docker Compose was used to compose applications and scale the container quickly.

Because docker compose starts services on the same host, when multiple container applications run on one host, it is necessary to understand the running status of the container, such as CPU usage, memory usage, network status, disk space and a series of time-varying data information, so monitoring is necessary.

Container monitoring scheme selection

There are various Monitoring schemes for containers, including docker Stats command, Scout,Data Dog,Sysdig Cloud,Sensu Monitoring Framework,CAdvisor, etc.

Using docker stats command, you can easily see the CPU, memory, network traffic and other data of all containers on the current host. However, the disadvantage of the Docker STATS command is that it only counts all containers of the current host, and the obtained monitoring data is real-time, with no place to store and no alarm function.

docker stats
Copy the code

Scout, Sysdig Cloud, and Data Dog all offer better services, but they are all hosted and charged. The Sensu Monitoring Framework is highly integrated and free, but is too complex to deploy. Overall consideration, we choose CAdvisor as the container monitoring tool.

CAdvisor is produced by Google. It is an open source product with complete monitoring indicators, convenient deployment, and an official Docker image. The disadvantage is that the integration degree is not high, and the data is saved locally for only 2 minutes by default. However, the container monitoring system can be conveniently built by adding InfluxDB to store data and docking Grafana to display charts. The data collection and chart display effect is good, and the system performance is almost not affected.

CAdvisor

CAdvisor is a container resource monitoring tool, including the container memory,CPU, network IO, disk IO monitoring, and provides a WEB page to view the real-time running status of the container. CAdvisor stores 2 minutes of data by default and only for a single physical machine. CAdvisor, however, provide a lot of data integration interface, support InfluxDB, Redis, Kafka, Elasticsearch integration, can add the corresponding configuration will monitor stored data sent to the database.

CAdvisor has two main functions:

Displays monitoring data at the Host and container levels.
Display historical change data.

InfluxDB

InfluxDB is an open source distributed time series, events, and metrics database written in the Go language with no external dependencies.

As mentioned above, by default,CAdvisor stores only the latest 2 minutes’ data on the local machine. For persistent storage and unified collection and display of monitoring data, data needs to be stored in the InfluxDB. InfluxDB is a timing database dedicated to timing related data, which is ideal for storing CAdvisor data. Furthermore, the CAdvisor itself already provides an integration method for InfluxDB, which you can specify when starting the container.

InfluxDB Main functions:

Based on time series, support correlation functions related to time (such as maximum, minimum, sum, etc.);
Measurability: You can calculate large amounts of data in real time;
Event-based: It supports arbitrary event data;

InfluxDB Main features:

No structure (no pattern);
It can be any number of columns;
Expandable;
Support the min, Max, sum, count, mean, median, and a series of functions, convenient statistics;
Native HTTP support, with built-in HTTP APIS;
Powerful SQL-like syntax;
Own management interface, easy to use

Granfana

Grafana is an open source data monitoring analysis visualization platform, support for multiple data source configuration (including InfluxDB support data sources, MySQL, Elasticsearch, OpenTSDB, Graphite, etc.) and rich plug-in and template function, support chart access control and alarm.

Grafan features:

Flexible and rich graphical options
Can mix a variety of styles
Support for day and night modes
Multiple data sources

CAdvisor+InfluxDB+Granfana

Grafana: Is responsible for analyzing and presenting temporal data

Install the deployment

The InfluxDB service is deployed

docker run -d --name influxdb -p 8086:8086 \
-v /data/influxdb:/var/lib/influxdb \
--hostname=influxdb \
influxdb
Copy the code

Description:

–name: Indicates the assigned name of the starting container
-p: indicates a mapping port. Port 8083 is the management port and port 8086 is the data port
-v: mount data -d: run the container in the background
Influxdb: Used by the docker container, which is pulled by default from the official docker repository

1. Access the management terminal of influxDB using the browser, http://ip:8083

2, create test database and root user, this will be used for granfa later

docker exec it influxdb influx

CREATE DATABASE "test"
CREATE USER "root" WITH PASSWORD 'root' WITH ALL PRIVILEGES
Copy the code

Deploy the CAdvisor service

Google’s CAdvisor can be used to gather temporal information about Docker containers, including resource usage and performance data as the containers run.

docker run --volume=/:rootfs:ro --volume=/var/run:/var/run:ro\
--volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro\
--volume=/dev/disk/:/dev/disk:ro --volume=/cgroup:/cgroup:ro \
--publish=8080:8080 --detach=true --privileged=true --name=cadvisor \
google/cadvisor:latest\
-storage_driver=influxdb-storage_driver_host=influxdb:8086\
-storage_driver_db=test-storage_driver_user=root\
-storage_driver_password=root
Copy the code

Description:

-d: Runs the container in the background
–name: Starts the container assignment name CAdvisor
-p: indicates mapping port 8080
–mount: Bind the host directories to the container. These directories are the directory files and monitoring content that the CAdvisor needs to collect
-storage_driver: Specifies the storage driver, database host, and database name of the CAdvisor

Test this by accessing port IP :8080

Deploy the Granfana service

Grafana is an open source temporal data analysis tool with a professional and simple interface.

docker run -d -p 3000:3000 \
-v /data/grafana:/var/lib/grafana \
--link=influxdb:influxdb\
--name grafana  grafana/grafana
Copy the code

Note: If you use -v to mount data, the container will fail to start. Mkdir :cannot create directory ‘/var/lib/grafana-plugins ‘:Permission denied docker run –entrypoint “id” Grafana /grafana gets uid,gid,groups(default 472);

Then use chown -r 472:472 /data/grafana/ to change the directory permissions to boot successfully.

To access the Grafana Web service through port IP :3000, change the admin user password for the first time. The default user name/password is admin/admin

Docker-compose rapid deployment

version: '3.1'

volumes:
  grafana_data: {}

services:
 influxdb:
  image: Tutum/influxdb: 0.9
  #image: tutum/influxdb
  #image: influxdb
  restart: always
  #user: 
  environment:
    - PRE_CREATE_DB=cadvisor
  ports:
    - "8083:8083"
    - "8086:8086"
  expose:
    - "8090"
    - "8099"
  volumes:
    - ./data/influxdb:/data

 cadvisor:
  # image: Google/cadvisor: v0.29.0
  image: google/cadvisor
  links:
    - influxdb:influxsrv
  command: -storage_driver=influxdb -storage_driver_db=cadvisor -storage_driver_host=influxsrv:8086
  restart: always
  ports:
    - "8080:8080"
  volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro

 grafana:
  # image: grafana/grafana: server
  user: "104"
  image: grafana/grafana
  user: "104"
  #user: "472"
  restart: always
  links:
    - influxdb:influxsrv
  ports:
    - "3000:3000"
  volumes:
    - grafana_data:/var/lib/grafana
  environment:
    - HTTP_USER=admin
    - HTTP_PASS=admin
    - INFLUXDB_HOST=influxsrv
    - INFLUXDB_PORT=8086
    - INFLUXDB_NAME=cadvisor
    - INFLUXDB_USER=root
    - INFLUXDB_PASS=root
Copy the code