Both operating systems are centos8

introduce

Build a set of application system that can scale quickly based on Docker swarm. Our system is not a microservice architecture. Considering that microservices need to face too many challenges (distributed transactions, etc.), we are just multiple individual services with few calls between services. The load balancing between services is realized by Docker service.

Build targets

Visual management cluster, unified gateway output API, unified log processing, unified storage processing, gateway traffic monitoring, failover.

The preparatory work

The content of the preparation work needs to be performed for each machine.

Disabling the Firewall

systemctl stop firewalld
systemctl disable firewalld
Copy the code

For security purposes, you are advised to reserve firewalls in the production environment and open only required ports.

Modify the DNS

vi /etc/resolv.conf 
Copy the code

Change nameserver to 114.114.114.114

Change the Hostname

Reset hostname for each machine, node-1, Node-2… . node8 hostnamectl set-hostname node-1 –static

Close the selinux

vim /etc/selinux/config
Copy the code

#Restarting the Server
reboot 
Copy the code

Install the docker – ce

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf install -y yum-utils device-mapper-persistent-data lvm2
dnf makecache
dnf install https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm
#Centos8 comes with Podman by default, so it conflicts and needs to be replaced automatically with parameters
dnf install docker-ce docker-ce-cli --allowerasing
#Verify that the installation was successful
docker version
#Boot from the rev.
systemctl enable docker
Copy the code

Modify the Docker configuration file

vim /etc/docker/daemon.json 
Copy the code

Configuration contents:

{
    "registry-mirrors": [
        "https://docker.mirrors.ustc.edu.cn"]."dns": ["114.114.114.114"]}Copy the code

Mainly accelerate the docker image pull.

Resources division

The management node

The number of management nodes must be odd

  • 172.16.113.4
  • 172.16.113.5
  • 172.16.113.6

Work node

  • 172.16.113.7
  • 172.16.113.8
  • 172.16.113.9

The gateway node

  • 172.16.113.10
  • 172.16.113.11

Ops node

  • 172.16.113.12

Example Initialize the swarm cluster

We perform the initialization command on 172.16.113.4

Docker swarm init --listen-addr 0.0.0.0:2377 --cert-expiry 87600h0m0s --task-history-limit 1Copy the code

Then we run it on every work node, operation node, gateway node

This command is printed in the console after initialization. Docker swarm join --token You can also run the docker swarm join-token worker command on the management node to obtain the docker swarm join --token SWMTKN - 1-5 ua3tzuqsglxry0kwb7tg3b2mtiqfg40dwt3na0zwa322tkdfn - 03 essdmekuv68m4s9l7kv6rud 172.16.113.4:2377Copy the code

We then run it on each of the remaining management nodes

Docker swarm join --token docker swarm join --token docker swarm join --token docker swarm join --token SWMTKN - 1-5 ua3tzuqsglxry0kwb7tg3b2mtiqfg40dwt3na0zwa322tkdfn - c25ey7vpenvhz4glf8vegnj26 172.16.113.4:2377Copy the code

Validation of the cluster

#On the administrative node, six nodes should be listed
docker node ls
Copy the code

Deploy Portainer – ce

Portainer we don’t need to install into a cluster, we install on the o&M node.

Create a network

On the O&M node

Docker network create \ --driver overlay \ --attachable \ --subnet 10.12.0.0/24 \ portainer_agent_networkCopy the code

Install the portainer – ce

docker run -d \
-p 8000:8000 \
-p 9000:9000 \
--name=portainer \
--network=portainer_agent_network \
--restart=always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /data/portainer:/data \
portainer/portainer-ce
Copy the code

Install the portainer – agent

docker service create \
    --name portainer_agent \
    --network portainer_agent_network \
    --mode global \
    --constraint 'node.platform.os == linux' \
    --mount type=bind,src=//var/run/docker.sock,dst=/var/run/docker.sock \
    --mount type=bind,src=//var/lib/docker/volumes,dst=/var/lib/docker/volumes \
    portainer/agent
Copy the code

Configuration portainer – ce

Open a browser and log in to the O&M node IP address :9000.

Deploy the Traefik gateway

Creating an Application Network

Run on the management node

Docker network create \ --driver overlay \ --attachable \ --subnet 10.100.0.0/24 \ application_netCopy the code

Creating a Configuration File

################################################################ # # Configuration sample for Traefik v2. # # For Traefik v1: https://github.com/traefik/traefik/blob/v1.7/traefik.sample.toml # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # ################################################################ # Global configuration ################################################################ [global] checkNewVersion = true sendAnonymousUsage = true ################################################################ # Entrypoints configuration ################################################################ # Entrypoints definition # # Optional # Default: [entryPoints] [entryPoints.web] address = ":80" [entryPoints.websecure] address = ":443" ################################################################ # Traefik logs configuration ################################################################ # Traefik logs # Enabled by default and log to stdout #  # Optional # [log] # Log level # # Optional # Default: "ERROR" # # level = "DEBUG" # Sets the filepath for the traefik log. If not specified, stdout will be used. # Intermediate directories are created if necessary. # # Optional # Default: os.Stdout # filePath = "log/traefik.log" # Format is either "json" or "common". # # Optional # Default: "common" # # format = "json" ################################################################ # Access logs configuration ################################################################ # Enable access logs # By default it will  write to stdout and produce logs in the textual # Common Log Format (CLF), extended with additional fields. # # Optional # [accessLog] # Sets the file path for the access log. If not specified, stdout will be used. # Intermediate directories are created if necessary. # # Optional # Default: os.Stdout # filePath = "/log/log.txt" # Format is either "json" or "common". # # Optional # Default: "common" # # format = "json" ################################################################ # API and dashboard configuration ################################################################ # Enable API and dashboard [api] # Enable  the API in insecure mode # # Optional # Default: false # insecure = true # Enabled Dashboard # # Optional # Default: true # dashboard = true ################################################################ # Ping configuration ################################################################ # Enable ping [ping] # Name of the related entry point # # Optional # Default: "traefik" # # entryPoint = "traefik" ################################################################ # Docker configuration backend ################################################################ # Enable Docker configuration backend [providers.docker] # Docker server endpoint. Can be a tcp or a unix socket endpoint. # # Required # Default: "Unix: / / / var/run/docker. The sock" # swarmMode = true endpoint = "SSH: / / [email protected]" network = "application_net #" Default host rule. # # Optional # Default: "Host(`{{ normalize .Name }}`)" # # defaultRule = "Host(`{{ normalize .Name }}.docker.localhost`)" # Expose containers by default in traefik # # Optional # Default: true # # exposedByDefault = false # # #[metrics] # [metrics.influxDB] # address = "influxdb:8089" # protocol = "udp" # database = "traefik"Copy the code

Note: The endpoint in the configuration file should be the IP addresses of any two management nodes. Here, SSH is used to ensure the security of the Docker Access API.

Configure SSH password-free login

Each management node must support SSH secret-free login and store the public key to the gateway node so that the gateway node can SSH to the management node without the password. The configuration of non-secret login please check your own information.

Create a Traefik container

Run on both gateways and create 2 gateways for high availability (keepalived can be used for vIP drift).

docker run -d -p 81:8080 -p 80:80 \
-v /data/traefik/conf/traefik.toml:/etc/traefik/traefik.toml \
--name=traefik \
--network=application_net \
-v /root/.ssh:/root/.ssh \
traefik:v2.4.8
Copy the code

At this point, you should see a container running Traefik on the Portainer.

This step is crucial! (Installing SSH)

Traefik is not connected to swarm properly yet. Ok! We enter traefik’s container at portainer and run

apk update
apk add openssh
Copy the code

Once installed, Traefik was able to automatically connect to the Swarm cluster, and the cluster and gateway were complete.

The deployment of application

Nginx is used as the application. Run this command on any management node

 docker service create \
 --replicas 10 \
 --publish 8080:80 \
 --network=application_net \
 --label="traefik.http.routers.nginx-service.rule=Host(\`nb.io\`)" \
 --label="traefik.http.services.nginx-service.loadbalancer.server.port=80" \
 --name nginx nginx
Copy the code

Nb. IO in Host should be changed to its own domain name. If you do not have a domain name, you can set up a private DNS server using sameersbn/bind. After the wait is complete, we can see on traefik’s webUI:

Now you can accessnb.ioTo access the Nginx service cluster.

Restrict the node on which applications are deployed

Swarm swarm swarm swarm swarm swarm swarm swarm swarm swarm swarm swarm swarm swarm swarm swarm swarm

Label nodes

Tag each node under Portainer’s Swarm menu, or use the command that needs to be executed on the admin node

Docker node update - label-add role= Application node-1 Docker node update - label-add role= Application Node-2 Docker node Update - label-add role= Application node-3 Docker node update - label-add role= Application Node -4 Docker node update - label-add role= Application node-5 Docker node update - label-add role= Application Node-6 Docker node update - label-add Role =gateway node-7 Docker node update - label-add role=gateway node-8 Docker node update - label-add role= SYS node-9Copy the code

Adjust service creation commands

 docker service create \
 --replicas 10 \
 --publish 8080:80 \
 --network=application_net \
 --constraint 'node.labels.role == application' \
 --label="traefik.http.routers.nginx-service.rule=Host(\`nb.io\`)" \
 --label="traefik.http.services.nginx-service.loadbalancer.server.port=80" \
 --name nginx nginx
Copy the code

Application services will now only be deployed on Node-1 through Node-6 nodes.

conclusion

Traefik and Portainer can also be deployed as a service, but it depends on the actual situation.