This article is published under a SIGNATURE 4.0 International (CC BY 4.0) license. Signature 4.0 International (CC BY 4.0)

Author: Su Yang

Statistical word count: 10813 words reading time: 22 minutes to read this article links: soulteary.com/2018/08/28/…

More complete Docker + Traefik use scheme

After stepping on the pit countless, many times after modification, the text in the draft box was finally able to shape, scattered flowers.

When the architecture was updated in June, openResty was removed as the front end of the server and Traefik was replaced with a bare Traefik. Because only 80/443 of the gateway was exposed, all the children exposed ports to a virtual network card with the expose scheme. Gateway mounted wildcard certificate, you can easily add and delete the application behind, although it is very comfortable to use, but there are two points always let me not very cool.

Because Docker directly uses iptable to operate port forwarding, a large amount of data has been accumulated before without modifying iptablefail2banRules are broken, which means you have to endure a lot of scanner contamination.
Because the application runs a completely isolated scheme, the client IP that the sub-container can get is the IP of the virtual network card. If you want to collect data, you have to check the account scheme, which is more troublesome.

I asked the master before going out to start a business, they are directly in the container to allocate public network, no humanity, and direct allocation of public IP configuration projects are also many…

How to achieve the convenience of existing use with the least configuration and the least resource consumption becomes the main purpose of this article.

Before the end of the vacation, I reviewed the server operating environment and Traefik usage, and made a note of it, which may be helpful to you if you are struggling with o&M services.

You need to provide a standard system environment before you can fiddle with specific configurations.

Configure the Docker runtime environment based on Ubuntu 18.04

In the third quarter of 2018, Ubuntu 18.04 has been updated for nearly a quarter, and basically all the updated software has been updated, and all the compatible software has also been compatible. Docker experienced the change of name after the version of iteration, a large wave of programming software swarmed, and now the version has also been upgraded to V18.

Considering that the later software will do more and more backward compatible things, and the maintenance period of 16.04 is nearly half, I will use the latest system and software to configure the basic environment this time. CoreOS is not selected here because I have some other software requirements, and I want to use it in a relatively neutral environment.

Until now, this article has been written as follows: all the foreign mainstream cloud manufacturers I use support Ubuntu 18.04. Among the domestic cloud manufacturers, Ali Cloud supports Ubuntu 18.04, but Tencent Cloud does not support Ubuntu 18.04.

If your cloud host vendor supports the latest version of the system, you can directly refer to the following commands to configure the basic environment.

If your cloud host vendor does not support the upgrade, run the do-release-upgrade command to manually upgrade the software. During the upgrade, yes can be used. Install the package Maintainer’s version.

Let’s start with how to configure the most basic system environment.

First update the software package list, upgrade the software version to the latest stable version, and then in order to avoid the residual of some old things in the system to affect the software operation, we will try to uninstall the old version of the software operation, and install some common software.

apt update && apt upgrade -y
apt remove docker docker-engine docker.io
apt install -y apt-transport-https  ca-certificates curl software-properties-common
Copy the code

Add the Docker official GPG Key to the system, verify the validity of the Key, and update the warehouse source to the system. Ubuntu 18.04 will directly trigger the operation of pulling the package list, which is more user-friendly. Finally, directly type the install command. Install the Community Version of Docker.

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
apt-key fingerprint 0EBFCD88
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt install -y docker-ce
Copy the code

Find the latest stable version installed in https://github.com/docker/compose/releases.

The curl -l https://github.com/docker/compose/releases/download/1.22.0/docker-compose- ` uname-s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
Copy the code

At this point, the underlying Docker environment is installed.

Docker version 18.06.1- CE, Build E68fc7a
docker-compose version 1.22.0, build f46880fe
Storage Driver: overlay2
Copy the code

Simple system hardening

The basic modification of SSH port, circumvention scanner, should be everyone can, will not mention, if not can baidu or read the previous blog articles, we talk about uFW firewall configuration.

By default, the service is not activated. Before activating the service, you must first exempt the SSH port to avoid using VNC login to remedy the situation.

> ufw status
Status: inactive

ufw all SSH_PORT
Rules updated
Rules updated (v6)
Copy the code

Of course, if you don’t like changing SSH ports, you can just use the following command.

ufw disable

ufw reset
ufw default deny incoming
ufw default allow outgoing

ufw allow 22/tcp
ufw allow 80/tcp

ufw enable
Copy the code

Then activate the firewall state.

ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup
Copy the code

Once the firewall is up, you can also fiddle with fail2ban to significantly reduce routine scanner log harassment, as well as some rudimentary guessing and infiltration, which I’ll cover in more detail later.

Docker port binding conflicts with the UFW. Procedure

After configuring the firewall, you can test the configuration to see if it works. Start an nginx:alpine image that maps to port 80, and then access the server’s public IP address from a browser or command line. You can see the familiar nginx default welcome page.

root@VM-0-10-ubuntu:~# docker run --rm -p 80:80 nginx:alpine
Unable to find image 'nginx:alpine' locally
alpine: Pulling from library/nginx
911c6d0c7995: Pull complete 
131e13eca73f: Pull complete 
95376bf29516: Pull complete 
6717402ec973: Pull complete 
Digest: sha256:23e4dacbc60479fa7f23b3b8e18aad41bd8445706d0538b25ba1d575a6e2410b
Status: Downloaded newer image for nginx:alpine
114.xxx.xxx.xxx - - [27/Aug/2018:16:44:17 +0000] "The GET/HTTP / 1.1"200, 612,"-" "Mozilla / 5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" "-"
2018/08/27 16:44:18 [error] 6#6: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 114. Xxx.xxx. XXX, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "119.28.182.48", referrer: "Http://119.28.182.48/"
114.xxx.xxx.xxx - - [27/Aug/2018:16:44:18 +0000] "The GET/favicon. Ico HTTP / 1.1"404, 571,"http://119.28.182.48/" "Mozilla / 5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" "-"
Copy the code

The reason why uFW does not take effect here is that Docker uses iptable by default to add some forwarding rules and does not include uFW rules at all.

The recommended solution on the web is to turn this feature off, using something like the following.

echo '{"iptables": false}' | sudo tee /etc/docker/daemon.json > /dev/null
Copy the code

But if you do that, then you won’t be able to get the IP information that the client is accessing, there will be some minor issues with the containers accessing the Internet directly, and the containers communicating with each other, and then you’ll have to add some IPtables to fix that.

If you still choose to disable the ipTable feature and implement the container, then if you want to obtain the client IP, you can only use the following methods:

The front end starts a service behind an L7 Haproxy/nginx reverse proxy.
Change the running mode to host, abandon container full virtualization, and expose ports directly tohostAt this point, it will no longer be able to passdocker psCheck the status of your container portdocker network inspectThe port on the nic is enabled, which is not conducive to maintenance.
- If you’re using the classicdocker runOrder, then you need to cooperate--net=hostParameters.
- If you are using nonswarmFor the compose of the schema, you need to declare itnetwork_mode: "host", the compose version requires a statement 3.2 and above,portThe actual export does not require a cumbersome schema definition.
- Do not try to create a custom network for host, the choreographer version as of this article’s completion, and the Docker version, this feature is not supported.
Modify the IPtable forwarding rules of UFW and Docker to complete the forwarding mode you want.

It can be seen that no matter what kind of scheme, it is very tedious, and not conducive to repeated deployment, the cost of future debugging and maintenance is too high.

A better solution

Keep the default configuration and behavior of the Docker, but leave port control to the UFW and the outer gateway. The child container still uses expose and privates the port to the gateway.

Traefik can be used to automatically monitor docker daemon status without disabling Consul/ZK. Service discovery, load balancing, automatic availability switchover, and even automatic binding of domain name certificates.

I had to say that the over-pursuit of the full container solution led me to use Traefik in containers. The problem is that by the time the gateway receives the data, it has already been forwarded by at least two virtual network cards, which costs performance. The problem of losing the client IP and disabling the Docker iptable forwarding feature in order to ensure a higher level of security is enough in the above section.

After we decided to run Traefik “naked”, we needed to make some changes to its configuration, and HERE is a simple configuration that I believe can meet many common scenarios.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# global Settings
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

Enable debug mode (default off)
debug = true

# Log level (default ERROR)
logLevel = "INFO"

# Global entry point type (default HTTP)
defaultEntryPoints = ["https"."http"]

Do not report statistics
sendAnonymousUsage = false

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# Entry point setup
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

[entryPoints]

    # default front-end
    [entryPoints.http]
        address = ": 80"
        compress = true
        [entryPoints.http.redirect]
            entryPoint = "https"
    [entryPoints.https]
        address = ": 443"
        compress = true
    [entryPoints.https.tls]
        [[entryPoints.https.tls.certificates]]
            certFile = "/data/ssl/your.com.cer"
            keyFile = "/data/ssl/your.com.key"

    Console port
    [entryPoints.traefik-api]
        address = ": 4399"

    # Ping port
    [entryPoints.traefik-ping]
        address = ": 4398"


# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# Traefik File configuration
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

[file]
    [backends]
        [backends.dashboard]
            [backends.dashboard.servers.server1]
                url = "http://127.0.0.1:4399"
        [backends.ping]
            [backends.ping.servers.server1]
                url = "http://127.0.0.1:4398"

[frontends]
    [frontends.dashboard]
        entrypoints = ["https"]
        backend = "dashboard"
        [frontends.dashboard.routes.route01]
            rule = "Host:dashboard.your.com"
    [frontends.ping]
        entrypoints = ["https"]
        backend = "ping"
        [frontends.ping.routes.route01]
            rule = "Host:ping.your.com"
        [frontends.ping.routes.route02]
            rule = "ReplacePathRegex: ^/ /ping"

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# Traefik logs configuration
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

# Traefik logs
# Enabled by default and log to stdout
#
# Optional
#
# Default: os.Stdout
[traefikLog]
  filePath = "/data/logs/traefik.log"

[accessLog]
  filePath = "/data/logs/access.log"

# Format is either "json" or "common".
#
# Optional
# Default: "common"
#
# format = "common"

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
Access log configuration
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

# Enable access logs
# By default it will write to stdout and produce logs in the textual
# Common Log Format (CLF), extended with additional fields.
#
# Optional
#
# [accessLog]

# Sets the file path for the access log. If not specified, stdout will be used.
# Intermediate directories are created if necessary.
#
# Optional
# Default: os.Stdout
#
# filePath = "/path/to/log/log.txt"

# Format is either "json" or "common".
#
# Optional
# Default: "common"
#
# format = "common"

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# API and console configuration
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

# Enable API and console
[api]
    # entry point name
    entryPoint = "traefik-api"

    # Enable console (default)
    dashboard = true

    # Default protocol
    defaultEntryPoints = ["https"]

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# Ping configuration
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

# enable ping
[ping]
    # entry point name
    entryPoint = "traefik-ping"

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# Docker backend configuration
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

# Enable the Docker backend
[docker]

# Docker service backend
endpoint = "unix:///var/run/docker.sock"
# default domain name
domain = "traefix.your.com"
# Monitor docker changes
watch = true

Use custom templates (optional)
# filename = "docker.tmpl"

Expose the container by default (on by default)
# If the option is disabled, the container does not contain the 'traefik.enable=true' tag and will not be exposed
exposedbydefault = false

# use the IP address of the bound port instead of the internal private network (default off)
usebindportip = false

Swarm Mode (disabled by default)
swarmmode = false

# Enable docker TLS connection.
#
# Optional
#
# [docker.tls]
# ca = "/etc/ssl/ca.crt"
# cert = "/etc/ssl/docker.crt"
# key = "/etc/ssl/docker.key"
# insecureskipverify = true
Copy the code

Modify the configuration, save it, and run it:

traefik -c /etc/traefik.toml
Copy the code

Of course, you can’t access your Traefik gateway service at this point. Why? Because the software port binding is restricted to the UFW rules, we will update the UFW rules to allow external access to our ports 80 and 443.

root@VM-0-10-ubuntu:~# ufw allow 80
Rule added
Rule added (v6)
Copy the code

If you did it right, you should be able to access your site at this point.

Of course, there is no assistance from Docker Daemon, so process management needs to be supervised. It is recommended to use Supervisor for auxiliary management. It has been introduced in previous blogs more than once, so you can refer to it if you are interested.

[program:traefik]
command=traefik -c /etc/traefik.toml --sendAnonymousUsage=false
user=root
autostart=true
startsecs=3
startretries=100
autorestart=true
stderr_logfile=/data/traefik/error.log
stderr_logfile_maxbytes=50MB
stderr_logfile_backups=10
stdout_logfile=/dara/traefik/access.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
Copy the code

With your gateway in place, we found a directory and used a software image called WHOami to help us verify that the gateway was working as expected. In addition to automatic service discovery and load resolution, the gateway also provided functions including statistics, forwarding, header rewriting and so on.

version: '3'

services:
  whoami:
    image: emilevauge/whoami
    expose:
      - 80
    labels:
      - "traefik.enable=true"
      - "traefik.port=80"
      - "traefik.frontend.rule=Host:who.your.com"
Copy the code

Save the above configuration as docker-comemage.yml and run it in the background.

A browser or command line visit who.your.com to get the following information:

Hostname: b0cd60b18550 IP: 127.0.0.1 IP: 172.20.0.2 GET/HTTP/1.1 Host: who.your.com User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36 Accept: text/html,application/xhtml+xml,application/xml; Q = 0.9, image/webp image/apng, * / *; Q =0.8 Accept-encoding: gzip, deflate, BR Accept-language: zh-cn,zh; Q = 0.9, en. Q = 0.8, ja. Q =0 cache-control: max-age=0 upgrade-insecure -Requests: 1 X-Forwarded-for: 221.xxx.xx. XXX X-Forwarded-host: who.your.com X-Forwarded-Port: 443 X-Forwarded-Proto: https X-Forwarded-Server: VM-0-10-ubuntu X-Real-Ip: 221.xxx.xx.xxxCopy the code

You can see that service discovery, SSL certificate mounting, source IP forwarding, and so on all work correctly, and not surprisingly, QPS is much higher (at least one less layer of network forwarding, at least one less layer of full virtualization).

In addition, because the docker configuration is not modified, the container will not be allowed to access the Internet. Simple verification:

root@VM-0-10-ubuntu:/data/who# docker run it --rm alpine ping -c 1 8.8.8.8
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
8e3ba11ec2a2: Pull complete 
Digest: sha256:7043076348bf5040220df6ad703798fd8593a0918d06d3ce30c6c93be117e430
Status: Downloaded newer image forAlpine :latest PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: Seq =0 TTL =46 time=14.075 ms -- 8.8.8.8 ping statistics -- 1 packets transmitted, 1 packets transmitted, 0% packet loss round-trip min/ AVG/Max = 14.075/14.075/14.075msCopy the code

The next step is to upgrade each server and migrate the data.

Links to resources

In fact, there is a lot of discussion in the community about obtaining external IP in containers, such as #15086 and so on, involving different network card modes, different orchestration tools, different port mapping modes, and a lot of extended topics.

And Docker and UFW firewall feud, in fact, is a long talk about aging, but do not know why, online can see the information on the modification of iptable…

Uncomplicated Firewall (UFW) is not blocking anything when using Docker
Running Docker behind the ufw firewall

other

Hopefully this article has given you some additional insight into how you can use Traefik and Docker to do servitization.

I now have a small toss group, which gathered some like to toss small partners.

In the case of no advertisement, we will talk about software, HomeLab and some programming problems together, and also share some technical salon information in the group from time to time.

Like to toss small partners welcome to scan code to add friends. (Please indicate source and purpose, otherwise it will not be approved.

All this stuff about getting into groups

More complete Docker + Traefik use scheme

More complete Docker + Traefik use scheme

Configure the Docker runtime environment based on Ubuntu 18.04

Simple system hardening

Docker port binding conflicts with the UFW. Procedure

A better solution

Links to resources

other

Related Posts

CSS3 new feature selector

React 16 Architecture Research Notes (there are Easter eggs at the end of this article

Li Kou [Binary tree topics] 👊 226. Flipping binary trees