nginx.org/
nginx.org/download/
Nginx implements TCP proxy functionality in two ways:
One is to use the nginx_tcp_proxy_module module, commonly used in earlier versions of Nginx. One is to use the ngx_STREAM_CORE_module module for 1.9 and later. This article covers the latter, which is implementing the TCP proxy using stream.
Linux CentOS installation and use
1. Build nginx compilation environment Yum -y install GCC GCC c++ autoconf automake yum -y install zlib zlib-devel openssl openssl-devel pcre-devel # The PCRE library was designed to enable Nginx to support the rewrite module with URI rewriting capabilities. If you do not install the PCRE library, nginx cannot use the rewrite module functionality. # Nginx uses this module when using HTTPS services. If you do not install the openSSL package, an error will be reported during the installation of nginx. Openssl has been installed by default. You only need to install OpenSSL-devel. 2. Download the nginx source file, go to nginx.org/download/, and select the nginx-1.12.2.tar.gz file tar-zxf nginx-1.12.2.tar.gz 3. Configure –with-stream nginx supports stream implementation for TCP proxy only when configure uses –with-stream. The default path is nginx path prefix: “/usr/local/nginx”. You are not advised to change the path. ./configure –prefix=/opt/nginx –with-stream –with-http_stub_status_module make make install 4. Into/opt/nginx/conf/nginx. Conf, parameters configuration as follows, pay attention to add a stream at the end of the module (it and HTTP, etc.) at the same level, the module, similar to HTTP and mail module, allow us to configure a group to monitor the services of a TCP connection. Allows you to configure TCP connections for multiple services by configuring the proxy_pass directive in the upstream server group. event { worker_connections 1024; multi_accept on; use epoll; } stream {upstream backend {server 172.16.6.161:1883 max_fails=3 fail_timeout=30s; Server 172.16.6.161:1888; Server 172.16.6.162:1883; } server { listen 1884; proxy_connect_timeout 3s; proxy_timeout 300s; proxy_pass backend; }}
The meanings of the parameters in the event configuration are as follows:
Worker_connections specifies the maximum number of connections that can be received at the same time. It is important to note that the maximum number of connections is jointly determined by worker processes. (2) The multi_accept configuration specifies that nginx accepts as many connections as possible after being notified of a new connection. (3) The use of epoll configuration specifies the method of thread polling, using epoll for Linux2.6 + or Kqueue for BSD like Mac
The meanings of the parameters in the stream configuration are as follows:
(1)listen specifies that Nginx will listen on TCP port 1884. (2)proxy_connect_timeout Specifies the timeout period for establishing a connection between Nginx and the proxied host. Syntax: proxy_connect_timeout; Default: proxy_connect_timeout 60s; Context: stream, server (3)proxy_timoute specifies the maximum interval between two adjacent read or write operations between Nginx and the client and between Nginx and the proxied host. If no data read or write operation occurs after this interval, the connection is disconnected. Firecat It is recommended to set the time to a longer time. Otherwise, the TCP connection will be disconnected. Syntax: proxy_timeout ; Default: proxy_timeout 10m; Context: stream, server (4)proxy_pass specifies the address and port number of the proxied host.
5. Go to /opt/nginx/sbin/ and the nginx execution file is displayed. View the compile-time parameters
./nginx -V
6. Check the syntax of the nginx.conf configuration file to prevent website restarts or reloading due to incorrect configuration./nginx -t 7. Start the nginx. / nginx
8. Check whether nginx is started successfully
netstat -lnpt | grep nginx
Or ps – ef | grep nginx see if nginx process to start the “recommended”
Or netstat nalp | grep 80 to see the process of port 80
9. Restart nginx
. / nginx -s reload the termination nginx kill 9 number > < nginx process (usually have 2 ~ 3 process, through the netstat – LNPT | grep nginx command can query process)
For example, kill -9 36533
2. Actual measurement
1. I tested only one server with its IP address 172.16.6.161.
(1) Configure the TCP port of the Nginx server to 1884
(2) Set TCP port 1883 for work server 1
(3) Set the TCP port of work server 2 to 1888
The client initiates a TCP connection to the Nginx server (172.16.6.161, 1884), and everything works fine.
2. I set the worker_connections parameter of event to 10, and the result is that only 3 TCP connections are successful.
# check the file descriptor usage:
lsof -i -n -P
lsof -i -n -P | grep :1884
# check 1884 port connection, observe the TCP state diagram netstat nalp | grep, 1884
Check the number of client connections for port 1884
netstat -nalp|grep 1884|wc -l
# change the maximum number of files in the current process ulimit -n 102400
3. I actually use client to initiate 100 TCP connections to nginx server, and Nginx will correspondingly establish 100 TCP connections with upstream Real Server. In addition, 10,000, 20,000, and 40,000 were the same. I use netstat nalp | 1884 | grep wc – and netstat – na l | grep ESTABLISHED | wc – l command and the wireshark tool detection: The client establishes TCP connections to nginx, and Nginx also establishes TCP connections to the upstream server. The number of front-end and back-end TCP connections is the same. The Wireshark can capture packets and view the three-way handshake.
4, test tools and source code
My personal Linux TCP Server test source code, C language
TCP_UDP_PerformanceTest
Powerful TcpServer stress test tool source code (attached to break the connection limit method and tools)
TCPCOPY:github.com/session-rep…
Iii. Personal summary
HTTP load balancing, which is usually all “layer 7 load balancing”, works at layer 7, the application layer. TCP load balancing, known as “layer 4 load balancing”, works at the “network layer” and the “transport layer”. For example, LVS (Linux Virtual Server) and F5 (a hardware load balancer) are also layer 4 load balancers.
Layer 4 load balancing: Only one TCP connection is established.
Layer 7 load balancer: A TCP connection is established between the load balancer and the client and back-end server respectively. That is, two TCP connections.
Regarding layer 4 load balancing, I have found that this is not true for Nginx, which still establishes a separate TCP connection. But there’s a problem with that, right? Is the Nginx server limited by port number 65535 when making TCP connections to the worker server? Does nginx maintain TCP long connections with clients and upstream servers as layer 4 loads? Does nginx connect to the client and upstream each time a client comes in? Same number of connections on both sides? Nginx has a limit on the number of TCP ports to connect upstream.
2. In my opinion, both Haproxy /Nginx have the problem of TCP source port exhaustion. As a reverse proxy, Haproxy /Nginx uses its OWN IP address as the source address to connect to the backend server. Based on THE TCP protocol, an operating system of any type can have only 64K source TCP ports for initiating TCP connections.
3. How to solve the problem of port exhaustion?
Haproxy works, so Nginx might as well give it a try. See the blog for details:
My personal haProxy-1.7.9 practice: Install, Compile and Test (★ FiRECat recommended, for TCP layer 4 load balancing ★)
Configuring haproxy to use multiple LAN Intranet IP addresses as load balancers to overcome the problem that haproxy supports only 64k connections (exceeding the 65535 port limit for a single IP address). Haproxy TCP source ports are exhausted (a network adapter has a maximum of 65535 ports)
Internet of Things Architecture Growth Path (10)-Nginx load Balancing – it talks about adding virtual network cards to achieve multiple IP
4. Implementation principle of TCP load balancing
When Nginx receives a new client connection from the listening port, it immediately executes a routing scheduling algorithm, obtains the specified service IP address to connect to, and then creates a new upstream connection to the specified server.
TCP load balancing supports the original Nginx scheduling algorithm, including Round Robin (default, Round Robin scheduling) and hash (select consistent). The scheduling information data also works with the robustness detection module to select the appropriate target upstream server for each connection. If using Hash load balancing scheduling, you can use $remote_ADDR (client IP) for a simple persistent session (connections from the same client IP always end up on the same server).
Like other upstream modules, the TCP Stream module supports custom load-balanced forwarding weights (weight=2) and backup and Down parameters for kicking down failed upstream servers. The max_conns parameter can limit the number of TCP connections on a server and set a proper value based on the server capacity, especially in high-concurrency scenarios, to protect against overload.
Nginx monitors client connections and upstream connections. Once data is received, Nginx immediately reads and pushes it to the upstream connection, without checking the data within the TCP connection. Nginx maintains a memory buffer for client and upstream data writes. If the client or server transfers a large amount of data, the buffer increases the size of the memory appropriately.
When Nginx receives a connection closure notification from either party, or the TCP connection is idle for longer than proxy_timeout, the connection will be closed. For TCP long connections, we should select an appropriate proxy_timeout and pay attention to the SOCke so_keepalive parameter to prevent premature disconnection.
5. Service robustness monitoring
The TCP load balancing module supports built-in robustness detection. If an upstream server rejects TCP connections after proxy_CONNECt_TIMEOUT, it is considered invalid. In this case, Nginx immediately tries to connect to another normal server in the upstream group. Connection failures will be recorded in the Nginx error log.
If a server repeatedly fails (exceeding the max_fails or fail_TIMEOUT configuration parameters), Nginx will also kick that server down. Nginx will occasionally try to reconnect a server 60 seconds after it has been kicked off to see if it is back to normal. If the server returns to normal, Nginx adds it back to the upstream group and slowly increases the percentage of connection requests.
Because typically a service has “hot data”, that is, 80% or more of the requests are actually blocked in the “hot data cache” and only a small percentage of the requests are actually processed. The “hot data cache” is not actually established at the time the machine is started up, and the sudden deluge of requests is likely to cause the machine to “take up” and fail again. In mysql, for example, more than 95% of our mysql queries are in the memory cache, and not many queries are actually executed. In fact, whether a single machine or a cluster has a high number of concurrent requests, restarting or switching may cause this risk. The solutions are as follows: (1) The number of requests increases gradually from a small number to a large number, and hotspot data accumulates gradually to reach the normal service status. (2) Prepare the “commonly used” data in advance, take the initiative to “warm up” the service, and then open the access to the server after the warm-up.
TCP load balancing is similar to LVS in principle. It works at a lower level and performs much better than the original HTTP load balancing. However, no better than LVS, which is placed in kernel modules, Nginx works in user mode, and Nginx is relatively heavy.
Four, other people’s home configuration
worker_processes auto; worker_cpu_affinity auto; error_log logs/error.log error; pid logs/nginx.pid; worker_rlimit_nofile 100000; events { use epoll; worker_connections 5000; } stream { upstream server { hash $remote_addr consistent; Server 172.16.1.11:8081 weight=1 max_fails=3 Fail_timeout =10s; Server 172.16.1.22:8081 weight=1 max_fails=3 FAIL_timeout =10s; } server { listen 8081; proxy_connect_timeout 1s; proxy_timeout 30s; proxy_pass server; }
}
Ps – ef | grep nginx see if nginx process can start
——
Internet of Things Architecture Growth Path (10)-Nginx load Balancing – it talks about adding virtual network cards to achieve multiple IP
Configure TCP and UDP load balancing
Welcome to sister:
My personal haProxy-1.7.9 practice: Install, Compile and Test (★ FiRECat recommended, for TCP layer 4 load balancing ★)