Nginx long and short connections

Nginx short connection

Let’s take a look at Nginx short connections.

Short connection for packet capture on Nginx

Tcpdump displays the result

[root@nginx-plus-2 ~]Tcpdump -i ens192 host 192.168.1.225 and TCP -a -s 0 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes

TCP three-way handshake
13:06:10.674757 IP 192.168.1.135.45750 > 192.168.1.225.webcache: Flags [S], seq 1889946011, win 29200, options [mss 1460,sackOK,TS val 2776265370 ecr 0,nop,wscale 7], length 0E.. The <; . @. @. {... p.I....... r............ .zr......... 13:06:10. 675391IP 192.168.1.225.webcache > 192.168.1.135.45750: Flags [S.], seq 1966123495, ack 1889946012, win 28960, options [mss 1460,sackOK,TS val 999954489 ecr 2776265370,nop,wscale 7], length 0
E..<..@[email protected] :..........
;..9.zr.....
13:06:10.675431 IP 192.168.1.135.45750 > 192.168.1.225.webcache: Flags [.], ack 1, win 229, options [nop,nop,TS val 2776265371 ecr 999954489], length 0
E..4; . @. @. {... p.I.u0............. .zr.; . 9#Nginx sends a GET request to Tomcat13:06:10. 675511IP 192.168.1.135.45750 > 192.168.1.225.webcache: Flags [P.], seq 1:158, ack 1, win 229, options [nop,nop,TS val 2776265371 ecr 999954489], length 157: HTTP: GET / HTTP/1.0E... ; .@[email protected]............ p.I.u0....... |... .zr.; . 9GET / HTTP/1.0
X-Forwarded-For: 192.168.1.35
X-Real-IP: 192.168.1.35
Host: 192.168.1.225:8080
Connection: close  # short connection
User-Agent: curl/7.64.1
Accept: */*


# What the upstream TOMCAT server returns
13:06:10.675759 IP 192.168.1.225.webcache > 192.168.1.135.45750: Flags [.], ack 158, win 235, options [nop,nop,TS val 999954490 ecr 2776265371], length 0
E..4. @. @..9. u0.. p.J9........... ; . : zr. 13:06:10. 682146IP 192.168.1.225.webcache > 192.168.1.135.45750: Flags [P.], seq 1:262, ack 158, win 235, options [nop,nop,TS val 999954496 ecr 2776265371], length 261: HTTP: HTTP/1.1 200 OK
E..9. @. @..3. u0.. p.J9..... h..... ; . @. Zr. HTTP / 1.1 200OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=A81D89FB21B81086B2ADE210BE79C825; Path=/; HttpOnlyContent-Type: text/html; charset=ISO-8859-1 Content-Length: 20 Date: Mon, 07Sep 2020 05:10:21 GMT
Connection: close

Welcome to Tomcat1


#Nginx responds to Tomcat's ack for returning content
13:06:10.682182 IP 192.168.1.135.45750 > 192.168.1.225.webcache: Flags [.], ack 262, win 237, options [nop,nop,TS val 2776265378 ecr 999954496], length 0
E..4; . @. @. {... p.J9u0............. .zr.; . @# Four waves, Nginx disconnects13:06:10. 682352IP 192.168.1.135.45750 > 192.168.1.225.webcache: Flags [F.], seq 158, ack 262, win 237, options [nop,nop,TS val 2776265378 ecr 999954496], length 0
E..4; . @. @. {... p.J9u0............. .zr.; . @Tomcat returns a FIN and ACK composite13:06:10. 682429IP 192.168.1.225.webcache > 192.168.1.135.45750: Flags [F.], seq 262, ack 158, win 235, options [nop,nop,TS val 999954497 ecr 2776265378], length 0
E..4. @. @..7. u0.. p.J9........... ; . A.z r. 13:06:10. 682449IP 192.168.1.135.45750 > 192.168.1.225.webcache: Flags [.], ack 263, win 237, options [nop,nop,TS val 2776265378 ecr 999954497], length 0
E..4; . @. @. {... p.J:u0............. .zr.; .A


# I don't know why this paragraph is added. If the client accesses Tomcat directly, there is no such paragraph
13:06:10.682522 IP 192.168.1.225.webcache > 192.168.1.135.45750: Flags [.], ack 159, win 235, options [nop,nop,TS val 999954497 ecr 2776265378], length 0
E..4. @. @..6. u0.. p.J:........... ; . A.zr.Copy the code

The Wireshark displays the results

A lot of TIME_WAIT situations occur
- There are two situations that can cause a lot of TIME_WAIT on the Nginx side:
  - Keepalive_requests is set to a small value. If this value is exceeded for high concurrency, nginx will force the client to close keepalive connections. (nginx actively closes the connection causing nginx TIME_WAIT)
  - The keepalive setting is too small (the number of idle connections is too small). As a result, nginx will frequently have connection number flapping (if the number exceeds this value, the connection will be closed), and constantly close or open the keepalive connection held by the back-end server.
- Cause a lot of TIME_WAIT on the backend server :(nginx closes the connection)

Nginx does not open a long connection to the backend, i.e. proxy_http_version 1.1 is not set. And proxy_set_header Connection “”; This results in a large number of time_waits on the server side in high concurrency each time the back-end server closes a connection.

A long connection

Nginx configures long connections

1, 2, and 3 must be configured

events {}
http {

    log_format haha '$remote_addr - $remote_user [$time_local] "$request" '
    '$status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for" $server_name $upstream_addr $upstream_status $request_time $upstream_response_time';
    upstream cr7back {
    #1. The maximum number of free connections in the pool,
    The keepalive parameter must be set carefully, especially for high QPS scenarios.
    # It is recommended to do an estimate and calculate the number of long connections required based on QPS and average response time.
    With 10000 QPS and 100 ms response time, the number of long connections required is approximately 1000.
    # Then set keepalive to 10% to 30% of the number of long connections.
        keepalive 10;  
        keepalive_requests 1000;  Set the maximum number of requests that can be serviced on a keep-alive connection. When the maximum number of requests is reached, the connection is closed. The default is 100.
        server 192.168.1.225:8080;
    }
    server {
#Nginx supports HTTP keep-alive connections by default.
The default timeout is 75 seconds, during which subsequent HTTP requests can reuse established TCP connections.
# Save the time and resources of creating a TCP handshake.
       keepalive_timeout 60 60;  
        access_log /var/log /nginx/haha.log haha;
        listen 80;
        location / {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Connection "keep-alive"; #2. Set Connection to a long Connection. Default is close
            proxy_http_version 1.1;  #3. Long connections are only supported in http1.1, and are not supported by 1.0 by default
            proxy_passhttp: //cr7back; }}Copy the code

Keepalive_timeout instruction

Grammar:keepalive_timeoutTimeout [header_timeout] Default value: keepalive_timeout75s; Use fields: HTTP, server,locationParameter: parameter1Specifies the keep-alive timeout of the client and server0The keep-alive function is disabled. parameter2(Optional) in the HTTP response header"Keep-Alive: timeout=time"Field sets a value in. Note that Connection: keep-alive does not indicate a long Connection.Copy the code

Nginx packet capture long connection

Tcpdump displays the result

[root@nginx-plus-2 ~]Tcpdump -i ens192 host 192.168.1.225 and TCP -a -s 0 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes

TCP three-way handshake
14:53:10.519365 IP 192.168.1.135.52224 > 192.168.1.225.webcache: Flags [S], seq 3447497727, win 29200, options [mss 1460,sackOK,TS val 2782685215 ecr 0,nop,wscale 7], length 0E.. The <.. @. @.."... |... r............ . h......... 14:53:10.520031 IP 192.168.1.225.webcache > 192.168.1.135.52224: Flags [S.], seq 1296273318, ack 3447497728, win 28960, options [mss 1460,sackOK,TS val 1006374594 ecr 2782685215,nop,wscale 7], length 0 E.. The <.. @. @... MC... |... q .d......... ; . h..... 14:53:10.520069 IP 192.168.1.135.52224 > 192.168.1.225.webcache: Flags [.], ack 1, win 229, options [nop,nop,TS val 2782685216 ecr 1006374594], length 0 E.. 4.. @. @..) . |.. MC............. . h ; . 14:53:10.520133 IP 192.168.1.135.52224 > 192.168.1.225. Webcache: Flags [P.], seq 1:152, ack 1, win 229, options [nop,nop,TS val 2782685216 ecr 1006374594], length 151: HTTP: GET/HTTP / 1.1 E... @. @... |.. MC....... v..... . h ; . GET/HTTP/1.1 X-Forwarded-For: 192.168.1.35 X-real-IP: 192.168.1.35 Connection: keep-alive # Cr7back the user-agent: curl / 7.64.1 Accept: 14:53:10.520450 IP 192.168.1.225.webcache > 192.168.1.135.52224: Flags [.], ack 152, win 235, options [nop,nop,TS val 1006374594 ecr 2782685216], length 0 E.. 4.e@[email protected]............. MC... |... ] . ; . H 14:53:10.522514 IP 192.168.1.225.webcache > 192.168.1.135.52224: Flags [P.], seq 1:243, ack 152, win 235, options [nop,nop,TS val 1006374596 ecr 2782685216], length 242: HTTP: HTTP / 1.1 200 OK E.. &.f@[email protected]............. MC... |... ; . HTTP / 1.1 h 200 OK Server: Apache - Coyote / 1.1 Set - cookies: JSESSIONID = F1102B1DF0A84DF7C477224D43BF2467; Path=/; HttpOnly Content-Type: text/html; charset=ISO-8859-1 Content-Length: 20 Date: Mon, 07 Sep 2020 06:57:21 GMT Welcome to Tomcat1 #Nginx for ack 14:53:10.522535 IP 192.168.1.135.52224 > 192.168.1.225. webCache: Flags [.], ACK 243, win 237, options [nop, NOp,TS val 2782685218 ECR 1006374596], length 0 E.. 4.. @. @.. '... |.. MC............. . h"; .# You can see that Nginx is not actively disconnecting
Copy the code

The Wireshark displays the results

TCP Keepalive and HTTP Keepalive

HTTP keep-alive is not the same as TCP keep-alive. HTTP keep-alive is designed to keep TCP alive longer so that multiple HTTP packets can be sent over the same connection, increasing socket efficiency. The TCP keep-alive mechanism checks the status of TCP connections. TCP Keep-alive preservation timer supports three kernel configuration parameters:

Tcp_keepalive_time # Indicates the keepalive idle period. The default value is 7200s. Tcp_keepalive_intvl # Indicates the interval for sending keepalive probe packets The number of times the probe packet is sent is 9 by defaultCopy the code

Temporary modification:

echo 1800 > /proc/sys/net/ipv4/tcp_keepalive_time
echo 15 > /proc/sys/net/ipv4/tcp_keepalive_intvl
echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes
Copy the code

Permanent change: Edit /etc/sysctl.conf

net.ipv4.tcp_keepalive_time=1800
net.ipv4.tcp_keepalive_intvl=15
net.ipv4.tcp_keepalive_probes=9
Copy the code

After the modification is complete, sysctl -p takes effect.

After a TCP connection is established at both ends of the network, the server kernel sends a detection packet to the client to determine the STATUS of the TCP connection (the client may crash, the application may be forcibly shut down, the host may be unreachable, etc.) after the TCP connection is idle (no data flows are sent between the two sides) and tcp_keepalive_time seconds. If no ack packet is received, tcp_keepalive_intvL attempts to send the probe packet again until the ack packet is received. If no ACK packet is received, tcp_keepalive_probes tries several times. The interval time here is 15s, 30s, 45s, 60s, 75s respectively. If no ACK packet is received after repeated tcp_keepalive_probes attempts, the TCP connection is discarded. The default idle time of TCP connection is 2 hours. Generally, 30 minutes is sufficient. That is, only if nginx’s keepalive_timeout value is set higher than tcp_keepalive_time and the last HTTP response sent from this TCP connection has elapsed after tcp_keepalive_time, The operating system sends a detection packet to determine whether to discard the TCP connection. This usually doesn’t happen, unless you need to.

Nginx short connection

Short connection for packet capture on Nginx

A long connection

Nginx configures long connections

Nginx packet capture long connection

TCP Keepalive and HTTP Keepalive

Related Posts

The most complete Redis operation, read this is enough

Visual bucket sorting algorithm

SQL