Over the weekend, I discussed some TCP problems with my friends. When I looked up the book “Linux Server High-performance Programming”, I found this sentence written in the book:
According to the book, when a connection in TIME_WAIT state receives a SYN of the same quad, it sends an RST message and disconnects.
In the book, the author just mentioned such a sentence, without giving evidence of source code or captured package map.
At first, I saw this logic and thought it made sense, but when I went to nibble on the TCP source code, IT didn’t.
So, today’s question is, “What happens when a connection in TIME_WAIT state receives a SYN with the same quad during a normal TCP wave?”
The symptom is shown in the following figure, with the server on the left and the client on the right:
Say first conclusion
Before I analyze THE TCP source code with you, I will directly say to you the conclusion.
In this case, the key is to check whether the SEQUENCE number and timestamp of the SYN is valid. After receiving a SYN, a connection in TIME_WAIT state determines whether the sequence number and timestamp of the SYN is valid and performs different actions according to the result.
What is a “legitimate” SYN?
- Valid SYN: The sequence number of the client’s SYN is larger than the sequence number expected next from the server, and the timestamp of the SYN is larger than the timestamp of the last packet received from the server.
- Illegal SYN: The sequence number of the client’s SYN is smaller than the sequence number expected to be received next on the server, or the timestamp of the SYN is smaller than the timestamp of the last packet received on the server.
The SYN is valid in the scenario where TCP timestamp is enabled on both parties. If TCP timestamp is not enabled on both parties, the SYN is valid as follows:
- Valid SYN: The client’s SYN has a larger sequence number than the server expects to receive next.
- Illegal SYN: The client’s SYN has a smaller sequence number than the server’s expected next.
Received a valid SYN
If a connection in TIME_WAIT state receives a “valid SYN,” the quad connection is reused, the 2MSL is skipped and the SYN_RECV state is set up.
Using the following example, both parties have TCP timestamp enabled. TSval is the timestamp when a packet is sent:
In the preceding figure, when a FIN packet is received for the third wave, TSval (21) of the packet is recorded and stored in the ts_recent variable. It then computes the next expected sequence number, which in this case is 301, stored in the rcv_nxt variable.
A connection in TIME_WAIT state receives a SYN that is “valid” because its SEq (400) is greater than rcv_NXt (301) and its TSval (30) is greater than ts_recent (21). This quad connection is then reused, skipping 2MSL to the SYN_RECV state, and the connection process can proceed.
An invalid SYN was received. Procedure
If the connection in TIME_WAIT state receives an invalid SYN, the client replies with an ACK packet for the fourth wave. After receiving the ACK num, the client replies with an RST packet to the server.
Using the following example, both parties have TCP timestamp enabled. TSval is the timestamp when a packet is sent:
In the preceding figure, when a FIN packet is received for the third wave, TSval (21) of the packet is recorded and stored in the ts_recent variable. It then computes the next expected sequence number, which in this case is 301, stored in the rcv_nxt variable.
A connection in TIME_WAIT state receives an “invalid SYN” because its SEQ (200) is less than rcv_NXt (301). The connection responds with an ACK packet that is the same as the fourth wave. If it finds that it does not expect to receive the confirmation number, it sends an RST packet to the server.
After the client does not receive the SYN + ACK packet after a period of time, the client retransmits the SYN packet timeout. When the number of retransmission times reaches the maximum, the client disconnects.
PS: If the connection is in TIME_WAIT state, will it be disconnected after receiving the RST?
Source code analysis
The following source code analysis is based on Linux 4.2 version of the kernel code.
After receiving a TCP packet, the Linux kernel executes the tcp_v4_rCV function. The main codes related to TIME_WAIT state are as follows:
int tcp_v4_rcv(struct sk_buff *skb)
{
struct sock *sk;.// After receiving the packet, this function is called to search for the corresponding sock
sk = __inet_lookup_skb(&tcp_hashinfo, skb, __tcp_hdrlen(th), th->source,
th->dest, sdif, &refcounted);
if(! sk)goto no_tcp_socket;
process:
// If the connection status is time_WAIT, do_time_wait will be jumped
if (sk->sk_state == TCP_TIME_WAIT)
gotodo_time_wait; . do_time_wait: ...The tcp_timeWAIT_state_process function processes packets received in time_wait state
switch (tcp_timewait_state_process(inet_twsk(sk), skb, th)) {
// If TCP_TW_SYN is used, this SYN is allowed to re-establish the connection
// allow TIM_WAIT state to jump to SYN_RECV
case TCP_TW_SYN: {
struct sock *sk2 =inet_lookup_listener(....) ;if (sk2) {
....
gotoprocess; }}// if it is TCP_TW_ACK, then return the ACK in memory
case TCP_TW_ACK:
tcp_v4_timewait_ack(sk, skb);
break;
// If it is TCP_TW_RST, send the RESET packet directly
case TCP_TW_RST:
tcp_v4_send_reset(sk, skb);
inet_twsk_deschedule_put(inet_twsk(sk));
goto discard_it;
// if it is TCP_TW_SUCCESS, the packet is discarded without any response
case TCP_TW_SUCCESS:;
}
goto discard_it;
}
Copy the codeThe procedure for this code:
- After receiving a packet, the command is called
__inet_lookup_skb()Function to find the corresponding SOCK structure; - If the state of the connection is
TIME_WAIT, will jump to do_time_wait processing; - by
tcp_timewait_state_process()Function to process the received message and perform corresponding processing according to the returned value.
Just to be clear, if the SYN received is valid, the tcp_timewait_state_process() function returns TCP_TW_SYN and then reuses the connection. If the SYN received is invalid, the tcp_timewait_state_process() function returns TCP_TW_ACK and then the last ACK.
Next, look at how the tcp_timewait_state_process() function determines SYN packets.
enum tcp_tw_status
tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
const struct tcphdr *th)
{...//paws_reject is false, indicating that no timestamp wrapping has occurred
//paws_reject is true, indicating that timestamp wrapping has occurred
bool paws_reject = false;
tmp_opt.saw_tstamp = 0;
//TCP header has options and old connection has timestamp option enabled
if (th->doff > (sizeof(*th) >> 2) && tcptw->tw_ts_recent_stamp) {
// Parse options
tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0.NULL);
if (tmp_opt.saw_tstamp) {
...
// Check whether the timestamp of the received packet is timestamp wrappedpaws_reject = tcp_paws_reject(&tmp_opt, th->rst); }}...// It is a SYN packet with no RST, no ACK, no timestamp wrapped, and no sequence wrapped.
if(th->syn && ! th->rst && ! th->ack && ! paws_reject && (after(TCP_SKB_CB(skb)->seq, tcptw->tw_rcv_nxt) || (tmp_opt.saw_tstamp &&// The new connection has a timestamp enabled
(s32)(tcptw->tw_ts_recent - tmp_opt.rcv_tsval) < 0))) { // The timestamp is not wrapped
// Initialize the serial number
u32 isn = tcptw->tw_snd_nxt + 65535 + 2;
if (isn == 0)
isn++;
TCP_SKB_CB(skb)->tcp_tw_isn = isn;
return TCP_TW_SYN; // Allow reusing TIME_WAIT quads to re-establish connections
}
if(! th->rst) {// If the timestamp is wrapped, or the message contains an ACK, the duration of the TIMEWAIT state is extended again
if (paws_reject || th->ack)
inet_twsk_schedule(tw, &tcp_death_row, TCP_TIMEWAIT_LEN,
TCP_TIMEWAIT_LEN);
// return TCP_TW_ACK to send the last ACK
return TCP_TW_ACK;
}
inet_twsk_put(tw);
return TCP_TW_SUCCESS;
}
Copy the codeIf the TCP timestamp mechanism is enabled, the tcp_paws_reject() function is used to determine whether the timestamp is wrapped, that is, whether the timestamp of the current received packet is greater than the timestamp of the last received packet:
- If greater than, no timestamp loopback occurred and the function returns false.
- If less than, timestamp winding occurred, and the function returns true.
When a SYN packet is received, the sequence number of the SYN packet is “greater” than the expected sequence number if the timestamp of the SYN packet is not wound, and the sequence number is not wound. A sequence number is initialized, TCP_TW_SYN is returned, the connection is reused, and 2MSL is skipped to the SYN_RECV state, and the connection is established.
If TCP timestamp is not enabled on either side, you only need to determine whether the SEQUENCE number of the SYN packet has been wrapped. If the SEQUENCE number of the SYN packet is greater than the expected sequence number next time, you can skip 2MSL and reuse the connection.
If the SYN packet is invalid, TCP_TW_ACK is returned, and the same ACK as the last one is sent.
In TIME_WAIT state, will the connection be disconnected after receiving an RST?
I left a question earlier: will connections in TIME_WAIT state be disconnected after receiving RST?
Net.ipv4. tcp_rfc1337 (default: 0) : net.ipv4.tcp_rfc1337
- If this parameter is set to 0, the TIME_WAIT state is terminated in advance after RST packets are received.
- If this parameter is set to 1, RST packets are discarded.
Source code processing is as follows:
enum tcp_tw_status
tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
const struct tcphdr *th)
{...// The timestamp of the RST packet is not wrapped
if(! paws_reject && (TCP_SKB_CB(skb)->seq == tcptw->tw_rcv_nxt && (TCP_SKB_CB(skb)->seq == TCP_SKB_CB(skb)->end_seq || th->rst))) {// Process RST packets
if (th->rst) {
// If this option is not enabled, tw will be reclaimed immediately when RST is received, but this is risky
if (twsk_net(tw)->ipv4.sysctl_tcp_rfc1337 == 0) {
kill:
// Delete the TW timer and release the TW
inet_twsk_deschedule_put(tw);
returnTCP_TW_SUCCESS; }}else {
// Re-extend the duration of TIMEWAIT stateinet_twsk_reschedule(tw, TCP_TIMEWAIT_LEN); }...returnTCP_TW_SUCCESS; }}Copy the codeTIME_WAIT indicates that a connection is released after receiving an RST packet. If this is done, 2MSL of time is skipped.
Sysctl_tcp_rfc1337 is proposed in RFC 1337 to avoid skipping 2MSL when RST packets are received in TIME_WAIT state. The document also describes potential problems caused by skipping 2MSL.
There are two main reasons why the TIME_WAIT state lasts for 2MSL:
- Prevent historical connections from being incorrectly received by subsequent connections of the same quad;
- Ensure that the “passive closing connection” can be properly closed;
What is the problem if the TIME_WAIT state is too short or does not exist?
While the TIME_WAIT state can be a bit long and unfriendly, it is designed to avoid clutter.
The book UNIX Network Programming says TIME_WAIT is our friend, it’s helpful, don’t try to avoid it, just figure it out.
So, I personally feel it’s safer to set net.ipv4.tcp_rFC1337 to 1.
conclusion
What happens when a connection in TIME_WAIT state receives a SYN from the same quad during normal TCP waving?
If both parties have time stamps enabled:
- If the sequence number of the client’s SYN is larger than the sequence number expected to be received next on the server, and the timestamp of the SYN is larger than the timestamp of the last packet received on the server. The quad connection is reused, the 2MSL is skipped, the SYN_RECV state is changed, and the connection process can proceed.
- If the sequence number of the client’s SYN is smaller than the sequence number expected to be received next on the server, or the timestamp of the SYN is smaller than the timestamp of the last packet received on the server. Then, the client replies an ACK packet for the fourth wave. After receiving the ACK packet, the client finds that it does not expect to receive the ACK number and sends an RST packet to the server.
In TIME_WAIT state, will the connection be disconnected after receiving an RST?
- if
net.ipv4.tcp_rfc1337If the parameter is 0, the TIME_WAIT state is terminated in advance and the connection is released. - if
net.ipv4.tcp_rfc1337If the parameter is 1, the RST packet is discarded.
Finished!