The background,
Review of Production congestion
Recently, in the process of a production release, there was a traffic jam due to sudden traffic. The system deployment diagram is as follows. The client accesses the Dubbo consumer through Http, and the consumer accesses the service provider through Dubbo. This is a single machine room, 8 consumers and 3 providers, a total of two machine rooms for external services.
An analysis of Dubbo congestion
During the release process, one machine room is removed, and the other machine room is used for external services. The removed machine room releases a new version, and then the new version is exchanged. Finally, both machine rooms use the new version for external services. The problem appears when the single room external service, at this time the single room or the old version of the application. I did not know that there would be a peak in the evening, but the peak of the evening and the peak of the morning were almost the same, and the single room could not carry such a large flow, resulting in congestion. These flows are characterized by high concurrency and large return packets for individual transactions. Because it is a product list page, multiple transactions will be sent to the background after being clicked.
When the problem occurred, because the status was not clear, the first cut to another machine room, the result was also congested, and finally the whole back, tossed for a period of time no problem. There were some phenomena:
(1) The CPU memory of the provider is not high. The maximum CPU of the first machine room is 66%(8-core VM), and the maximum CPU of the second machine room is 40%(16-core VM). Maximum consumer CPU is only 30% + (two consumer nodes on the same VIRTUAL machine)
(2) During congestion, the service provider’s Dubbo business thread pool (more on this thread pool below) is not full, up to 300 and up to 500. But when the machine room was removed, meaning there was no external traffic, the thread pool was full and it took several minutes to process the backlog of requests.
(3) The number of requests per second into the Dubbo business thread pool calculated by the monitoring tool is sometimes 0 and sometimes extremely large in the congestion. In normal days, this value does not exist when it is 0.
Guess the cause of the accident
Other indicators didn’t detect abnormalities, also have no dozen Dump, we and our Dubbo configuration by analyzing this phenomenon, speculation is occurred in the network congestion, and the key parameters affecting the jam is Dubbo agreement number of connections, we default to a single connection, the quantity is less, but consumers have not been able to fully use the network resources.
By default, each Dubbo consumer establishes a long connection with the Dubbo provider. The official Dubbo recommendation for this is:
The default Dubbo protocol uses a single long connection and NIO asynchronous communication, which is suitable for small data volume and large concurrent service invocation, and the number of service consumer machines is much larger than the number of service provider machines.
Conversely, the default Dubbo protocol is not suitable for services that transmit large amounts of data, such as files and videos, unless the request volume is very low.
(dubbo.apache.org/zh-cn/docs/…).
Here are some answers to frequently asked questions from Dubbo:
Why should there be more consumers than providers?
Because the DuBBo protocol uses a single long connection, assuming that the network is a GIGABit nic, according to the test experience data, each connection can be loaded with a maximum of 7 Mbytes (it may be different in different environments, for your reference). Theoretically, a service provider needs 20 service consumers to load the NIC.
Why can’t you pass a big bag?
The dubbo protocol uses a single long connection. If the packet size of each request is 500KByte and the network is a gigabit nic, the maximum value of each connection is 7MByte(it may be different in different environments, for your reference). The maximum value of TPS(transactions processed per second) of a single service provider is: 128MByte / 500KByte = 262. Maximum TPS(transactions per second) for a single consumer to invoke a single service provider: 7MByte / 500KByte = 14. If acceptable, consider using it, otherwise the network will become a bottleneck.
Why asynchronous single long connections?
Because the current situation of services is that there are few service providers, usually only a few machines, but there are many consumers of the service, so the whole website may access the service. For example, Morgan’s provider has only 6 providers, but there are hundreds of consumers, and there are 150 million calls every day. If the regular Hessian service is adopted, Service providers can easily be overwhelmed by single connections, ensuring that a single consumer does not overwhelm the provider, long connections, reducing connection handshake validation, etc., and using asynchronous IO, multiplexing thread pools, preventing C10K problems.
Because we don’t have a large number of consumers or providers, it is likely that there is a bottleneck in network traffic due to insufficient connections. Below we examine the Dubbo protocol in detail and some experiments to verify our guess.
Ii. Detailed explanation of Dubbo communication process
We are using an older version of Dubbo, 2.5.x, which uses Netty version 3.2.5. The latest version of Dubbo has some modifications to the threading model. Our analysis below uses 2.5.10 as an example.
Figure and part of the code to illustrate the Dubbo protocol call process, code only write some key parts, using netty3, Dubbo thread pool no queue, synchronous call, the following code contains Dubbo and Netty code.
The entire Dubbo call process is as follows:
An analysis of Dubbo congestion
1. Ask to join the queue
We call an RPC service through Dubbo, and the calling thread encapsulates the request into a queue. This queue is a Netty queue. The queue is defined as follows: it is an Linked queue with no limit of length.
class NioWorker implements Runnable {
...
private final Queue<Runnable> writeTaskQueue = newLinkedTransferQueue<Runnable>(); . }Copy the codeThe main thread through a series of calls, eventually, through the method of NioClientSocketPipelineSink class put the request in the queue, the queue of requests, contains a request ID, this ID is very important.
2. Call thread wait
Once enqueued, Netty returns a Future to the calling thread, which then waits on the Future. This Future is defined by Dubbo and is called DefaultFuture. The caller calls defaultFuture. get(timeout) and waits for notification.
public class DubboInvoker<T> extends AbstractInvoker<T> {
...
@Override
protected Result doInvoke(final Invocation invocation) throws Throwable {
...
return (Result) currentClient.request(inv, timeout).get(); Currentclient. request(inv, timeout) returns a DefaultFuture}... }Copy the codeSo if we look at the implementation of DefaultFuture,
public class DefaultFuture implements ResponseFuture {
private static final Map<Long, Channel> CHANNELS = new ConcurrentHashMap<Long, Channel>();
private static final Map<Long, DefaultFuture> FUTURES = new ConcurrentHashMap<Long, DefaultFuture>();
// invoke id.
private final long id; //Dubbo request ID, each consumer is a long type starting from 0
private final Channel channel;
private final Request request;
private final int timeout;
private final Lock lock = new ReentrantLock();
private final Condition done = lock.newCondition();
private final long start = System.currentTimeMillis();
private volatile long sent;
private volatile Response response;
private volatile ResponseCallback callback;
public DefaultFuture(Channel channel, Request request, int timeout) {
this.channel = channel;
this.request = request;
this.id = request.getId();
this.timeout = timeout > 0 ? timeout : channel.getUrl().getPositiveParameter(Constants.TIMEOUT_KEY, Constants.DEFAULT_TIMEOUT);
// put into waiting map.
FUTURES.put(id, this); // Add the Future to the global Future Map with the id as the key
CHANNELS.put(id, channel);
}
Copy the code3. The IO thread reads the data in the queue
This work is done by Netty’s IO thread pool, known as NioWorker, which corresponds to a class called NioWorker. Select (); select (); select ();
public void run() {
for(;;) {... SelectorUtil.select(selector);
proce***egisterTaskQueue();
processWriteTaskQueue(); // Process the write request in the queue first
processSelectedKeys(selector.selectedKeys()); // Reprocess the select event. } } private void processWriteTaskQueue() throws IOException {for (;;) {
final Runnable task = writeTaskQueue.poll();// This queue is the queue into which the calling thread puts the request
if (task == null) {
break;
}
task.run(); / / write datacleanUpCancelledKeys(); }}Copy the code4. The IO thread writes data to the Socket buffer
This step is very important and related to the performance issues we encountered, again NioWorker, task.run() from the previous step, which is implemented as follows:
void writeFromTaskLoop(final NioSocketChannel ch) {
if(! ch.writeSuspended) {// This area is very important, if writeSuspended, then just skip this write
write0(ch);
}
}
private void write0(NioSocketChannel channel) {
......
final int writeSpinCount = channel.getConfig().getWriteSpinCount(); // Netty is a configurable parameter. The default value is 16
synchronized (channel.writeLock) {
channel.inWriteNowLoop = true;
for (;;) {
for (int i = writeSpinCount; i > 0; i --) { // A maximum of 16 attempts per session
localWrittenBytes = buf.transferTo(ch);
if(localWrittenBytes ! =0) {
writtenBytes += localWrittenBytes;
break;
}
if (buf.finished()) {
break; }}if (buf.finished()) {
// Successful write - proceed to the next message.
buf.release();
channel.currentWriteEvent = null;
channel.currentWriteBuffer = null;
evt = null;
buf = null;
future.setSuccess();
} else {
// Not written fully - perhaps the kernel buffer is full.
If the kernel buffer is full, writeSuspended is set to true
addOpWrite = true;
channel.writeSuspended = true; . }...if (open) {
if (addOpWrite) {
setOpWrite(channel);
} else if(removeOpWrite) { clearOpWrite(channel); }}... } fireWriteComplete(channel, writtenBytes); }Copy the codeNormally, write requests in the queue are processed by processWriteTaskQueue, but they are also registered with the Selector, and if processWriteTaskQueue writes successfully, it gets rid of the write requests on the Selector. If the Socket’s write buffer is full, NIO returns immediately, and BIO waits forever. Netty uses NIO, and after 16 attempts, it still can’t write, so it sets writeSuspended to true so that all subsequent write requests are skipped. So when will it be written again? And then the selector, if it finds the socket writable, it writes that data in.
The following is the process written in processSelectedKeys, which sets writeSuspended to false because it finds that the socket is writable.
void writeFromSelectorLoop(final SelectionKey k) {
NioSocketChannel ch = (NioSocketChannel) k.attachment();
ch.writeSuspended = false;
write0(ch);
}
Copy the code5. Data is transferred from the consumer’s socket send buffer to the provider’s receive buffer
This is implemented by the operating system and network adapter. If the write succeeds, it does not mean that the peer end can receive the write. Of course, TCP tries to ensure that the peer end can receive the write through the retransmission mechanism.
6. The SERVER I/O thread reads the request data from the buffer
This is implemented by the NIO thread on the server side in processSelectedKeys.
public void run() {
for(;;) {... SelectorUtil.select(selector);
proce***egisterTaskQueue();
processWriteTaskQueue();
processSelectedKeys(selector.selectedKeys()); // Reprocess the select event. } } private void processSelectedKeys(Set<SelectionKey> selectedKeys) throws IOException {for (Iterator<SelectionKey> i = selectedKeys.iterator(); i.hasNext();) {
SelectionKey k = i.next();
i.remove();
try {
int readyOps = k.readyOps();
if((readyOps & SelectionKey.OP_READ) ! =0 || readyOps == 0) {
if(! read(k)) {// Connection already closed - no need to handle write.
continue; }}if((readyOps & SelectionKey.OP_WRITE) ! =0) {
writeFromSelectorLoop(k);
}
} catch (CancelledKeyException e) {
close(k);
}
if (cleanUpCancelledKeys()) {
break; // break the loop to avoid ConcurrentModificationException
}
}
}
private boolean read(SelectionKey k) {
......
// Fire the event.
fireMessageReceived(channel, buffer); // After reading, this function is eventually called to send an event that receives the message. }Copy the code7. The IO thread passes the request to the Dubbo thread pool
The outbound Handler varies according to the configuration. Set Dispatch to all and outbound Handler as follows: Threadpool exhausted (exhausted) {ExecutorService: Threadpool exhausted (exhausted) {ExecutorService: Threadpool exhausted (exhausted);
public class AllChannelHandler extends WrappedChannelHandler {
......
public void received(Channel channel, Object message) throws RemotingException {
ExecutorService cexecutor = getExecutorService();
try {
cexecutor.execute(new ChannelEventRunnable(channel, handler, ChannelState.RECEIVED, message));
} catch (Throwable t) {
//TODO A temporary solution to the problem that the exception information can not be sent to the opposite end after the thread pool is full. Need a refactoring
//fix The thread pool is full, refuses to call, does not return, and causes the consumer to wait for time out
if(message instanceof Request && t instanceof RejectedExecutionException){
Request request = (Request)message;
if(request.isTwoWay()){
String msg = "Server side(" + url.getIp() + "," + url.getPort() + ") threadpool is exhausted ,detail msg:" + t.getMessage();
Response response = new Response(request.getId(), request.getVersion());
response.setStatus(Response.SERVER_THREADPOOL_EXHAUSTED_ERROR);
response.setErrorMessage(msg);
channel.send(response);
return;
}
}
throw new ExecutionException(message, channel, getClass() + " error when process received event .", t); }}... }Copy the code8. The Dubbo thread pool on the server processes the request and queues the returned packet
The thread pool invokes the following function
public class HeaderExchangeHandler implements ChannelHandlerDelegate {
......
Response handleRequest(ExchangeChannel channel, Request req) throws RemotingException {
Response res = newResponse(req.getId(), req.getVersion()); .// find handler by message class.
Object msg = req.getData();
try {
// handle data.
Object result = handler.reply(channel, msg); // Real business logic class
res.setStatus(Response.OK);
res.setResult(result);
} catch (Throwable e) {
res.setStatus(Response.SERVICE_ERROR);
res.setErrorMessage(StringUtils.toString(e));
}
return res;
}
public void received(Channel channel, Object message) throws RemotingException {
......
if (message instanceof Request) {
// handle request.
Request request = (Request) message;
if (request.isTwoWay()) {
Response response = handleRequest(exchangeChannel, request); // Process the business logic and get a Response
channel.send(response); / / write the response back}}... }Copy the codeChannel. The send (response) finally call NioServerSocketPipelineSink method put back a message in the queue.
9. The server I/O thread retrieves data from the queue
Same as flow 3
10. The server I/O thread writes the reply data to the Socket send buffer
When the IO thread writes data to the TCP buffer, it succeeds. But if the buffer is full, it won’t fit. The results are different for blocking and non-blocking IO, where blocking IO waits forever, while non-blocking IO fails immediately, leaving the caller to choose a policy.
The Netty policy is to write data for a maximum of 16 times. If the write operation fails, the SYSTEM stops the WRITE operation of the I/O thread and waits until the connection becomes writable. The default writeSpinCount value is 16 and you can adjust the parameter.
for (int i = writeSpinCount; i > 0; i --) {
localWrittenBytes = buf.transferTo(ch);
if(localWrittenBytes ! =0) {
writtenBytes += localWrittenBytes;
break;
}
if (buf.finished()) {
break; }}if (buf.finished()) {
// Successful write - proceed to the next message.
buf.release();
channel.currentWriteEvent = null;
channel.currentWriteBuffer = null;
evt = null;
buf = null;
future.setSuccess();
} else {
// Not written fully - perhaps the kernel buffer is full.
addOpWrite = true;
channel.writeSuspended = true;
Copy the code11. Data transmission
The transmission of data over the network depends on the bandwidth and network environment.
The client IO thread reads data from the buffer
This process is the same as flow 6
13. The IO thread delivers data to the Dubbo business thread pool
This step is the same as in process 7, where the thread pool name is DubboClientHandler.
14. The business thread pool notifies the main thread based on the message ID
We get Response from the received function of HeaderExchangeHandler, and then we call handleResponse,
public class HeaderExchangeHandler implements ChannelHandlerDelegate {
static void handleResponse(Channel channel, Response response) throws RemotingException {
if(response ! = null && ! response.isHeartbeat()) { DefaultFuture.received(channel, response); } } public void received(Channel channel, Object message) throws RemotingException { ......if(message instanceof Response) { handleResponse(channel, (Response) message); }... }Copy the codeDefaultFuture retrieves the Future based on the ID, notifying the calling thread
public static void received(Channel channel, Response response) {
......
DefaultFuture future = FUTURES.remove(response.getId());
if(future ! = null) { future.doReceived(response); }... }Copy the codeAt this point, the main thread retrieves the returned data and the call ends.
Key parameters affecting the above process
Protocol parameters
When we use Dubbo, we need to configure the protocol on the server side, for example
<dubbo:protocol name="dubbo" port="20880" dispatcher="all" threadpool="fixed" threads="2000" />
Copy the codeThe following are some performance related parameters in the protocol. In our usage scenario, fixed is selected for the thread pool, the size is 500, the queue is 0, and the rest are the default values.
| attribute | Corresponding URL parameter | type | If required | The default value | role | describe |
|---|---|---|---|---|---|---|
| name | string | mandatory | dubbo | Performance tuning | The name of the protocol | |
| threadpool | threadpool | string | optional | fixed | Performance tuning | The type of the thread pool can be fixed or cached. |
| threads | threads | int | optional | 200 | Performance tuning | Service thread pool size (fixed size) |
| queues | queues | int | optional | 0 | Performance tuning | When the thread pool is full, it is recommended not to set the size of the queue waiting for execution. When the thread pool is full, it should fail immediately and retry other service provider machines instead of queuing unless there is a special requirement. |
| iothreads | iothreads | int | optional | The number of CPU + 1 | Performance tuning | IO thread pool size (fixed size) |
| accepts | accepts | int | optional | 0 | Performance tuning | This is the maximum number of connections that can be established on the entire server. For example, if 2000 connections have been established, new connections will be rejected to protect the service provider. |
| dispatcher | dispatcher | string | optional | The default value of dubbo is all | Performance tuning | Protocol message dispatch mode, used to specify the thread model, such as all, Direct, message, execution, connection of the Dubbo protocol. This mainly involves the division of labor between the IO thread pool and the business thread pool. In general, leaving the business thread pool to handle connection establishment, heartbeat, and so on does not have much impact. |
| payload | payload | int | optional | 8388608(=8M) | Performance tuning | Request and response packet size limits, in bytes. This is the maximum length allowed for a single packet. Dubbo is not suitable for long packets, so it is limited. |
| buffer | buffer | int | optional | 8192 | Performance tuning | Size of network read/write buffer. Note that this is not the TCP Buffer, this is the application layer Buffer for reading and writing network packets. |
| codec | codec | string | optional | dubbo | Performance tuning | Protocol encoding mode |
| serialization | serialization | string | optional | The default value is Hessian2 for Dubbo, Java for RMI, and JSON for HTTP | Performance tuning | Agreement serialization way, when the protocol support multiple serialization methods used, such as: dubbo agreement dubbo, hessian2, Java, compactedjava, and HTTP protocol json |
| transporter | transporter | string | optional | The default dubbo protocol is NetTY | Performance tuning | The types of server and client implementations of protocols, such as Mina and Netty for Dubbo, can be split into server and client configurations |
| server | server | string | optional | By default, the dubbo protocol is NetTY, and the HTTP protocol is servlet | Performance tuning | Server side implementation types of protocols, such as mina and Netty for Dubbo, Jetty and Servlet for HTTP, etc |
| client | client | string | optional | The default dubbo protocol is NetTY | Performance tuning | Client implementation type of protocol, e.g. Mina, Netty, dubbo, etc |
| charset | charset | string | optional | UTF-8 | Performance tuning | Serialization encoding |
| heartbeat | heartbeat | int | optional | 0 | Performance tuning | Heartbeat interval: For a long connection, when the physical layer is disconnected, for example, when a network cable is removed, the TCP FIN message cannot be sent and the peer party cannot receive the disconnect event. In this case, the heartbeat is required to help check whether the connection is disconnected |
Service parameters
For each Dubbo service, there is a configuration. The full parameter configuration is here: dubbo.apache.org/zh-cn/docs/…
Let’s focus on a few performance-related ones. In our scenario, the number of retries is set to 0, failfast is used in cluster mode, and the rest is the default.
| attribute | Corresponding URL parameter | type | If required | The default value | role | describe | compatibility |
|---|---|---|---|---|---|---|---|
| delay | delay | int | optional | 0 | Performance tuning | Delay service registration time (ms). If this parameter is set to -1, it indicates that the service is delayed until the Spring container initialization is complete | 1.0.14 or later |
| timeout | timeout | int | optional | 1000 | Performance tuning | Remote service invocation timeout in milliseconds | Version 2.0.0 or later |
| retries | retries | int | optional | 2 | Performance tuning | Set this parameter to 0. Number of remote service invocation retries excluding the first invocation | Version 2.0.0 or later |
| connections | connections | int | optional | 1 | Performance tuning | For the maximum number of connections per provider, short connection protocols such as RMI, HTTP, and Hessian represent the number of limited connections, while dubbo represents the number of established long connections | Version 2.0.0 or later |
| loadbalance | loadbalance | string | optional | random | Performance tuning | Load balancing strategy, optional value: the random, roundrobin, leastactive, respectively: random, polling, the least active call | Version 2.0.0 or later |
| async | async | boolean | optional | false | Performance tuning | Whether to execute asynchronously by default, not reliably asynchronously, only ignoring the return value and not blocking the thread of execution | Version 2.0.0 or later |
| weight | weight | int | optional | Performance tuning | Service weight | 2.0.5 Or later | |
| executes | executes | int | optional | 0 | Performance tuning | The maximum number of requests that a service provider can execute in parallel per service per method | 2.0.5 Or later |
| proxy | proxy | string | optional | javassist | Performance tuning | Generate dynamic proxy mode, optional: JDK/Javassist | 2.0.5 Or later |
| cluster | cluster | string | optional | failover | Performance tuning | Cluster, optional: failover/failfast/failsafe/failback/forking | 2.0.5 Or later |
The main reason for the congestion is that the service’s connections Settings are too small. Dubbo does not provide a global connection count configuration, but only a personalized connection count configuration for a particular transaction.
Experiment on the effect of connection number and Socket buffer on performance
Verify the impact of connection count and buffer size on transport performance with a simple Dubbo service.
You can adjust the size of the TCP buffer by modifying the system parameters.
Tcp_rmem is the send buffer and tcp_wmem is the receive buffer. The three values indicate the minimum value, default value and maximum value. We can set them to the same.
net.ipv4.tcp_rmem = 4096 873800 16777216
net.ipv4.tcp_wmem = 4096 873800 16777216
Copy the codeThen run sysctl -p for this to take effect.
The server code is as follows: accept a packet, then return twice the packet length, sleep 0-300ms randomly, so the mean should be 150ms. The server prints the TPS and response time every 10 seconds, where THE TPS is the time to complete the function call, not the transfer, and the response time is also the time for the function.
// Server implementation
public String sayHello(String name) {
counter.getAndIncrement();
long start = System.currentTimeMillis();
try {
Thread.sleep(rand.nextInt(300));
} catch (InterruptedException e) {
}
String result = "Hello " + name + name + ", response form provider: " + RpcContext.getContext().getLocalAddress();
long end = System.currentTimeMillis();
timer.getAndAdd(end-start);
return result;
}
Copy the codeThe client starts N threads, and each thread continuously invokes the Dubbo service. The QPS and response time are printed every 10 seconds, including the network transmission time.
for(int i = 0; i < N; i ++) {
threads[i] = new Thread(new Runnable() {
@Override
public void run() {
while(true) { Long start = System.currentTimeMillis(); String hello = service.sayHello(z); Long end = System.currentTimeMillis(); totalTime.getAndAdd(end-start); counter.getAndIncrement(); }}}); threads[i].start(); }Copy the codeYou can run the ss-it command to view details about the current TCP socket, including send-q, CWND, and RTT.
(base) niuxinli@ubuntu:~$ ss -it
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 36 192.1681.7.:ssh 192.1681.4.:58931
cubic wscale:8.2 rto:236 rtt:33.837/8.625 ato:40 mss:1460 pmtu:1500 rcvmss:1460 advmss:1460 cwnd:10 bytes_acked:559805 bytes_received:54694 segs_out:2754 segs_in:2971 data_segs_out:2299 data_segs_in:1398 send 3.5Mbps pacing_rate 6.9Mbps delivery_rate 44.8Mbps busy:36820ms unacked:1 rcv_rtt:513649 rcv_space:16130 rcv_ssthresh:14924 minrtt:0.112
ESTAB 0 0 192.1681.7.:36666 192.1681.7.:2181
cubic wscale:7.7 rto:204 rtt:0.273/0.04 ato:40 mss:33344 pmtu:65535 rcvmss:536 advmss:65483 cwnd:10 bytes_acked:2781 bytes_received:3941 segs_out:332 segs_in:170 data_segs_out:165 data_segs_in:165 send 9771.1Mbps lastsnd:4960 lastrcv:4960 lastack:4960 pacing_rate 19497.6Mbps delivery_rate 7621.5Mbps app_limited busy:60ms rcv_space:65535 rcv_ssthresh:66607 minrtt:0.035
ESTAB 0 27474 192.1681.7.:20880 192.1681.. 5:60760
cubic wscale:7.7 rto:204 rtt:1.277/0.239 ato:40 mss:1448 pmtu:1500 rcvmss:1448 advmss:1448 cwnd:625 ssthresh:20 bytes_acked:96432644704 bytes_received:49286576300 segs_out:68505947 segs_in:36666870 data_segs_out:67058676 data_segs_in:35833689 send 5669.5Mbps pacing_rate 6801.4Mbps delivery_rate 627.4Mbps app_limited busy:1340536ms rwnd_limited:400372ms(29.9%) sndbuf_limited:433724ms(32.4%) unacked:70 retrans:0/5 rcv_rtt:1.308 rcv_space:336692 rcv_ssthresh:2095692 notsent:6638 minrtt:0.097
Copy the codeYou can also run netstat-nat to view information about the current TCP socket, such as recv-q and send-q.
(base) niuxinli@ubuntu:~$ netstat -nat
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0. 0. 0:20880 0.0. 0. 0:* LISTEN
tcp 0 36 192.1681.7.:22 192.1681.4.:58931 ESTABLISHED
tcp 0 0 192.1681.7.:36666 192.1681.7.:2181 ESTABLISHED
tcp 0 65160 192.1681.7.:20880 192.1681.. 5:60760 ESTABLISHED
Copy the codeThe meanings of recV-q and send-q are as follows:
Recv-Q
Established: The count of bytes not copied by the user program connected to this socket.
Send-Q
Established: The count of bytes not acknowledged by the remote host.
Copy the codeRecv-q is the data that has reached the receive buffer, but has not yet been read by the application code. Send-q indicates the data that has reached the sending buffer but has not received an Ack reply. These two kinds of data normally do not accumulate. If they accumulate, there may be a problem.
Experiment 1: single connection, changing TCP buffer
Results:
| role | The Socket buffer | The response time | |
|---|---|---|---|
| The service side | 32k/32k | 150ms | |
| Client (800 concurrent) | 32k/32k | 430ms | |
| Client (1 concurrent) | 32k/32k | 150ms |
Continue to increase the buffer size
| role | The Socket buffer | The response time | CPU |
|---|---|---|---|
| The service side | 64k/64k | 150ms | user 2%, sys 9% |
| Client (800 concurrent) | 64k/64k | 275ms | user 4%, sys 13% |
| Client (1 concurrent) | 64k/64k | 150ms | user 4%, sys 13% |
The second column below is the size of send-q, which indicates the data written to the buffer that has not been confirmed by the peer end. The maximum value of the Send buffer is about 64K, indicating that the buffer is insufficient.
An analysis of Dubbo congestion
By increasing the buffer further to 4M, we can see that the response time drops further, but we still lose a lot of time in transmission because there is no pressure on the server application layer.
| role | The Socket buffer | The response time | CPU |
|---|---|---|---|
| The service side | 4M/4M | 150ms | user 4%, sys 10% |
| Client (800 concurrent) | 4M/4M | 210ms | user 10%, sys 12% |
| Client (1 concurrent) | 4M/4M | 150ms | user 10%, sys 12% |
The TCP buffer on both the server and client is not full
An analysis of Dubbo congestion
The service side
The client
At this point, no matter how large the TCP buffer is, it does not matter, because the bottleneck is not there, but the number of connections. Because in Dubbo, a connection is bound to a NioWorker thread, and all reads and writes are completed by this connection, and the transmission speed exceeds the read and write capacity of a single thread. Therefore, we can see that on the client side, a large amount of data is squeezed in the receiving buffer and not read away, so the transmission rate of the peer side will also slow down.
Second experiment: multiple connections, fixed buffer
The response time of pure business functions on the server side is very stable. When the buffer is small, increasing the number of connections can reduce the time, but it cannot reach the optimum, so the buffer cannot be set too small. The default value of Linux is 4M, at 4M, 4 connections can basically minimize the response time.
| role | The Socket buffer | The response time |
|---|---|---|
| The service side | 32k/32k | 150ms |
| Client (800 concurrent,1 connection) | 32k/32k | 430ms |
| Client (800 concurrent,2 connections) | 32k/32k | 205ms |
| Client (800 concurrent,4 connections) | 32k/32k | 160ms |
| Client (800 concurrent,6 connections) | 32k/32k | 156ms |
| Client (800 concurrent,8 connections) | 32k/32k | 156ms |
| Client (800 concurrent,2 connections) | 1M/1M | 156ms |
| Client (800 concurrent,2 connections) | 4M/4M | 156ms |
| Client (800 concurrent,4 connections) | 4M/4M | 151ms |
| Client (800 concurrent,6 connections) | 4M/4M | 151ms |
conclusion
To make full use of network bandwidth, ensure that the buffer is not too small. If the buffer is too small, more packets may be transmitted at a time than the buffer, which seriously affects the transmission efficiency. Too many connections are needed to make full use of CPU resources, at least more than the number of CPU cores.
From: blog.51cto.com/nxlhero/251…
