Online alarm

A number of current limiting alerts were made during the event last weekend. So please operation and maintenance god to add machines first, temporarily withstand the pressure. Expansion wave added some machines. And then you see more interface response timeouts.

2019-06-22 23:32:07,957 WARN [New I/O server worker #1-9] com.alibaba.dubbo.common.threadpool.support.AbortPolicyWithReport:warn:54 [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-172.***:62075, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 771 (completed: 571), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://172.***:62075! , dubbo version: 2.5.3, current host: 172.16.6.3

Thread pool is EXHAUSTED!

Troubleshoot problems

what ? The thread pool is exhausted.

1. Does the client call retry count too many times?

Check that all requests are set to 0 retries.

2. Is the response timeout period of the interface too long?

Single request timeout duration 3s.

3. Is the thread blocked?

Open the Arthas.

thread -b

thread

Find that all SimpleAsyncTaskExecutor are blocked. So what are they doing?

episode

After o&M restarts the service, the SimpleAsyncTaskExecutor thread becomes blocked. (This is all blocked)

thread 378

thread 146

conclusion

Capacity expansion ignores database links. The problem of setting a whitelist is fixed.

thinking

Dubbo’s Dispatcher strategy is not connected to Dubbo’s dispatcher strategy. Change the Dispatcher policy of Dubbo Provider to message

<dubbo:protocol name= "dubbo" port= "8888" threads= "500" dispatcher= "message" />

The dispatcher defaults to all, and all requests are sent to the business thread pool. The default dubbo business thread pool size is 200, wait queue length is 0, and subsequent requests are discarded when the thread pool is full. However, discarded requests are also sent to the business thread pool for processing, at which point the server may reject the request but the consumer may wait until the timeout occurs

reference

1. blog.csdn.net/lg772ef/art… 2. www.cnblogs.com/zhukunrong/…