Background:

Xx service interfaces are used to obtain information about the whole country. In real production scenarios, data is frequently invoked and a large amount of data is required. Therefore, you need to perform performance tests on the service interfaces to verify the interface performance and the bearing capacity under high loads.

Interface processing logic: obtain nationwide regional information. Obtain information from mysql for the first time. After obtaining information, hset to Redis.

Question:

20 Continuously pressurizing the interface at the same time, the pressure load is constantly increasing, the press end monitors an error message and returns connection failure (10 concurrent normal).

Application service throws an exception: getList: break: area: list the error

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

Location and investigation:

From the exception information thrown by the application, we can see that more redis thread pool resources cannot be obtained, but 20 concurrent threads have not caused high pressure. Further investigation:

Hardware resources: The server resource usage is normal. The CPU, memory, and disk resources are sufficient. Check whether resources are affected

Redis configuration and Performance: MaxIdle =300 and redis.pool. MaxTotal =600. The maximum number of connections is large enough

Redis connection number is normal, netstat nap | grep redis | wc -l, more than 100 active connection.

Redis-info Displays redis information. The connection is normal. More than 100 connections are normal.

Redis-monitor obtains data properly, and both GET and HGET data are normal.

Check the Redis logs and find the following problems:

WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot or run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect.

Optimization solution:

The Error message in the Reid log indicates that Linux kernel parameters need to be adjusted

Overcommit_memory =1, and then sysctl -p to make the configuration file take effect

Note: Kernel parameter VM. Overcommit_memory indicates the memory allocation policy. The value can be 0, 1, or 2:

Indicates that the kernel will check to see if there is enough memory available for use by the process;

If there is enough memory available, the memory request is allowed.

Otherwise, the memory application fails and the error is returned to the application process.

1, indicates that the kernel is allowed to allocate all physical memory regardless of the current memory state.

2, indicates that the kernel is allowed to allocate more memory than the sum of all physical memory and swap space

Related questions:

MaxWaitMillis = “Could not get a resource from the pool”

As can be seen from the GenericObjectPool source code borrowObject(Long borrowMaxWaitMillis) method

if(p == null) {

if(borrowMaxWaitMillis < 0L) {

p = (PooledObject)this.idleObjects.takeFirst();

} else {

waitTime = System.currentTimeMillis();

p = (PooledObject)this.idleObjects.pollFirst(borrowMaxWaitMillis, TimeUnit.MILLISECONDS);

waitTime = System.currentTimeMillis() – waitTime; // P =null when the waiting time exceeds borrowMaxWaitMillis

}

}

if(p == null) {

throw new NoSuchElementException(“Timeout waiting for idle object”); The thrown exception is caught by the pool class

}

The pool of source code

try {

return this.internalPool.borrowObject();

} catch (Exception var2) {

throw new JedisConnectionException(“Could not get a resource from the pool”, var2);

}

So simply setting the MaxWaitMillis setting to jedis is a little larger to reduce the problem caused by MaxWaitMillis

Setting too large a resource from the pool will create a large cost of hardware performance under high load concurrent, can be measured according to the target concurrent data to get reasonable parameter Settings, and achieve better performance without excessive consumption of server resources

Get jedis and return jedis from jedispool

Set testOnBorrow and testOnReturn to false

When get and return jedis are set to true, jedis will ping redis.

From the GenericObjectPool source code borrowObject(Long borrowMaxWaitMillis) method, we can see:

if(p ! = null && (this.getTestOnBorrow() || create && this.getTestOnCreate())) {

boolean validate = false;

Throwable validationThrowable1 = null;

try {

validate = this.factory.validateObject(p);

Get directly first to verify whether it can be used

} catch (Throwable var13) {

PoolUtils.checkRethrow(var13);

validationThrowable1 = var13;

}

JedisFactory source code validateObject(PooledObjectpooledJedis) method can be seen

BinaryJedis jedis = (BinaryJedis)pooledJedis.getObject();

try {

return jedis.isConnected() && jedis.ping().equals(“PONG”);

} catch (Exception var4) {

return false;

}

Could not get a resource from the pool

Setting testOnBorrow and testOnReturn to false for jedis is 1.4 times faster than true, but this may cause problems that can be further studied.