MaxLifetime parameters
-
The maxLifetime parameter must be smaller than the database time_wait, which defaults to 1800000, or 30 minutes. If the value is set to 0, the lifetime is infinite. If it is not 0 and less than 30 seconds, it is reset back to 30 minutes. The HikariConfig class has a verification rule for this parameter.
-
HikariPool class, when we initialize the connection pool, its construction method, instantiate the enclosing POOL_ENTRY_CREATOR = new HikariPool. PoolEntryCreator (); This class implements the Callable interface and is used to initialize the connection.
public Boolean call(a) throws Exception {
for(long sleepBackoff = 250L; HikariPool.this.poolState == 0 && HikariPool.this.totalConnections.get() < HikariPool.this.config.getMaximumPoolSize(); sleepBackoff = Math.min(TimeUnit.SECONDS.toMillis(10L), Math.min(HikariPool.this.connectionTimeout, (long) ((double)sleepBackoff * 1.5D)))) {
PoolEntry poolEntry = HikariPool.this.createPoolEntry();
if(poolEntry ! =null) {
HikariPool.this.totalConnections.incrementAndGet();
HikariPool.this.connectionBag.add(poolEntry);
return Boolean.TRUE;
}
UtilityElf.quietlySleep(sleepBackoff);
}
return Boolean.FALSE;
}
Copy the codeCall createPoolEntry() to generate a connection.
private PoolEntry createPoolEntry(a) {
try {
final PoolEntry poolEntry = this.newPoolEntry();
long maxLifetime = this.config.getMaxLifetime();
if (maxLifetime > 0L) {
long variance = maxLifetime > 10000L ? ThreadLocalRandom.current().nextLong(maxLifetime / 40L) : 0L;
long lifetime = maxLifetime - variance;
poolEntry.setFutureEol(this.houseKeepingExecutorService.schedule(new Runnable() {
public void run(a) {
HikariPool.this.softEvictConnection(poolEntry, "(connection has passed maxLifetime)".false);
}
}, lifetime, TimeUnit.MILLISECONDS));
}
this.LOGGER.debug("{} - Added connection {}".this.poolName, poolEntry.connection);
return poolEntry;
} catch (Exception var8) {
if (this.poolState == 0) {
this.LOGGER.debug("{} - Cannot acquire connection from data source".this.poolName, var8);
}
return null; }}Copy the codeIn this method, a delay task is set. The specific delay execution time is calculated according to maxLifetime. The difference between the trigger time and maxLifetime is calculated according to maxLifetime > 10_000? ThreadLocalRandom.current().nextLong( maxLifetime / 40 ) : 0; To calculate (up to 2.5% of the MaxLifetime) that triggers EVIT before the connection lifetime is about to reach maxLifetime to prevent a large number of connections from failing at the same time due to maxLifetime.
When triggered, evICT will be marked as true. Evict simply means that the connection in the connection pool is not available, but it is still in the connection pool and can still be borrowed. Only when getConnection is checked, if isMarkedEvicted, the connection will be removed from the connection pool. And then close.
-
In HikariCP, physical connections are closed through a separate thread pool, closeConnectionExecutor. Automatic connection closure is triggered when three conditions occur:
-
Disconnection;
-
Connection lifetime exceeded maxLifeTime
-
Connection idleTimeout of idleTimeout
After closeConnectionExecutor closes the connection, the fillPool() method is called to fill the connection pool
validationTimeout
- ValidationTimeout Specifies the timeout period for verifying connection validity (
The default value is 5 seconds and the minimum value is 250 milliseconds),HikariPool.getConnectionMethod is calledisConnectionAlive(Connection connection)Verify the connection.
For jDBC4, use isUseJdbc4Validation, which validates the connection directly with Connection.isValid (validationSeconds). Otherwise, the connectionTestQuery query statement is used for query validation.
leakDetectionThreshold`
-
This parameter is used to enable connection leak detection. When obtaining a connection through getConnection(), Another createProxyConnection() method is called to get the connection, where we focus on this. Leaktask.schedule (poolEntry).
public final Connection getConnection(long hardTimeout) throws SQLException { this.suspendResumeLock.acquire(); long startTime = clockSource.currentTime(); try { long timeout = hardTimeout; do { PoolEntry poolEntry = (PoolEntry)this.connectionBag.borrow(timeout, TimeUnit.MILLISECONDS); if (poolEntry == null) { break; } long now = clockSource.currentTime(); if(! poolEntry.isMarkedEvicted() && (clockSource.elapsedMillis(poolEntry.lastAccessed, now) <=this.ALIVE_BYPASS_WINDOW_MS || this.isConnectionAlive(poolEntry.connection))) { this.metricsTracker.recordBorrowStats(poolEntry, startTime); // Get the connection Connection var10 = poolEntry.createProxyConnection(this.leakTask.schedule(poolEntry), now); return var10; } Copy the codeThe Schedule method returns a ProxyLeakTask object
/ / return ProxyLeakTask ProxyLeakTask schedule(PoolEntry bagEntry) { return this.leakDetectionThreshold == 0L ? NO_LEAK : new ProxyLeakTask(this, bagEntry); } Copy the codeCheck whether the leakDetectionThreshold parameter is 0. The default value is 0 and detection is not enabled. Otherwise, a delayed execution task will start at exactly the set leakDetectionThreshold, which is used to throw the Apparent Connection Leak detected exception.
private ProxyLeakTask(ProxyLeakTask parent, PoolEntry poolEntry) { this.exception = new Exception("Apparent connection leak detected"); this.connectionName = poolEntry.connection.toString(); // Delay execution this.scheduledFuture = parent.executorService.schedule(this, parent.leakDetectionThreshold, TimeUnit.MILLISECONDS); } Copy the codeIntercept some exceptions as follows
22:14:49.096 volte-cmd-service-test [HikariPool-1 housekeeper] WARN com.zaxxer.hikari.pool.ProxyLeakTask - Connection leak detection triggered for com.mysql.jdbc.JDBC4Connection@429fe922, stack trace follows java.lang.Exception: Apparent connection leak detected at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProvid erImpl.java:122) at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:386) at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl .java:87) at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.jav a:112) at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:489) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:215) at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:200) at org.springframework.orm.jpa.vendor.HibernateJpaDialect$HibernateConnectionHandle.doGetConnection(HibernateJpaDialect.jav a:414) at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:177) Copy the codeIf the leakDetectionThreshold is exceeded, the above exception will be thrown.
HouseKeeper
It is an internal class in HikariPool that implements the Runnable interface and mainly manages connections. When HikariPool is initialized, a scheduleWithFixedDelay task is created. (Fixed delay time is fixed between two tasks, but the execution duration of each task may be variable. The difference with scheduleFixedRate is that: No matter whether the task is completed, the next task will be executed at the specified time.) The default execution time is 30 seconds to refresh the configuration.
public HikariPool(HikariConfig config) {
super(config);
this.ALIVE_BYPASS_WINDOW_MS = Long.getLong("com.zaxxer.hikari.aliveBypassWindowMs", TimeUnit.MILLISECONDS.toMillis(500L));
this.HOUSEKEEPING_PERIOD_MS = Long.getLong("com.zaxxer.hikari.housekeeping.periodMs", TimeUnit.SECONDS.toMillis(30L));
this.POOL_ENTRY_CREATOR = new HikariPool.PoolEntryCreator();
this.connectionBag = new ConcurrentBag(this);
this.totalConnections = new AtomicInteger();
this.suspendResumeLock = config.isAllowPoolSuspension() ? new SuspendResumeLock() : SuspendResumeLock.FAUX_LOCK;
this.checkFailFast();
if(config.getMetricsTrackerFactory() ! =null) {
this.setMetricsTrackerFactory(config.getMetricsTrackerFactory());
} else {
this.setMetricRegistry(config.getMetricRegistry());
}
this.setHealthCheckRegistry(config.getHealthCheckRegistry());
this.registerMBeans(this);
ThreadFactory threadFactory = config.getThreadFactory();
this.addConnectionExecutor = UtilityElf.createThreadPoolExecutor(config.getMaximumPoolSize(), this.poolName + " connection adder", threadFactory, new DiscardPolicy());
this.closeConnectionExecutor = UtilityElf.createThreadPoolExecutor(config.getMaximumPoolSize(), this.poolName + " connection closer", threadFactory, new CallerRunsPolicy());
// Create a scheduled task class
if (config.getScheduledExecutorService() == null) { ThreadFactory threadFactory = threadFactory ! =null ? threadFactory : new DefaultThreadFactory(this.poolName + " housekeeper".true);
this.houseKeepingExecutorService = new ScheduledThreadPoolExecutor(1, (ThreadFactory)threadFactory, new DiscardPolicy());
// Pass the false argument to this method. After shutdown(), the task to be processed will not be executed.
this.houseKeepingExecutorService.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);
// After canceling the task, determine whether the task needs to be removed from the blocking queue
this.houseKeepingExecutorService.setRemoveOnCancelPolicy(true);
} else {
this.houseKeepingExecutorService = config.getScheduledExecutorService();
}
this.leakTask = new ProxyLeakTask(config.getLeakDetectionThreshold(), this.houseKeepingExecutorService);
// Initialize the HouseKeeper
this.houseKeepingExecutorService.scheduleWithFixedDelay(new HikariPool.HouseKeeper(), 100L.this.HOUSEKEEPING_PERIOD_MS, TimeUnit.MILLISECONDS);
}
Copy the codeTime callback processing
In the run method of HouseKeeper, time will be judged first.
This is mainly through a time difference to determine whether there is a callback within the time difference return, initialization will generate a timestamp through the following constructor
// Previous = current time -30s(default scheduled task interval)
private HouseKeeper(a) {
this.previous = HikariPool.clockSource.plusMillis(HikariPool.clockSource.currentTime(), -HikariPool.this.HOUSEKEEPING_PERIOD_MS);
}
Copy the codeIf the task is executed 30 seconds after the first initialization or 30 seconds after the last task, if the current timestamp +128ms is less than the previous timestamp +30 seconds, the callback is available
// Detect the retrograde time, according to the NTP specification allows +128ms
if (HikariPool.clockSource.plusMillis(now, 128L) < HikariPool.clockSource.plusMillis(this.previous, HikariPool.this.HOUSEKEEPING_PERIOD_MS)) {
HikariPool.this.LOGGER.warn("{} - Retrograde clock change detected (housekeeper delta={}), soft-evicting connections from pool.", HikariPool.this.poolName, HikariPool.clockSource.elapsedDisplayString(this.previous, now));
this.previous = now;
HikariPool.this.softEvictConnections();
HikariPool.this.fillPool();
return;
}
Copy the codeAt this point, the log is printed and previous is reset to the current time, the connection is set to unavailable, and the connection is regenerated.
Keep connections to a Minimum
-
If the time is correct, idleTimeout is judged. If idleTimeout is greater than 0, the current idle connection is retrieved
-
Check whether minimumIdle is greater than minimumIdle. If it is greater than minimumIdle, sort the current idle connections based on lastemail exchange
-
If the idle time for each connection taken out has exceeded idleTimeout and the connection is successfully changed from NOT_IN_USE(idle) to RESERVED(marked RESERVED)
-
The connection is closed
-
Finally, create a new connection
public void run(a) { try { // Refresh connectionTimeout and validationTimeout HikariPool.this.connectionTimeout = HikariPool.this.config.getConnectionTimeout(); HikariPool.this.validationTimeout = HikariPool.this.config.getValidationTimeout(); HikariPool.this.leakTask.updateLeakDetectionThreshold(HikariPool.this.config.getLeakDetectionThreshold()); long idleTimeout = HikariPool.this.config.getIdleTimeout(); long now = HikariPool.clockSource.currentTime(); // Determine the clock callback if (HikariPool.clockSource.plusMillis(now, 128L) < HikariPool.clockSource.plusMillis(this.previous, HikariPool.this.HOUSEKEEPING_PERIOD_MS)) { HikariPool.this.LOGGER.warn("{} - Retrograde clock change detected (housekeeper delta={}), soft-evicting connections from pool.", HikariPool.this.poolName, HikariPool.clockSource.elapsedDisplayString(this.previous, now)); this.previous = now; HikariPool.this.softEvictConnections(); HikariPool.this.fillPool(); return; } if (now > HikariPool.clockSource.plusMillis(this.previous, 3L * HikariPool.this.HOUSEKEEPING_PERIOD_MS / 2L)) { HikariPool.this.LOGGER.warn("{} - Thread starvation or clock leap detected (housekeeper delta={}).", HikariPool.this.poolName, HikariPool.clockSource.elapsedDisplayString(this.previous, now)); } this.previous = now; String afterPrefix = "Pool "; if (idleTimeout > 0L) { IN_USE(1: in use), NOT_IN_USE(0: idle), REMOVED(-1: REMOVED), RESERVED(-1: marked as RESERVED) List<PoolEntry> idleList = HikariPool.this.connectionBag.values(0); int removable = idleList.size() - HikariPool.this.config.getMinimumIdle(); if (removable > 0) { HikariPool.this.logPoolState("Before cleanup "); afterPrefix = "After cleanup "; / / sorting idleList.sort(PoolEntry.LASTACCESS_COMPARABLE); Iterator var8 = idleList.iterator(); while(var8.hasNext()) { PoolEntry poolEntry = (PoolEntry)var8.next(); //idleTimeout The connection status is changed if (HikariPool.clockSource.elapsedMillis(poolEntry.lastAccessed, now) > idleTimeout && HikariPool.this.connectionBag.reserve(poolEntry)) { HikariPool.this.closeConnection(poolEntry, "(connection has passed idleTimeout)"); --removable; if (removable == 0) { break;//keep min idle cons } } } } } HikariPool.this.logPoolState(afterPrefix); HikariPool.this.fillPool(); } catch (Exception var10) { HikariPool.this.LOGGER.error("Unexpected exception in housekeeping task", var10); }}}Copy the code
The problem
Inconsistent minimumIdle
MinimumIdle
/ / HikariCP - 3.4.5. Jar
private synchronized void fillPool(a)
{
final int connectionsToAdd = Math.min(config.getMaximumPoolSize() - getTotalConnections(), config.getMinimumIdle() - getIdleConnections())
- addConnectionQueueReadOnlyView.size();
if (connectionsToAdd <= 0) logger.debug("{} - Fill pool skipped, pool is at sufficient level.", poolName);
// The number generated is subtracted by 1
for (int i = 0; i < connectionsToAdd; i++) {
addConnectionExecutor.submit((i < connectionsToAdd - 1)? poolEntryCreator : postFillPoolEntryCreator); }}// hikarICP-2.5.1
for (int i = 0; i < connectionsToAdd; i++) {
addBagItem();
}
Copy the codeTimeout problems
The error log is as follows:
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
Copy the code-
Show variables like ‘%timeout%’ (default: 8 hours)
-
If the configuration is correct, it may not be related to HikariCP. This error occurs when no connection is available and times out when a borrow request is made to the pool. Firstly, at this time, we should consider whether the number of connection pools is reasonable, which is related to the business volume. Second, see if our code has slow SQL; The third point, related to the persistence framework used, is to analyze who holds our connection, what its connection management method is, and under what circumstances the connection will be returned.
This article is because I met the wrong but can’t locate decided to study, I the cause of this error is on the third point above, my project USES is the jpa database interaction, and is a very simple single table query, connection should be returned ChiZhongCai soon, but after my test, when after a database query, Instead of being released, the connection was returned after my entire session had ended.
The core of JPA is hibernate-core. I have queried the connection release strategy of Hibernate on the Internet and know the reason. Hibernate. Connection.release_ mode has the following four properties:
- On_close, the JDBC connection is released only when the Session is explicitly closed or disconnected
- After_transaction, which releases links at the end of each transaction
- After_statement actively releases the connection after each JDBC call
- Auto, select AFTER_statement for JTA and CMT transaction policies, and AFTER_TRANSACTION for JDBC transaction policies
My SpringBoot project version is 1.x, and even when I upgrade hibernate-core to a higher version and start things, I still release connections based on on_close; I tested the 2.x version, which is on_close when things are not turned on, and after_TRANSACTION when things are turned on. It is not clear why the 1.x version does not turn on things.
The resources
-
Flowchart of the way to explain, very clear and easy to understand, the version is relatively high
How does HikariCP manage database connections?