The scene simulation

One day, the application online after the restart the application, after running for a period of time, all through the Apache HttpClient sends a request, will be submitted to the org. Apache. HTTP. Conn. ConnectionPoolTimeoutException: Error waiting for connection from pool. A large number of subsequent applications have similar status. The application release does not involve the change of the Apache HttpClient usage mode. After the application version rollback, the problem still exists.

In order to locate the problem, the r & D students began to read the release history and code changes of the project, but no abnormality was found. Attempts at manometry were not repeated in environments other than on line.

The scene processing

Principles of on-site treatment:Solve the problem first and analyze the cause afterwards.

Repetition POC

The complete POC code, which runs directly in IDEA, is available in Resources

Let’s start with a question: why did we replace the Entity object in this case, and what was the original requirement? Due to business needs, we need to read the returned content in advance for logging, but the original BasicHttpEntity does not support multiple reads through entity.toString, so we replace this with BufferedHttpEntity for repeatable reads.

If we were using some deep-wrapped bytecode injection framework, we might write code like this

// Bytecode injection code
/ / injection: org. Apache. HTTP. Message. BasicHttpResponse# getEntity
public Object overwrite (EnhanceInstance enhanceInstance, Method method, Object[] allArguments, Class
       [] argumentsTypes, OverwriteMethod originMethod) throws Throwable {
    HttpEntity httpEntity = (HttpEntity) TracerReflectionUtils
        .getFieldValueByClassAndFiledName(BasicHttpResponse.class,
                                          "entity",
                                          enhanceInstance);
    if (null! = httpEntity && ! httpEntity.isRepeatable() &&null == enhanceInstance.getDynamicField()) {
        ((BasicHttpResponse) enhanceInstance)
        	.setEntity(new BufferedHttpEntity(httpEntity));
        enhanceInstance.setDynamicField(true);
        return httpEntity;
    }
    return originMethod.call(allArguments);
}
Copy the code

Note that lines 4, 9, and 12, if we implement using Javassist, are equivalent to

StringBuilder sb = new StringBuilder();
sb.append("org.apache.http.entity.BufferedHttpEntity newEntity = new org.apache.http.entity.BufferedHttpEntity(this.entity);");
sb.append("org.apache.http.HttpEntity originalEntity = this.entity;");
sb.append("this.entity = newEntity;");
sb.append("return originalEntity;");
method.insertBefore(sb.toString());
Copy the code

Problem orientation

Start with HttpClient.execute and trace all the way to MainClientExec (which handles connections and communications) with the following key code.

// file => org/apache/http/impl/execchain/MainClientExec.java
// line => 333
// check for entity, release connection if possible
// Response is an instance of BasicHttpResponse, which is the object of the bytecode change
final HttpEntity entity = response.getEntity(); 
if (entity == null| |! entity.isStreaming()) {// connection not needed and (assumed to be) in re-usable state
    // If entity is not stream data, release the current connection directly
    connHolder.releaseConnection();
    return new HttpResponseProxy(response, null);
}

// Monitor whether the data flow reaches the EOF and determine whether to release the current connection
return new HttpResponseProxy(response, connHolder);
Copy the code

Trace into ResponseEntityProxy

// file => org/apache/http/impl/execchain/ResponseEntityProxy.java

// line => 53
public HttpResponseProxy(final HttpResponse original, final ConnectionHolder connHolder) {
    this.original = original;
    this.connHolder = connHolder;
    // Bind data flow and connection, we continue to debug trace in
    ResponseEntityProxy.enchance(original, connHolder);
}
Copy the code

Continue tracing into ResponseEntityProxy

// file => org/apache/http/impl/execchain/ResponseEntityProxy.java

// line => 50
public static void enchance(final HttpResponse response, final ConnectionHolder connHolder) {
    final HttpEntity entity = response.getEntity();
    // Note that the ResponseEntityProxy listener is bound only when entity is a Stream
    if(entity ! =null&& entity.isStreaming() && connHolder ! =null) {
        response.setEntity(newResponseEntityProxy(entity, connHolder)); }}Copy the code

ResponseEntityProxyIs emitted by listening on the StreameofDetected, the use ofconnHolder.releaseConnectionTo release the connection.

Let’s go back to the beginning of our getEntity injection code and see how the code interacts with each other.

StringBuilder sb = new StringBuilder();
// The BufferedHttpEntity here is not a stream, i.e. isStreaming = false
sb.append("org.apache.http.entity.BufferedHttpEntity newEntity = new org.apache.http.entity.BufferedHttpEntity(this.entity);");
// The original entity is BasciHttpEntity, and its isStreaming = true
sb.append("org.apache.http.HttpEntity originalEntity = this.entity;");
// The original value sets newEntity
sb.append("this.entity = newEntity;");
// Return the original value
sb.append("return originalEntity;");
method.insertBefore(sb.toString());
Copy the code

The first call to getEntity returns an originalEntity of type BasciHttpEntity, causing the code to release the connection directly to be skipped

// file => org/apache/http/impl/execchain/MainClientExec.java
// line => 333
final HttpEntity entity = response.getEntity(); 
// Return the entity as originalEntity (BasciHttpEntity). The if logic cannot be entered to release the connection directly
if (entity == null| |! entity.isStreaming()) { connHolder.releaseConnection();return new HttpResponseProxy(response, null);
}
Copy the code

The second call to getEntity, which returns the first newEntity of type BufferedHttpEntity, also fails to access the if logic to listen for EOF to release the connection.

// file => org/apache/http/impl/execchain/ResponseEntityProxy.java

// line => 50
public static void enchance(final HttpResponse response, final ConnectionHolder connHolder) {
    final HttpEntity entity = response.getEntity();
    // newEntity (BufferedHttpEntity) is returned. Again, there is no access to this if logic to listen for EOF to release the connection
    if(entity ! =null&& entity.isStreaming() && connHolder ! =null) {
        response.setEntity(newResponseEntityProxy(entity, connHolder)); }}Copy the code

In summary, our request perfectly missed two opportunities to release the connection, resulting in the connection being held for a long time. A Timeout waiting for connection from pool error occurs when the number of unreleased connections reaches the upper limit of the connection pool.

How to avoid potholes

Bytecode injection itself is an accident-prone operation that can be circumvented in the following ways

  • CodeReview, but this has high technical requirements for team members
  • Try to make problems exposed in advance, test, pre-release environment also load the same environment dependencies as online
  • Continuing to learn as a technology and have a source-level understanding of key projects

The resources

  • Apache Httpclient 4.5.13, Github address: github.com/plusmancn/p…
  • HttpClient 4.3 connection pool parameter configuration: www.cnblogs.com/trust-freed…