preface

Nginx is fun to use, but deadly when things go wrong. Although there are many ways to solve the problem this time, I still hope to connect the knowledge with one problem and dig the root cause deeply.

Phenomenon of the problem

The file size is more than 400 MB. After a period of time, the file fails to be uploaded. The following figure shows an error:

Troubleshoot problems

Information collection

There are two errors in this problem, one is a network error, and the other returns 413. Request Entity Too Large With years of experience as a driver, check the nginx configuration and find that the client_max_body_size of the configuration is 300m. Change its configuration to client_MAX_body_size to 2000m. Upload the file again and test. Again, but this time only net::ERROR_CONNECTION_ABORTED.

Again with the experience of the old driver, suspected that the file is too large, caused by the timeout, as to where the timeout, we have to look at the specific situation. I ran it through the reappearance and the entire link log. The overall information I have sorted out is as follows.Net ::ERROR_CONNECTION_ABORTED returns 504. What the hell is going on here?

Problem analysis

The SLB log shows 504, while the NGINx log shows a return code of 499. It can be concluded that the response time on the NGINx side was too long, and SLB actively disconnected the connection.

504: Gateway Timeout Indicates that the Gateway times out

499: Client Closed Request The Client is actively disconnected

Since SLB is not managed by us, we need to confirm proxy_read_timeout with the cloud vendor. After confirmation, its configuration is 60s. This is consistent with our nGINx’s 499 request time of 60 seconds.

Request timeout (proxy_read_timeout) The timeout period can be configured by the user. The default value is 60 seconds. This timeout is for the back end, that is, HTTP requests and responses between SLB and RS. This time is the difference between the two times the data is received from the back end, not the entire receiving time. The timeout occurs in either of the following cases: • The SLB forwards an HTTP request to the back end, but the back end does not reply any HTTP response within 60 seconds. • The SLB forwards the HTTP request to the back end, which replies with part of the HTTP response data, and does not receive the rest of the HTTP response for another 60 seconds. Before the HTTP response is complete, the backend cannot idle for more than 60 seconds. If the backend server does not respond within the timeout period, the load balancer, acting as a proxy, will give up waiting and return the HTTP 504 error code to the client. Example configurations are as follows: proxy_read_timeout 60;

In summary, the problem occurs because the proxy_read_timeout of SLB is 60 seconds. When uploading a large file to SLB, SLB forwards the file to the back-end nginx, but no HTTP response is received after 60 seconds, SLB disconnects the connection. Failed to upload the file.

Net ::ERROR_CONNECTION_ABORTED indicates that during file uploading, the HTTP server breaks the link when it does not read the HTTP request body of the client. The main reason for this error is that the uploaded file is too large, the server cannot continue to read the request, and the link breaks prematurely. The initial suspicion here is that the version of the browser causes the HTTP request timeout time to be different. The user has net::ERROR_CONNECTION_ABORTED because the request timed out before the file was uploaded.

There is still a question, why is the request time in SLB close to 500s?

To continue our analysis based on this case, we need to figure out how the forwarding request is implemented. To get to the bottom of this, let’s take a look at how Nginx handles a request.

How does Nginx handle requests

HTTP request processing is divided into the following phases

  1. Initialize the HTTP Request (read the data from the client and generate an HTTP Request object that contains all the information about the Request)
  2. Processing request header
  3. Processing request body
  4. If so, call the handler associated with the request (URL or Location)
  5. Each phase Handler is called in turn for processing

The specific processIn phase Handler, it usually goes through the following phases:

NGX_HTTP_POST_READ_PHASE: Read the request content phase

NGX_HTTP_SERVER_REWRITE_PHASE: Server request address rewriting phase

NGX_HTTP_FIND_CONFIG_PHASE: configuration lookup phase

NGX_HTTP_REWRITE_PHASE: Location Request address rewriting phase

NGX_HTTP_POST_REWRITE_PHASE: Request address rewrite submission phase

NGX_HTTP_PREACCESS_PHASE: access permission check preparation phase

NGX_HTTP_ACCESS_PHASE: access permission checking phase

NGX_HTTP_POST_ACCESS_PHASE: Access check commit phase

NGX_HTTP_TRY_FILES_PHASE: try_files processing phase

NGX_HTTP_CONTENT_PHASE: Content generation phase

NGX_HTTP_LOG_PHASE: log module processing phase

The NGX_HTTP_CONTENT_PHASE and content generation phase will be handed over to an appropriate content handler, in this case proxy_pass

Handle the request body forwarding process

Nginx handles the request body itself

After the above phase division of the request, let’s focus on how Nginx forwards the request when the file is uploaded. The handler for proxy_pass calls ngx_http_read_client_request_body() to read the request body. ** It is worth noting that these modules will read the full body of the client request before forwarding the data to the back end. ** This explains why the SLB request time is nearly 500s, while the NGINx request time is 60 seconds. The proxy_read_timeout situation occurs because the file must be uploaded to the SLB before the data is forwarded. Another point of knowledge:

Due to memory limitations, requests read by the ngX_http_read_client_request_body () interface may be partially or completely written to a temporary file. Depending on the size of the request body and the configuration of the associated directives, the request body may be placed in one contiguously contiguously spaced memory or in two separate chunks of memory. It might all be in a temporary file, and it might end up with some in memory and the rest in a temporary file.

Ngx_http_proxy_handler uses ngx_http_read_client_request_body, ngx_http_upstream_init as the callback function. When the request body is read, Ngx_http_upstream_init () is executed to forward the request.

static ngx_int_t
ngx_http_proxy_handler(ngx_http_request_t *r)
{
    ...
    rc = ngx_http_read_client_request_body(r, ngx_http_upstream_init);


    if (rc >= NGX_HTTP_SPECIAL_RESPONSE) {
        return rc;
    }

    return NGX_DONE;
}
Copy the code

Upstream handles the request body

When using the upstream module, we can understand what the mechanism of the lower upstream is and thus how the status code is formed across the request link.

  1. Upstream, ngx_HTTP_upstream_init

    • Deleting a Timeout Timer

    • Establish to upstream request

    • Mount handlers that include the end-of-request cleanup function for upstream to be used in Step 6

  2. Create the upstream link, ngx_http_upstream_connect

    • Set up the socket and connetion, initiate the TCP connection request, send the request using epoll, and connect to the upstream handler, including the handler that handles the upstream response in steps 4 and 5

  3. The request sent upstream is ngx_http_upstream_send_request

  4. Process_header parses the request header, and NGX_HTTP_upstream_PROCESS_headers handles the request header

  5. Handle the upstream response body, ngX_HTTP_upstream_process_body_in_memory

  6. End upstream request, ngx_HTTP_upstream_cleanup

Upstream (upstream) during the request phase, the SLB will set a timeout timer in the upstream module. After 60 seconds, the SLB will disconnect and nGINx will also disconnect. So that explains the status codes.

In addition, during this phase, nginx will define three timeout periods by default:

  • Proxy_connect_timeout, 75s by default, sets connection timeout with upstream server.
  • Proxy_send_timeout, default 60s, this specifies the timeout period for sending requests to upstream server. The timeout is set not for the entire send period, but for the two write operations. Upstream (upstream) will close the connection if no new data is received after the timeout
  • Proxy_read_timeout, default 60s, this directive sets the read timeout period with the proxy server. It determines how long Nginx will wait to receive a response to a request. This time is not the time to get the whole response, but the time to two reading operations

The solution

According to the question, I provide the following ideas, you can choose according to their own situation.

  1. Upload the compressed file or improve the local bandwidth. Try to complete the upload within 60s
  2. Configure SLB to increase proxy_read_timeout time (usually cannot be changed)
  3. With the sharding upload approach, this requires code changes on the front and back ends. Reference: www.cnblogs.com/tugenhua070…

The problem summary

After the above analysis, we understand the following points:

  1. Client_max_body_size Specifies the size of the request body that Nginx accepts.
  2. Error description 504 and 499.
  3. Proxy_read_timeout Specifies the timeout period for reading back-end services.
  4. Nginx requests are made up of several phases. The actual processing of the request body occurs in NGX_HTTP_CONTENT_PHASE.
  5. The difference between the request time of SLB and that of NGINx is that SLB obtains the request body and forwards it to the back-end service.

conclusion

The article is bound to have some not rigorous place, but also hope that we contain, we absorb the essence (if any), to its dregs. If you are interested, you can close my public account: Gungunxi. My wechat id is lCOMedy2021

Reference documentation

To accept the principle of package: blog.csdn.net/ApeLife/art…

Configuration parameters: www.cnblogs.com/lemon-flm/p…

Nginx principle: cloud.tencent.com/developer/a…

Tengine.taobao.org/book/chapte…

Source analysis: www.kancloud.cn/digest/unde…