Nginx implements real-time log reporting in pure configuration

Nginx is commonly used as a load balancing gateway. A large number of logs are generated. However, because Nginx configuration file is a declarative programming paradigm, which is not convenient to describe flow control, log reporting cannot be realized by simple instructions.

Nginx log reporting usually requires writing a shell script or other language script to periodically parse Nginx log files and report them.

Using NJS module, real-time log reporting can be realized.

However, due to the limitations of instructions supported by NJS module, log reporting cannot be achieved by a single instruction. Non-obstructive real-time log reporting can be implemented by combining multiple commands.

This scheme is implemented in Nginx and does not rely on Node, Python and other processes

Implementation approach

There are many Nginx directives, and the following is one that has recently been explored. If there is a more elegant implementation method, welcome to leave a message.

Although the powerful Njs module can write JS scripts, the instructions of Njs module have many limitations and can not realize arbitrary functions like Node.

To implement real-time log reporting, the following two capabilities must be met:

1. Each request can be triggered
2. Background report, does not block the current request processing process

The usual Js_set directive, although it can be fired on every request, only supports synchronous operations. Unable to use fetch, subrequest method.

The fetch function can be used in the js_content directive. But it can only be used in location. So you can use other directives to forward requests to js_Content’s path, where the reporting of the log is done.

The auth_REQUEST directive in the HTTP_auth_request_module is used to verify the request permissions, such as JWT. This directive fires on each request, creating a child request that determines the result of the permission verification based on the return of the request.

Therefore, the two modules can be combined to achieve log reporting.

Implementation steps

1. Compile Nginx

This requires Nginx to support the ngx_HTTP_js_module and ngX_HTTP_AUTH_request_module modules. These two modules are not installed by default. You need to compile your own implementation

• NJS module installation can be referenced when JS encounters Nginx
• The http_auth_request_module simply adds parameters at compile time--with-http_auth_request_moduleCan be

compile

/configure --add-module=[NJS module path]/NJS/nginx --with-http_auth_request_module make && make installCopy the code

2. The configuration file is as follows

http {
    js_import  http.js;   # Import js files
   
    server {
        listen 80;
    
        auth_request /proxy_report; This directive fires at the start of each request, creating a subrequest to forward to the proxy_report path
        location / { 
            index  index.html index.htm;
        }
        
        location /proxy_report {
            internal; # limit accepting internal requests only
            Save the uri and method data of the original request in the header. Because the auth_REQUEST request modifies this data.
            proxy_set_header X-Original-URI $request_uri;
            proxy_set_header X-Original-METHOD $request_method;
            # forward to another server
            proxy_pass http://localhost:8080/report;
        }
    }
    
    server {
        listen 8080;
        # The reporting interface is placed in another server, and there is no auth_request directive in this server, so as to avoid cyclic triggering requests
        location /report {
            # Through the js_content directive to introduce a JS processing script, complete the reporting operationjs_content http.report; }}}Copy the code

/ / HTTP js file

import qs from "querystring";

async function report (r) {
    let args = {
        // Get the original URI and method data from the header
        uri: r.headersIn['X-Original-URI'].method: r.headersIn['X-Original-METHOD'].remoteAddress: r.remoteAddress,
        status: r.status,
        headersIn: JSON.stringifry(r.headersIn),
    }
    // Send an asynchronous request without blocking the process of the current request
    ngx.fetch('http://[service path]?${qs.stringify(args)}`, {
        method: 'GET',})// Return the status code of 200 to make the verification instruction successful
    r.return(200)}export default { report }
Copy the code