I recently found that the Dashboard page of Grafana’s official website featured a page for analyzing Nginx logs using Loki, which is basically a typical application of LogQL V2 syntax that has been introduced in the products since Loki2.0. Perhaps because of the increasing lack of familiarity with the new syntax, the community has also made a quick demo to illustrate the use of the new syntax.

In this video, we see that the LogQL V2 syntax provides a new way to analyze Nginx logs. The demo can be found in the Dashboard on the official website. But today, I want to write about how ingress-Nginx can also use a video-like interface.

1. Ingress – Nginx log

Logs are the source of Loki. In ingress-nginx, the log format is defined in the nginx-Configuration configMap. Again we need to add two configurations to declare ingress-nginx’s global logging format.

apiVersion: v1
data:
  log-format-escape-json: "true"
  log-format-upstream: '{"timestamp": "$time_iso8601", "requestID": "$req_id", "proxyUpstreamName":
    "$proxy_upstream_name","host": "$host","proxyAlternativeUpstreamName": "$proxy_alternative_upstream_name","upstreamStatus":
    "$upstream_status", "geoip_country_code": "$geoip_country_code","upstreamAddr": "$upstream_addr","request_time":
    "$request_time","httpRequest":{"requestMethod": "$request_method", "requestUrl":
    "$request_uri", "status": $status,"requestSize": "$request_length", "responseSize":
    "$upstream_response_length", "userAgent": "$http_user_agent", "remoteIp": "$remote_addr",
    "referer": "$http_referer", "latency": "$upstream_response_time", "protocol":"$server_protocol"}}'
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  name: nginx-configuration
  namespace: ingress-nginx
Copy the code

Note: Ingress-nginx has the geoIP module enabled by default. The variable $geoIP_COUNTRY_code in the log format prints the actual country code

2. Collect logs

Ingress-nginx in Kubernetes allows you to collect logs using promtail, Fluentd, or any of the other three client tools. Xiao Bai is not introduced here. Or you can refer to my previous article “The Loki and FluentD Thing” to consider how Fluentd uses Kubernetes metadata to define log labels and send them to Loki.

3. Grafana

Install world map panel

Grafana provides a worldmap-panel for data visualization of a worldmap, often used to analyze different values of different regions. In Nginx’s logs, we mainly retrieve country codes to visualize the rough distribution of log requests.

The plug-in installation is simple. Restart the grafana service by executing the following command in the grafana service instance:

grafana-cli plugins install grafana-worldmap-panel
Copy the code

Import the Loki V2 Web Analytics Dashboard

Download it on Grafana’s websiteLoki v2 Web Analytics DashboardJson file and import it into your grafana

Note: The log format in ingress-nginx is quite different from that provided by the official dashboard. The official dashboard is imported here to save time for my own drawing. You can also configure ingress-nginx in the json format provided by the official log

Adjust LogQL V2 query statements

Edit a Panel and change filename=/var/log/nginx/json_access.log from the original LogQL statement to the ingress-nginx log tag in your Loki.

conclusion

If you are not familiar with the LogQL V2 syntax, it is recommended to read the LogQL documentation before you start. It will help you to understand the essence of LogQL, or you can refer to my previous article “LogQL syntax is greatly enhanced with major 2.0 update for Loki”. In short, LogQL queries give Loki systems a lot of flexibility, but it’s not the community’s intention to use Loki logs for measurement.

Pay attention to the public account “cloud native Xiaobai”, get more exciting content