How to monitor Nginx Web servers using Elastic Stack
The author Kerry Gallagher
In this article, we’ll show you how to monitor NGINX using various components of Elastic Stack. We use MetricBeat and FileBeat to collect data. This data will be sent to Elasticsearch and stored there. Finally, we’ll use Kibana to view the data.
MetricBeat will collect data related to the connection (activity, processed, accepted, and so on), as well as the total number of client requests. FileBeat collects data related to access logs and error logs. The information collected will vary from setting to setting, but in most cases, we can use this information to infer certain things, such as:
- If the number of error logs for a resource spikes, it may mean that we have deleted a resource that is still needed.
- Access to the log allows you to see when a service is at its peak (and thus determine the best time to perform operations such as maintenance).
- A sudden spike in the number of client requests indicates the possibility of a malicious attack (such as a DDoS attack).
The focus of this article is on monitoring, so things like Elasticsearch Settings will be briefly skimmed over. I’m on a Mac, so I’ll be using the Mac distributions of Elasticsearch, MetricBeat, FileBeat and Kibana. Distributions for other platforms can be found on the download page.
Install the Elasticsearch
MetricBeat and FileBeat (actually all Beats) need Elasticsearch clusters to store data. We will set up a simple Elasticsearch cluster locally. Since we don’t publish ElasticSearch to the public, we don’t have to spend time configuring security.
Step 1: Download ElasticSearch
curl -L -O
https://artifacts.elastic.co/…
Step 2: Unzip the archive file
The tar – XVF elasticsearch 7.1.0 – Darwin – x86_64. Tar. Gz
Step 3: Change to the bin directory
CD elasticsearch – 7.1.0 / bin
Step 4: Start our single-node cluster
./elasticsearch
By default, your cluster will run at localhost:9200.
Run Nginx
There are many ways to run Nginx – on a host alone, through a Docker container, or in a Kubernetes setup, and so on. Beats has an automatic discovery feature that listens for container API events to track server containers when they are accelerated and deleted.
Since Nginx can be set up and configured in a number of ways, this article will make no assumptions and show examples of host configuration and automatic discovery (in this case, Docker).
Configure MetricBeat and FileBeat
Now, we’ll configure Beats to start collecting and sending data.
Metricbeat
We will use MetricBeat to gather metrics.
Step 1: Download MetricBeat:
curl -L -O
https://artifacts.elastic.co/…
Step 2: Unzip the archive file
The tar – XVF metricbeat 7.1.0 – Darwin – x86_64. Tar. Gz
Step 3: Change the directory
CD metricbeat – 7.1.0 – Darwin – x86_64
Configure the time!
By default, the configuration file (metricbeat.yml) will look for the Elasticsearch cluster running on localhost:9200 that we configured earlier.
Step 4: Enable the relevant MetricBeat module
By default, only system-level metrics are collected:
./metricbeat modules enable nginx
Step 5: Change file ownership
We need to change the ownership of several files because we will be running MetricBeat as root (or, you can choose to disable strict permission checking on the command line with the –strict. Perms =false option) :
sudo chown root metricbeat.yml
sudo chown root modules.d/system.yml
sudo chown root modules.d/nginx.yml
Step 6: Enable metric sets
If you open the modules.d/nginx.yml file, you can enable the StubStatus metric set, including uncommenting the following:
metricsets: # – stubstatus
This metric set collects data from the NGINX NGX_HTTP_STUB_STATUS module, so be sure to configure this NGINX module for the metric set to work properly.
In addition, you can change the host monitored here, which by default is print . For a host-based setup, this is all the information we need.
Step 7: Automatic discovery Settings (optional)
When using the auto-discovery setting, the equivalent configuration between the three providers (Docker, Kubernetes, and Jolokia) will differ. The Docker example will look like this, where we added MetricBeat. AutoDiscover at the bottom of the metricBeat. YML configuration file.
metricbeat.autodiscover: providers: – type: docker
templates: \- condition: contains: docker.container.image: nginx
config: \- module: nginx
metricsets: \["stubstatus"\] hosts: "${data.host}:${data.port}"
With this configuration, all Docker containers that use images with NGINX in their names will be matched (Contains will perform substring matching, rather than exact matching), and a config will be used to enable the NGINX module with a set of substate metrics.
Step 8: Run MetricBeat
Good, now that we’ve configured it, we can then run Metricbeat with flags, send logs to stderr and disable syslog/ file output:
sudo ./metricbeat -e
If all is well, you should see some initial output, and then more output each time MetricBeat sends data back to the cluster.
Filebeat
We will use FileBeat to collect the logs.
Step 1: Download FileBeat
curl -L -O
https://artifacts.elastic.co/…
Step 2: Unzip the archive file
The tar – XVF filebeat 7.1.0 – Darwin – x86_64. Tar. Gz
Step 3: Change the directory
CD filebeat – 7.1.0 – Darwin – x86_64
Step 4: Enable the NGINX module
./filebeat modules enable nginx
By default, access logs and error logs are collected. As for the log file paths, FileBeat will try to determine these paths depending on your operating system, but if you need to explicitly set different paths, you can set them in the modules.d/nginx.yml configuration file.
Step 5: Change file ownership
sudo chown root filebeat.yml
sudo chown root modules.d/nginx.yml
sudo chown root module/nginx/access/manifest.yml
sudo chown root module/nginx/error/manifest.yml
For a host-based setup, this is all the information we need.
Step 6: Automatic discovery Settings (optional)
As mentioned above, additional configuration is required to use auto-discovery. Like the previous section, the Docker example needs to add the auto-discovery section to the config, and filebeat.yml will look something like this:
filebeat.autodiscover: providers: – type: docker
templates: \- condition: contains: docker.container.image: nginx
config: \- module: nginx
access: input: type: docker
containers.ids: \- "${data.docker.container.id}" error: input: type: docker
containers.ids: \- "${data.docker.container.id}"
Where we configure the access and error options to use Docker input. Docker input will be in its path (Docker log base path, by default is/var/lib/Docker/containers) under the search container log. We specify container. ids using the container ID that matches the template criteria. In short, Log (by default) will come from the/var/lib/docker/containers/ac29b98ad83ca43bb4c15ae8f0d03aff8c7d57bf5dee9024124374b92b14b0f2 / will vary (ID).
Step 7: Run FileBeat
sudo ./filebeat -e
Install Kibana
Now that we have MetricBeat and FileBeat sending data about the NGINX server, we need a way to view the data, and that’s using Kibana.
Step 1: Download Kibana
curl -O
https://artifacts.elastic.co/…
Step 2: Unzip the archive file and change the directory
The tar – XZF kibana 7.1.0 – Darwin – x86_64. Tar. Gz
CD kibana – 7.1.0 – Darwin – x86_64
Step 3: Run Kibana
./bin/kibana
By default, Kibana will use the Elasticsearch host http://localhost:9200, and Kibana can be accessed from at.
If you navigate to http://localhost:5601, you should see the following welcome:
You can choose to try manipulating the sample data, but here we click Explore on My Own.
Visualize NGINX data in Kibana
Next, we’ll show you how to use Kibana to view and analyze data.
The infrastructure
If we navigate to the “Infrastructure” application on the Kibana sidebar, we’ll be able to see a snapshot view of the Infrastructure (at the last moment). Depending on the configuration being used, data based on non-auto-discovery Settings can be accessed under ‘Hosts’. For auto-discovery Settings, the Docker and Kubernetes buttons (based on the provider being used) will direct you to the correct data set:
Because I am using host-based Settings, I can click on the ‘Hosts’ (host) button to view the following data being sent from MetricBeat:
If we select one of the nodes and choose View Metrics, we can see a detailed metric page for that single NGINX server.
The log
If we navigate to the “Logs” application, we will be able to see the access and error Logs sent from FileBeat.
Preconfigured Nginx dashboard
Now that we have a running and accessible instance of Kibana, we can load some prefabricated dashboards.
To load the MetricBeat dashboard, run the following command in the MetricBeat directory:
sudo ./metricbeat setup –dashboards
Execute the same command for FileBeat:
sudo ./filebeat setup –dashboards
Now if we navigate to the “Dashboards” TAB in Kibana, we should see the following:
If we narrow down to the “nginx” search, we see the following dashboards available:
For example, the _[FileBeat Nginx] access log and error log ECS_ dashboards would look like this:
So far, we’ve seen how to use Elastic Stack to monitor NGINX servers. You can use various options (for example, different grouping and filtering options) to gain insight into the information that is really important to you.
Personally try
Getting started with Elastic Stack is easy — you can download the various products using the curl command provided, or you can browse our download page to get them. Even easier, you can sign up for a 14-day free trial of Elasticsearch before you get the product. If you run into any problems, visit the discussion forum for more information.
To learn more about Elastic technology, please follow and sign up for the webinar. The upcoming schedule is as follows:
Wednesday, February 19, 2020 15:00-16:00 Building Omni-Observable Instances Using Elastic Stack
Wednesday, February 26, 2016 15:00-16:00 Kibana Lens Webinar
Wednesday, March 4, 2020 15:00-16:00 Elastic Endpoint Security Overview Network
Monitor website resources with Elastic Stack Wednesday, March 11, 2020 15:00-16:00