If you’ve installed the latest Elastic Stack, you’ve probably noticed that Beats is not the recommended way to ingest data, but the Elastic Agent. We can take a look at the picture below:

The figure above shows the Elastic Stack as a whole. One of the most important changes we’ll find is the use of Integrations instead of Beats and Logstash, although they still apply. The recommended way is to use Integration to ingest data.

In today’s article, we’ll show you how to use Fleet to set up Elastic Agents and ingest the required data. You will understand:

Configuration Fleet
Use the Elastic Agent to send data to the Elastic Stack
Monitor logs and metrics for systems and services across the company

In today’s exercise, I’ll use Elastic Stack 8.0 to demonstrate this. The test environment I used was as follows:

I installed Elastic Stack 8.0 on macOS and Fleet Server and Elastic Agent on Ubuntu OS.

The installation

Elasticsearch and Kibana must be installed before we proceed to the following exercises. We can refer to the previous article:

Elastic Stack 8.0 Installation – Securing your Elastic Stack is now easier than ever
Elastic: Use Docker to install Elastic Stack 8.0

Install Elasticsearch and Kibana as required above. In order for Fleet to work properly, the built-in API service must be started. Elasticsearch config/ elasticSearch.yml

xpack.security.authc.api_key.enabled: true
Copy the code

Once configured, we can restart Elasticsearch. We also need to do an additional configuration for Kibana. We need to modify the config/kibana.yml file. At the very end of the file, add the following line:

 2.  xpack.encryptedSavedObjects.encryptionKey: 'fhjskloppd678ehkdfdlliverpoolfcr'
Copy the code

If you don’t want to use the above Settings, you can use the following method:

./bin/kibana-encryption-keys generate
Copy the code

From the output above, we can see that there are three output keys. We can copy all three at the same time and add them to the end of config/kibana.yml. Of course, we could just copy one of them. Let’s restart Kibana.

This completes Elasticsearch and Kibana configuration. For installations of Elastic Stack prior to 8.0, read my previous article “Observability: How to Use Fleet to Ingest System logs and Metrics in the latest Elastic Stack.”

In addition, Kibana requires an Internet connection to download integration packages from the Elastic Package Registry. Make sure the Kibana server can connect to port 443 on epr.Elastice. co. If your environment has network traffic limitations, there are ways to address this requirement. See Air Gap Environment for more information.

Currently, Fleet can only be used by users with superuser roles.

Configuration Fleet

Use Fleet in Kibana to import logs, metrics, and security data into the Elastic Stack. The first time you use Fleet, you may need to set it up and add Fleet Server. Before doing the configuration, let’s first check to see if any integration is installed:

Let’s open the Fleet page:

Since we haven’t specified the Fleet Server’s address yet, we click Edit Hosts:

In our configuration, we want to install Fleet Server on an Ubuntu OS machine, so in the input box above, we enter the Ubuntu OS address 192.168.0.4. Port address 8220 is the default address. Click Save and Apply Settings above:

It is also correct to check the address of Elasticsearch above.

Click on the Agents TAB:

For our self-managed non, there is a Default Fleet Server Policy. We can also create our own Policy.

What exactly is a Fleet Server Policy?

It is the Agent Policy that contains the Fleet Server integration. In a self-managed cluster, the default Fleet Server policy is automatically created when Fleet is run for the first time. If you want to deploy a self-managed Fleet Server to use our hosted Elasticsearch service, you must explicitly create a new agent policy that includes the Fleet Server integration.

Since our Ubuntu OS is Linux 64-bit, we chose the option above to download it. We enter the following command in Ubuntu terminal:

The curl - O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.0.0-linux-x86_64.tar.gzCopy the code

1. liuxg@liuxgu:~/fleet$ curl -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.0.0-linux-x86_64.tar.gz 2. % % of Total Received  % Xferd Average Speed Time Time Time Current 3. Dload Upload Total Spent Left Speed 4. 100 256M 100 256M 0 0 9775k 0 0:00:26 0:00:26 --:--:-- 9257kCopy the code

We use the following command to install:

The tar XZF elastic - agent - 8.0.0 - Linux - x86_64. Tar. GzCopy the code

Click on Generate Service Token above:

Waiting for Fleet Server to connect Waiting for Fleet Server to connect . We click the two document buttons above to copy and save the Service token. We run the above command on an Ubuntu machine:

Let’s go back to the Kibana interface:

Fleet Server Connected is displayed. It clearly shows that our Fleet has been successfully connected. We click Continue above:

It shows that our Agents have been successfully deployed and its state is Healthy.

If you have multiple Fleet Servers, it is recommended that you create a unique service token for each Fleet Server. To generate Service tokens, see elasticSearch – Service-Tokens. For more information about Fleet Server, see the link.

Above, we have successfully created an Agent for the Default Fleet Server Policy. It has Fleet Server installed. This is a necessary step; a Fleet Server can manage many Agents. For more information on the architecture of Fleet, see the article “Observability: Using Fleet to Easily import Nginx logs and metrics.”

Add Elastic Agent to Fleet

Elastic Agent is a single unified Agent that you can deploy to hosts or containers to collect data and send it to the Elastic Stack. In the background, an Elastic Agent runs Beats Shippers or an Elastic Endpoint for your configuration.

Send logs and metrics to the Elastic Stack. You’ll need to download Elastic Agent on that machine and unzip it:

macOS

1. The curl - L - O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.0.0-darwin-x86_64.tar.gz 2. Tar XZVF elastic - agent - 8.0.0 - Darwin - x86_64. Tar. GzCopy the code

Linux OS

1. The curl - L - O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.0.0-linux-x86_64.tar.gz 2. Tar XZVF elastic - agent - 8.0.0 - Linux - x86_64. Tar. GzCopy the code

Windows

1. # PowerShell 5.0+ 2. wget https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.0.0-windows-x86_64.zip - OutFile Elastic - agent - 8.0.0 - Windows - x86_64. Zip 3. Expand - Archive. \ elastic - agent - 8.0.0 - Windows - x86_64. ZipCopy the code

1. The curl - L - O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.0.0-amd64.deb 2. Sudo DPKG -i Elastic - agent - 8.0.0 - amd64. DebCopy the code

1. The curl - L - O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.0.0-x86_64.rpm 2. Sudo RPM - vi Elastic - agent - 8.0.0 - x86_64. RPMCopy the code

For more information about downloading Elastic Agent, see the official Download Elastic Agent link.

In our case, what we need on the phone is the log and metrics on Ubuntu OS, so we downloaded the Linux version. Fleet Server was already unzipped and running when we installed it earlier, so we didn’t need to do anything special.

Next we add Agent:

Above, we choose the Default Policy, which currently has an integration called System. You can also create your own policy. Choosing the Default policy will make our exercise progress faster. We downloaded the corresponding Agent according to the Go to Download page. This is the same page as we used for Fleet Server. I don’t want to talk about it here.

We click the document button above to register:

Sudo. / elastic - agent install - url = HTTP: / / http://192.168.0.4:8220 --enrollment-token=amw2a0FIOEJnUDdoZ29vRmJBVkY6TDRCZ3liNnpTSFNZdENnLXFrOExSZw==Copy the code

Please note that http://192.168.0.4:8220 is the address of our Fleet Server, which means Agent registration is sent directly to the Fleet Server. Of course, if you are on any other platform, you will need to choose from the above page.

Run the command above:

When we installed Fleet Server, the Elastic Agent was already installed, so the above error message appeared. If you have other Linux machines, you can register your Elastic Agent using this method.

Now, I’ve got one of my Raspberry PI Ubuntu OS machines up and running, and we use the following method to register:

1. liuxg@liuxgarm:~/fleet$ curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.0.0-linux-arm64.tar.gz 2. % % of Total Received % Xferd Average Speed Time Time Time Current 3. Dload Upload Total Spent Left Speed 4. 100 241M 100 241M 0 0 5615k 0 0:00:44 0:00:44 --:--:-- 5953K 5. liuxg@liuxgarm:~/fleet$ls 6. elastic-agent-8.0.0-linux-arm64.tar.gz 7. liuxg@liuxgarm:~/fleet$tar XZF elastic-agent-8.0.0-linux-arm64.tar.gz 8. liuxg@liuxgarm:~/fleet$CD liuxg@liuxgarm:~/fleet/elastic/agent-8.0.0-linux-arm64$sudo./elastic-agent install - the url = http://192.168.0.4:8220 - enrollment - token = amw2a0FIOEJnUDdoZ29vRmJBVkY6TDRCZ3liNnpTSFNZdENnLXFrOExSZw = = 10. [sudo] password for liuxg: 11. Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:Y 12. Error: Error: connection to fleet-server is insecure, strongly recommended to use a secure connection (override with --insecure) 13. For help, Both please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.0/fleet-troubleshooting.html 14. The Error: enroll command failed with exit code: 1 15. For help, Both please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.0/fleet-troubleshooting.html 16. liuxg@liuxgarm:~/fleet/elastic-agent-8.0.0-linux-arm64$sudo./elastic-agent install --url=http://192.168.0.4:8220 --enrollment-token=amw2a0FIOEJnUDdoZ29vRmJBVkY6TDRCZ3liNnpTSFNZdENnLXFrOExSZw== --insecure 17. Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:Y 18. {" log level ":" warn ", "@ timestamp" : "the T16 2022-02-16: all. 428 + 0800", "the logger" : "the TLS", "the origin" : {" file. The name ":" tlscommon / Tls_config. Go, "" file. The line" : 105}, "message" : "SSL/TLS verifications disabled.", "ecs. Version 1.6.0" : ""} 19. {" log level ", "info", "@ timestamp" : "the T16 2022-02-16: all. 544 + 0800", "the origin" : {" file. The name ":" CMD/enroll_cmd. Go ", "the file. The line ":456},"message":"Starting enrollment to URL: http://192.168.0.4:8220/ ", "ecs version 1.6.0" : ""} 20. {" log level ", "info", "@ timestamp" : "the T16 2022-02-16:" 501 + 0800 ", "the origin" : {" file. The name ":" CMD/enroll_cmd. Go ", "the file. The line ":256},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"} 21. Successfully enrolled the Elastic Agent. 22. Elastic Agent has been successfully installed.Copy the code

As shown above, we got an error because we were trying to register a Fleet Server with a self-signed certificate. I added an option –insecure to the command above:

Sudo. / elastic - agent install - url = HTTP: / / http://192.168.0.4:8220 --enrollment-token=amw2a0FIOEJnUDdoZ29vRmJBVkY6TDRCZ3liNnpTSFNZdENnLXFrOExSZw== --insecureCopy the code

So our run was successful. Back to the Kibana interface:

At this point, we see a new addition called Liuxgarm agent. Of the two agents above, Fleet Server runs on liuxgu machines. If we want to collect logs and metrics for this machine, we need to add System Integration to the Default Fleet Server Policy.