With the release of Elastic 7.8, we’re pleased to announce a major improvement that will make it easier to send data to the Elastic Stack. First, we’ll launch an experimental version of our Elastic Agent, a unified Agent that simplifies installation and administration.
Second, we’ll launch Ingest Manager, a new application in Kibana that lets you quickly add integration of popular services and platforms with a few clicks. It will also help you centralize your entire Elastic Agent team.
This experimental version will showcase our vision of the future and allow you to try out our new solutions. There is no migration path for future releases, so you must test in a dedicated cluster.
The challenges of large scale data ingestion
A key factor in choosing a monitoring solution is how easily you can extract data from applications and platforms. Spending time setting up and managing solutions adds to your costs.
Beats is the current solution for transferring data to the Elastic Stack, and they make a lot of things easier than general-purpose solutions like syslog. However, when users first get used to it, they must install modules on the command line, edit long YAML files, copy passwords or configure and use keystores. We want to make getting started as simple as running a command. If you know how to install and run the previous Beats, please refer to my previous article “Beats: Getting Started with Beats (PART 1) (Part 2)”. For more Beats articles, see Beats link.
As user adoption of Elastic Stack increased, they found that we now have seven Beats in total, in addition to APM proxy, APM server, endpoint security, etc. Installing so many binaries can be complex, especially in an enterprise environment, where each binary involves installation through a configuration management solution, progressive deployment, change management, and audit requirements.
Another challenge is the process of adding integrations for new data sources. Today, users must edit YAML files and upload them to all servers. They often use tools like Ansible or Chef to roll out configurations. Unfortunately, this makes adding new data sources a complex process that often requires third-party tools and coordination between teams. This is even more complicated when dealing with thousands of agents spread across multiple networks and data centers.
Import Elastic Agent and Ingest Manager
Elastic Agent is a unified way to add monitoring of logs, metrics, and other types of data to all hosts. You no longer need to install multiple Beats and other agents. This will make deployment across the infrastructure easier and faster. In addition, Elastic Agent has a single unified configuration. Therefore, there is no need to edit multiple configuration files for Filebeat, Metricbeat, etc. This will make it easier to add integration for new data sources.
Ingest Manager provides a Web-based Kibana UI to add and manage integration of popular services and platforms. This release supports nine integrations, and we plan to add support for over 100 Beats modules in the next few releases. Our integration not only provides an easy way to add new data sources, but also provides off-the-shelf assets such as dashboards, visualizations, and pipes to extract structured fields from the logs. You don’t have to spend a lot of effort configuring the system because it will automatically do the usual services for you. This makes it much easier to gain insight in seconds.
Composition of Ingest Manager
- Integrations
- Configurations
- Fleet
- Data Streams
- Enroll New Agent
Simplify the configuration
With our configuration editor user interface, integration can now be configured more easily. Instead of long YAML files with lots of irrelevant Settings, we now offer a convenient Web-based interface that is more concise and provides guidance and validation.
In the screenshot below, the user is invited to select the broker configuration that has selected the default Settings. The proxy configuration can be applied to multiple agents. This makes it easier to manage configurations on a large scale.
Next, the user defines his data source by providing a name and description. They can then configure access and error log paths. When users are done, they can save the data source. This adds NGINX monitoring to all agents registered with the default proxy configuration. These agents will receive updates the next time they log in. It’s much easier to deploy these configurations automatically than having to do it yourself using SSH, Ansible Playbook, etc.
Power users sometimes like YAML files and apis. Ingest Manager has an API-first design, so anything you can do in the interface can also be done using the API. This makes it easy to automate and integrate with other systems.
Centrally manage your Fleet
It’s Fleet. You can view the status of all elastic agents on the Fleet page. Here you can see which agents are online, which are wrong, and when they were last checked in. You can also view the version of the agent binaries and configurations.
Fleet acts as a communication channel back to Elastic Agents. The agent will check regularly for the latest updates. Any number of agents can be registered in each agent configuration, which allows you to scale up to thousands of hosts. When you change the agent configuration, all agents will receive the update the next time you check in. You no longer need to use SSH, Ansible Playbooks, or other configuration methods to distribute configuration updates yourself.
Data flow makes index management easier
Data collected by Elastic Agents is stored in an index that is more granular than the default obtained with Filebeat. The advantage is that it allows users to better understand the source of data volumes and control lifecycle management policies and indexing permissions. We are calling these new indexes “Data Streams,” and we will refine this concept more in future releases.
In the screenshot below, you can see that we have divided the data flow (or index) by data set, type, and namespace. The dataset is defined by the integration and describes the fields and other Settings for each index. For example, you might have a process metric data set with one field describing whether the process is running. Another data set for the disk I/O metric will have a field that describes the number of bytes read. This solves the problem of indexes with hundreds or thousands of fields, because we only need to store a few fields in each index. This makes them more compact and has faster auto-complete, plus the Discover page will show only the relevant fields.
Namespaces are user-defined strings that let you group your data in any way you like. For example, you can group data by environment (product, quality inspection) or team name. This makes it easier to search for data from a given source using an index schema, or to give users access to the data by assigning an index schema to user roles. Many of our customers already organize indexes in this way, and now we provide best practices by default.
Beats and the future of Beats centralised management
Beats isn’t going away, and users can continue to use them with Elastic Agent. In fact, Elastic Agent runs Beats in the background. Elastic Agent is a lightweight interface at the top that simplifies deployment and centralizes management.
Beats Central Management is a Beta product that we released a few years ago for Central Management. As we learned more about our customers’ use cases, we decided to redesign the system for the new Ingest Manager. Beats Central Management is deprecated and still valid, but we don’t officially support it.
Ingest Manager will replace Beats Central Management and make it easier to manage many agents on a large scale. This is an experimental release, so we recommend that you wait until Ingest Manager is generally available (GA) before using it in production.
Limitations of this release
You must test in dedicated clusters. In future releases, we plan to add support for new ways to manage scrolling indexes that will make the user experience easier. However, all data stored in this release will not be migrated in our next release and you will have to erase all data and Settings changed in Kibana and Elasticsearch to avoid future conflicts. We recommend that you use dedicated test clusters or deployments that can be removed when you are done.
Currently, Ingest Manager is only available to users with superuser roles. This role is required to create indexes, install integration assets, and update broker configurations. In order to use Fleet, the Elastic Agent must connect directly to Kibana. You can also run Elastic Agents in standalone mode if an Internet connection is not available or not required.
In addition, this release only includes support for nine integrations, with more support to come in future releases:
- System logs and metrics
- Custom logs
- AWS
- Nginx
- Redis
- Mysql
- Kafka
- Cisco devices
- Netflow logs
You can read more about this restriction in our documentation. The experimental version is not officially supported, but we encourage you to report problems in our forums.
Give it a try
You can try the new Elastic Agent and Ingest Manager yourself. You can download Elastic Agent from our Download page. The easiest way to get started is to create a new cloud deployment to test. Since this is an experimental version, it can only be used in a dedicated test environment and can be removed when completed.
Next, you must enable Ingest Manger by opening a tag. This step is temporary only for the current release, and by default Ingest Manager will be enabled in future releases. On the “Create Deployment” page, enter the following flag as a user setting alternative to enable Ingest Manager.
xpack.ingestManager.enabled: true
Copy the codeIt can be difficult to find where to configure, so see the screen capture below. At the bottom of the Kibana configuration box.
Local clustering requires some additional steps (such as enabling security), so please refer to the documentation on how to enable Ingest Manager in this case.
The first time you open this application, it will ask you to enable Fleet for central administration. On the Overview page, click the Enroll a New Agent button. After the pop-up window is displayed, follow the instructions to register and run Elastic Agent. Learn more in our Ingest Manager documentation.
We’re making the current version of Elastic Agent and Ingest Manager available for free. We want to encourage collaboration with the community so that you can find the Elastic Agent and Ingest Manager code on GitHub.
Read more
This is our first article on Ingest Manager. We will detail how to start Ingest Management in a local deployment in the next second article, “Observability: Simplifying data import with Elastic Agent and Ingest Manager (PART 2).”
If you want to learn more about how Elastic Agent and Ingest Manager work, please read my colleague’s post “Unlocking Elastic’s latest data acquisition module – Ingest Manager and Elastic Agent.”