In 2018, the Kangaroo Cloud technology team decided to open source the project on the technical community to contribute to the open source community.
In the same year, FlinkX and FlinkStreamSQL were launched successively. After three years of development, the Star numbers reached 2.2K and 1.5K respectively, ranking among the best in Flink ecological field and gaining extensive community support.
For more than three years, kangaroo Cloud stack technology team has been taking technology as the core, adhering to the concept of open source, digital technology dividend back to the community, and constantly share the results for everyone.
At ArchSummit2021 Global Architect Summit in April this year, the digital stack technology team shared the “Practice and Thinking of Cloud native Big Data System Architecture” for everyone, and promised to open source the practice of digital stack in cloud native successively.
What is EasyAgent
EasyAgent is a system designed for remote unified management of the life cycle of host Agent. Since its launch in 2017, it has been running smoothly in financial, government and enterprise production environments for more than tens of millions of hours.
In a large-scale log collection system, the EasyAgent can centrally manage 10,000 log collectors (such as elastic FileBeat) to implement collector installation, upgrade, start, stop management, and configuration update.
In the number stack operation and maintenance manager EasyManager, through EasyAgent can realize the number stack service installation, upgrade, uninstall, start and stop and configuration management. EasyAgent is the key component of operation and maintenance manager of data stack and the basic component to realize automatic deployment of data stack.
Ii. What problems does EasyAgent solve
If now “hot” Kubernetes is the container scheduling scheduler, then based on EasyAgent can achieve the traditional host process (service) scheduling. EasyAgent, like Daemonset deployed on each host, controls the life cycle of managed processes (services) in the way of Sidecar.
In Linux, automatic startup and manual startup and stop management of services (processes) can be realized through Service and Systemctl, and the open source library Supervisor can also realize daemon and manual startup and stop of services (processes), but neither of them can support remote scheduling and control of services (processes).
For system-level services (which can be called into the OS image), open source solutions such as system native and Supervisor can basically meet the requirements. However, when dynamic scheduling (installation, uninstallation, migration), start, stop, and configuration change of service processes are required, the system native and Supervisor solutions cannot meet the requirements.
EasyAgent is produced in such a demand background, it gets through a link between the server and the network host, to achieve the target host service (process) choreography and life cycle management.
In the enterprise software system delivery industry, the first step of software delivery is deployment. How to quickly realize the automatic deployment, operation and maintenance of self-developed commercial software often represents the delivery efficiency and delivery cost of a ToB enterprise.
Mature commercial companies are choosing to develop their own automated deployment and operation tools. EasyAgent is the basic component to realize the native deployment of data stack cloud, which solves the automation problems of deployment and monitoring.
Three, EasyAgent how to solve the above problems
EasyAgent is a bridge between the target host and the upper application. Next, we explain how to solve the above problems from the perspective of the upper application and the target host.
1, from the upper application perspective
EasyAgent’s original intention is to provide a solution for remote unified management of host service (process) life cycle, so we first abstract the life cycle of service (process).
Referring to Service and Supervisor, we abstract the life cycle of service (process) into seven processes: query, installation, configuration, start, stop, execution, and uninstallation. We encapsulate seven APIS for upper-layer business systems to call. Upper-layer businesses organize and call corresponding APIS to meet their needs according to business requirements. Such as automated deployment platform business perspective.
The Sidecar running on the host side is developed by Golang language and can be compatible with hosts of multiple OS and Arch. In order to reduce the requirement of network policy, EasyAgent adopts Push mode, and the host end registers with the server side one-way.
The sidecAR deployment process on the host is the host access process. After the host access, the SIDecAR establishes TCP with the server and responds to the control commands sent from the server in real time. The SIDecar communicates with the server through gRPC. There are three communication interfaces. For details, see:
Github.com/DTStack/eas…
#syntax = “proto3”;
service EasyAgentService {
rpc registerSidecar (RegisterRequest) returns (RegisterResponse);
rpc readyForControl (ControlRequest) returns (stream ControlResponse);
rpc reportEvent (Event) returns (EmptyResponse);
Copy the code}
Upper-layer services installed by Install exist as subprocesses of sidecar on the target host. Sidecar guards and controls the start and stop of the service processes.
Similarly, upper-layer businesses can remotely view and update the configuration of services through the Exec and Config interfaces. Service status and performance data collected on the Sidecar end are synchronized to the server end through reportEvent. The server end can synchronize service status and performance data to third-party storage, such as Elasticsearch and InfluxDB. This enables dynamic observation of the state of the service. Four, how to experience EasyAgent
EasyAgent Open Source project address:
Github.com/DTStack/eas…
If you want to better understand the EasyAgent project, go to learn Quick Star (tutorial), three steps to understand how to use EasyAgent:
Github.com/DTStack/eas…
If you have any fantastic ideas for extending EasyAgent, please head over to the Github community to discuss them via Issue.
Stack is a cloud native – site-based data central-platform PaaS, and we have an interesting open source project on Github and Gitee: FlinkX is a batch data synchronization tool based on Flink. It can collect static data and real-time data. It is a data synchronization engine with all-domain, heterogeneous and batch integration. If you like, please give us a star! Star! Star!
Github open Source project: github.com/DTStack/fli…
Gitee Open Source project: gitee.com/dtstack_dev…