1. Creation and development of YARN
YARN design roadmap
Yet Another Resource Negotiator in YARN
- Split resource management and job monitoring into separate processes
- Resource management is independent of applications. Each application has a separate job monitoring process
YARN basic components
ResourceManager
Global resource manager, responsible for resource management and allocation of the entire system
- Process client requests
- Start/monitor ApplicationMaster
- Monitor the NodeManager
- Resource allocation and scheduling
NodeManager
Node manager An agent that resides on each node in a YARN cluster
- Manage resources on a single node
- Process commands from ResourceManger
- Process commands from ApplicationMaster
ApplicationMaster
Application manager, responsible for the management of all applications in the system
- Data segmentation
- Apply for resources for the application and allocate them
- Task monitoring and fault tolerance
The relationship between the three
Basic Workflow
- The client request