3.1 Design objectives when the VM is running

When implementing the Nomad-based run time of the edge computing platform, the most important work is to implement a driver, according to the community model, specifically a driver plug-in. The runtime driver plug-in is a server in C/S architecture, which implements the standard interface of the runtime driver plug-in exposed by the community, accepts Nomad as the client to make RPC calls, and handles the corresponding requests in loose coupling. Therefore, based on the definition of the runtime driver interface content, this article lists the following functional design goals:

L Implement a gRPC server that accepts requests for the run time driver interface.

L Obtain plug-in information. The response information contains metadata information for the plug-in.

L Settings configuration. Set up the configuration for the plug-in when it is first started, including the encoded plug-in configuration block and the configuration of the Nomad agent that executes the runtime driver plug-in. The plug-in has enough information to run independently.

L Get the runtime driver’s Schema. Configuration In the plug-in Configuration block of the client Configuration, the Configuration corresponding to this pattern is defined using HashiCorp Configuration Language (HCL).

L Get the configuration mode of the run-time driver task. The configuration for this pattern is defined using HCL.

L Capabilities: Functions implemented by the run time driver, such as whether the task supports the operating system semaphore, whether the command can be executed, and the file system isolation type supported by the run time driver.

L Fingerprint, the health check mechanism of the Nomad runtime driver. Allows the runtime driver to report its node properties and health to the Nomad client periodically until the context terminates.

L VM task life cycle management, which enables the runtime driver interface to manage VM workloads, including starting, recovering, waiting, stopping, and destroying tasks.

N Vm network. In the startup task, the running driver creates and configures the VM network and returns the description of the VM network, including the IP address. The network scheme is determined by the predefined job configuration file.

N VM image: The image path is specified by the job configuration during the start task, or the image is pulled from the downloader.

N Task state storage, leaving the details of the storage to the runtime driver implementation. For example, the task details of the runtime driver are stored in memory, and the stored state can be used to restore the tasks of the runtime driver after the Nomad client restarts.

L Task log management: Store VM task logs in the format defined by the run time driver interface to the path required by the interface. Nomad supports writing to these paths across platforms by creating first-in, first-out queues for standard output and standard error, while also taking care of the scrolling and processing of the task logs.

L Task monitoring data, the runtime driver is also responsible for specific monitoring data collection, Nomad receives statistical data, regardless of how the runtime driver monitoring data is collected.

L Task event notification. The runtime driver emits the event, and the Nomad client polls and listens for the event.

L Tasks respond to operating system signals. Support for sending operating system signals (SIGHUP, SIGKILL, SIGUSR1, etc.) to tasks. A capability of the runtime driver.

L Execute commands in the context of a task. A capability of the runtime driver. However, it is not supported when the VM is running.

3.2 Operating mechanism analysis of Nomad Driver plug-in during Runtime

Now that the requirements and interfaces for managing virtual machine workloads are clear, you need to understand the Nomad runtime driver plug-in mechanism and Libvirt API model before you can implement the Libvirt virtual machine. This part of the work is based on the plug-in framework introduced in v0.9 of the Nomad community, which was briefly introduced in the previous technical overview.

As shown in Figure 3.1, when a job is submitted by the Nomad CLI, the job consists of task groups, and the tasks in the task group run on the same device node. Each task needs to specify the runtime driver to run the workload. The CLI submitted job is PUT to the Endpoint of the Nomad server through the REST interface, and the byte stream is parsed and encapsulated into the corresponding request. The Nomad server load manages a notification bus for job queues, schedulers, and server agents. Nomad implements a finite state machine shared with the Raft algorithm to provide strong consistency.

Figure 3.1 Main modules involved in runtime driver job execution

Before processing the run-time driver job evaluation, the Nomad server agent performs a number of operations, including initializing defaults, adding implicit constraints, and verifying the job. Homework sheets of thread scheduling workers (Worker) processing, each server can run multiple dispatching workers – leader or a Follower (followers), responsible for operation team waiting for evaluation, call the scheduler, submit plans, etc., around the lifecycle of the task assignment, the scheduler connecting with the business logic of the components needed for the facilities, Make it work together. Job evaluation Dispatcher (JobEvalDispatcher) submits a job and creates an evaluation for it, such as PeriodicDispatch for tracking and starting periodic jobs. An EvalBroker is an agent used to manage estimates that are placed into the estimate broker when an estimate is created due to changes to the job specification or node. The agents are sorted by priority and scheduler type, allowing the highest-priority jobs to be taken off the queue first, while allowing the sub-schedulers to take only the jobs they know how to handle. The evaluation agent is designed to work entirely in memory and is managed by the server lead node. It relies on explicit Ack/Nack messages to guarantee at least one delivery semantics. BlockedEvals are used to track assessments that should not be queued until a particular type of node is available. When an assessment runs through the scheduler and produces a failed allocation, it goes into a blocked state. The block is removed only when the capacity to run the failed allocation becomes available. The Planner is used to manage the assignment plan and submit the submission plan of the task assignment to the current server leader through the schedule queue. The Watcher is used to monitor the deployment created by the scheduler and its assignment, and to trigger the scheduler when a health state transition is assigned. Nomad provides three types of Scheduler (Scheduler), provides a service Scheduler, resident service reference Borg system, using one of the most appropriate scoring algorithm, to meet operating constraints, most of the node to place the task group, and select the best node for larger ranking candidate nodes can increase the scheduled time, But it will provide a greater guarantee of the optimality of job placement; A batch scheduler is provided for jobs that are less sensitive to short-term performance fluctuations and have a short resident time. After finding a node set that meets job constraints, it uses the functionality described in Sparrow[30] scheduler proposed by Berkeley to limit the number of ranked nodes and delay optimization of the batch workload. There is also a system scheduler for registering jobs that should run on all clients that meet job constraints, which is useful for deploying and managing tasks that should exist on each node in the cluster. The system scheduler is also called when the client joins the cluster or transitions to the ready state. Starting with Nomad V0.9, the system scheduler preempts low-priority tasks running on the node if there is not enough capacity to place the system jobs. In addition to the three built-in schedulers mentioned above, users can also customize the scheduler to their needs.

After job scheduling, the Nomad server assigns the task to a client device node that meets the constraints. This node must support the runtime driver required by the task. In the driver plug-in mechanism, the Nomad client is equivalent to the client of the run time driver plug-in and communicates with the run time task driver plug-in through gRPC. The driver plug-in server and the client implement the necessary driver plug-in interface, and the driver plug-in client may also implement other interfaces according to the characteristics of the interworking runtime. The driver plug-in client encapsulates many API models. In communication with the driver plug-in server, the driver client is receiving information in most cases, and the specific implementation of the driver plug-in interface is still in the driver plug-in server.

The runtime task driver plug-in contains the driver plug-in server that performs the specific task workload. A DriverNetwork is the network information returned by a run time task driver after starting a task. The DriverNetwork cannot change between calls for the same assignment. This part creates the specific network scheme required by the runtime. The driver plug-in client needs to return the driver network information, mainly the IP address and port of task running for service registration and health check. The TaskHandle is a state shared by the runtime driver and the Nomad client. It is returned to the driver plug-in client after a task has been started and is used to recover the task during a runtime driver restart. The version of the task handle is set by the runtime driver, allowing it to handle upgrades from the older DriverState structure. The event notifier (Eventer) is used to control the broadcast of task events to all consumers, and its lifecycle is tied to that of the driver. Executor is used to start and monitor user processes at run time. Resource isolation is required between processes, and resource restrictions are not mandatory. However, for Linux processes, placing processes in Cgroups enables more precise cleanup of processes. The default execution implementation on Linux sets Cgroups to provide resource and file system isolation in addition to process monitoring and to measure memory and CPU-related resource data. The executor may have a layer of Shim depending on the runtime. For example, when the runc/ LibContainer runtime is connected, the LibcontainerExecutor implementation uses libContainer Shim to take over, create and manage the container. Set configured isolation and restrictions before Execve enters the user process. The actuator also abstracts the plug-in mechanism of THE C/S architecture based on RPC decoupling, for the potential need of runtime driver actuator extension, which is also compatible with the actuator before version 0.9 in this way.

The above three parts briefly analyze the main modules involved in the runtime driver job execution. The principles of Nomad are simplicity and flexibility, in addition to its support for microservices, batch, containerized, and non-containerized workloads, as well as its own configuration language. The following is a sample HCL configuration, as shown in the figure.

Figure 3.2 HCL configuration sample

Nomad client-driven configurations and job configurations are defined in the HCL format, which is the unified configuration language for HashiCorp open source products. HCL is a human-machine friendly structured configuration language designed specifically for servers, DevOps tools, etc. It is a declarative infrastructure that is code, and provides accompanying command-line tools for use. HCL is fully compatible with JSON and is a superset of JSON. It makes up for the disadvantages of JSON, such as lack of annotation and verbosity. HCL is also friendlier than YAML, which requires careful use of whitespace and special characters. It is perfectly legal to use the JSON format as input to the HCL configuration, so even if HCL is the proprietary configuration language, JSON as the interoperability layer can interoperate with other systems.

To access runtime driver interface implementation way, realize the virtual during Libvirt, give the runtime drives a lot of autonomy and independence, including the definition of the configuration mode of virtual during driving, driving job configuration schema definition, configuration, parsing, the lifecycle of the task to run the closed loop, and Nomad agent participation has been greatly reduced, Set only part of necessary information and receive information, and participate in decision-making on key life cycle nodes such as scheduling.