Author: Yang Xiaohe/Post editor: Zhang Handong

A: huawei | based on the next generation of Rust – StratoVirt virtualization platform

StratoVirt is a lightweight virtualization platform of open source in openEuler community, which is competitive in the industry with light weight, low noise and strong security. The StratoVirt process runs in user mode. Before the VM starts, StratoVirt completes preparations for the vm startup, including initialization of VM memory, CPU registers, and devices. The AddressSpace management module of StratoVirt, AddressSpace, implements memory initialization and vm AddressSpace management. The following describes the components of the StratoVirt address space management module and its position in StratoVirt.

Stratovirt ├ ─ ─ address_space │ ├ ─ ─ Cargo. Toml │ └ ─ ─ the SRC │ ├ ─ ─ the address. The rs │ ├ ─ ─ address_space. Rs │ ├ ─ ─ host_mmap. Rs │ ├ ─ ─ Lib. Rs │ ├ ─ ─ listener. Rs │ └ ─ ─ region. The rs ├ ─ ─ boot_loader ├ ─ ─ Cargo. Lock ├ ─ ─ Cargo. Toml ├ ─ ─ CPU ├ ─ ─ devices ├ ─ ─ ├── heavy exercises ── heavy exercises ── heavy exercises ── heavy exercises ── heavy exercises ── heavy exercisesCopy the code

The overall design of StratoVirt address space module

The main structural meanings in the figure above are as follows:

  • AddressSpace

    Address space: a management structure for the address space module, responsible for vm address space management

  • Region

    Indicates an address range. Users in this address range can be classified into the following types:

    1. RAM: specifies the MEMORY segment used by the VM.
    2. IO: specifies the IP address segment used by the VM device.
    3. Container: Used as a Container, it can contain multiple objectsRegion. For example, address management for PCI bus domains can be usedContainertheRegion, which can contain the address range used by PCI devices in PCI bus domain. This type ofRegionAddress management that helps manage and distinguish between memory domains and PCI bus domains.

The design of address space module adopts the combination of tree structure and flat view. The tree structure allows you to quickly learn the topology relationship between regions. This hierarchical and classified design can manage and distinguish the address management of memory domain and PCI bus domain, and form a tree management structure corresponding to PCI device tree. For the FlatView FlatView, it is a linear view based on the address range and priority attributes of these regions. When the AddressSpace management structure is used to access the device or memory, the FlatView FlatView can be used to locate the corresponding Region more conveniently and quickly.

In the tree topology, each Region corresponds to a priority attribute. If the address range occupied by a lower-priority Region overlaps that occupied by a higher-priority Region, the overlap of the lower-priority Region will be overwritten. That is, not visible in the FlatView FlatView.

Updates to tree topology results are likely to result in updates to flat FlatView. Some devices or modules need to get the latest flat view and perform some operations accordingly. For example, the Vhost device needs to synchronize all memory information in the flat view to the kernel Vhost module so that the message notification process can be completed through shared memory. In addition, you need to register the allocated and mapped vm physical address and host virtual address information with the KVM module to speed up memory access performance with hardware-assisted virtualization. Based on the above requirements, we introduce the linked list of address space listener functions in the figure above. The linked list is called successively after FlatView FlatView is updated, which can conveniently complete information synchronization. This list allows other modules to add a custom listening callback function.

Address space optimization

As the basic module and access-intensive module of StratoVirt, address space module not only needs to meet the interface ease of use and functional robustness, but also needs to constantly optimize performance. The following are several optimization points during the iteration.

Topology structure update optimization

The AddressSpace management module provides interfaces for adding and deleting regions into the tree topology, and sets the AddressSpace structure to manage the entire data structure and generate the updated FlatView structure.

To add a sub-region, call the add_subregion interface of the Region structure. The parent Region must be of the Container type. If subregions are added or removed from a Region in the tree and the topology of the tree changes, how does the AddressSpace structure of the FlatView, which generates and updates the FlatView, know that the change has occurred?

The simplest implementation is to add a member to a Region structure and point to the AddressSpace to which it belongs, as shown in the figure above. If you are familiar with Rust, this implementation introduces the problem of resource reference. As a result, AddressSpace and Region cannot release memory resources at the end of their life cycles due to mutual references. Therefore, in the tree structure of the AddressSpace module, all regions use STD ::sync::Weak Pointers to their AddressSpace Pointers. The Weak pointer does not increase the reference count of the object to which it points, ensuring the destruction and resource release of the corresponding structure at the end of the life cycle.

pub struct Region {
    region_type: RegionType,
    priority: Arc<AtomicI32>,
    size: Arc<AtomicU64>,
    offset: Arc<Mutex<GuestAddress>>,
    mem_mapping: Option<Arc<HostMemMapping>>,
    ops: Option<RegionOps>,
    io_evtfds: Arc<Mutex<Vec<RegionIoEventFd>>>,
    space: Arc<RwLock<Weak<AddressSpace>>>,
    subregions: Arc<RwLock<Vec<Region>>>,
}
Copy the code

Lock the optimization

Lock granularity minimization

In order to increase the usability of interfaces, the AddressSpace module must be designed to ensure multi-threaded security. AddressSpace structure as the main interface is as follows. As you can see, the key members of AddressSpace start with Arc

> ensures the security of multi-threaded sharing.
<..>

pub struct AddressSpace {
    root: Region,
    flat_view: ArcSwap<FlatView>,
    listeners: Arc<Mutex<Vec<Box<dyn Listener>>>>,
    ioeventfds: Arc<Mutex<Vec<RegionIoEventFd>>>,
}
Copy the code

Address space space management minimizes the granularity of locks at design time to reduce the impact of multithreaded data contention.

Lock performance optimization

Pub struct AddressSpace {root: Region, flat_view: Arc<RwLock<FlatView>>, listeners: Arc<Mutex<Vec<Box<dyn Listener>>>>, ioeventfds: Arc<Mutex<Vec<RegionIoEventFd>>>, }Copy the code

FlatView, representing FlatView, plays an important role in the key data structure of address space management module. First, when the tree topology changes, such as adding or deleting regions, FlatView FlatView changes. Therefore, the write lock of flat_view member in AddressSpace should be obtained for updating FlatView. Second, when the device accesses the memory and the VCPU logs out of the StratoVirt to access the device, the flat_view member of AddressSpace is used to obtain the read lock, locate the corresponding Region, and perform read and write operations.

However, there are a couple of problems with using Rust read-write locks. First, Rust read-write locks have been tested to perform worse than mutex locks. The performance of read/write locks and mutex is worse than that of atomic types. Secondly, in some scenarios, the address space management module needs to implement the function reentrant support, that is, the tree topology and FlatView FlatView can be updated even when the FlatView read lock is held (for example, PCI bar space update, You need to access the device register via AddressSpace to set the address and add the AddressSpace PCI Bar space to AddressSpace).

Based on the above issues and scenario requirements, using read and write locks and mutex in Rust can cause problems. RCU locks not only satisfy the concurrency of multiple readers and few writers, but also allow simultaneous reading and writing. Through the investigation of self-implementation and existing third-party libraries, we finally choose the RCU-like mechanism of arc_swap third-party library, which can not only meet the requirements of reentrant, but also improve the performance of memory access through the address space module by more than 20%.

pub struct AddressSpace {
    root: Region,
    flat_view: ArcSwap<FlatView>,
    listeners: Arc<Mutex<Vec<Box<dyn Listener>>>>,
    ioeventfds: Arc<Mutex<Vec<RegionIoEventFd>>>,
}
Copy the code

Pay attention to our

StratoVirt is currently open source in the openEuler community, an open source, free Linux distribution platform that works with developers around the world to build an open, diverse, and architecturally inclusive software ecosystem. In the future, we will share a series of topics so that you can have a more detailed understanding of StratoVirt implementation. We are looking forward to your participation!

Project address: gitee.com/openeuler/s…

Project wiki: gitee.com/openeuler/s…

Project exchange: Virt mailing list or submit an issue.