Currently IaaS software is more like cloud controller software and lacks many features to become a complete cloud solution. As an evolving technology, it is very difficult to predict all the features required for a complete solution, so an IaaS software cannot complete all of its features at the beginning. Based on the above facts, an IaaS software architecture must be able to maintain core structure stability while adding new features. ZStack’s general-purpose plug-in system allows features to be implemented as plug-ins (in or out of threads), which not only extends ZStack’s capabilities, but also injects business logic into it to change default behavior.

motivation

Subbu Allamaraju, eBay’s lead engineer in managing OpenStack private cloud, said:

However, OpenStack is cloud controller software. While the community has contributed greatly to the formation of OpenStack, an installed instance of OpenStack is not a cloud. As a manipulator you have to deal with a lot of additional operations that the user doesn’t necessarily know about. These include basic staff training, initialization, maintenance, configuration management, patching, packaging, upgrading, high availability, monitoring, measurement, user support, capacity forecasting and management, billing or refund, resource reclamation, security, firewalls, DNS, integration with other internal infrastructure and tools, and so on and so on. These activities will take a lot of time and energy. OpenStack provides some of the ingredients necessary to create a cloud, but it doesn’t package the cloud well.

This expresses the current state of IaaS software, with the exception of some very well built IaaS (like AWS), most IaaS (including our ZStack) are still not a complete cloud solution. The public cloud has a more mature model of what a public cloud solution should look like, thanks to pioneers like Amazon who have been exploring it for years. Still under development, private clouds do not yet have a proven and complete solution. Unlike dedicated public cloud software, which can be customized for the manufacturer’s infrastructure and services; Open source IaaS software must consider the needs of both public and private clouds, making it more difficult to create a complete solution. Because there was no way to predict what a complete solution would look like, our only solution was to provide a plug-in architecture that could add plug-ins without compromising core business stability.

The problem

A lot of software claims to be plug-in, but a lot of it isn’t really plug-in, or at least not completely plug-in. Before explaining why, let’s look at the two main forms of plug-in architecture. Although many articles have been written on this topic, in our experience we have grouped all plug-ins into two structures that can be accurately described as the policy pattern and the observer pattern in GoF Design Patterns.

Plug-ins derived from the strategist pattern

This form of plug-in usually extends software specific functionality by providing different implementations. Or add new functionality by adding plug-in APIs. A lot of familiar software is built in this way, for example, operating system drivers, web browser plug-ins. This plug-in composition works by allowing applications to access the plug-in through well-defined protocols.

Plugins derived from observer mode

This form of plug-in typically injects the application’s business logic for specific events. Once an event occurs, the plug-in that hangs on it will be called to execute a piece of code that may even change the flow of execution, for example, by throwing an exception to stop the flow of execution when the event meets certain conditions. Plug-ins based on this pattern are typically transparent to the end user and implemented purely internally, with, for example, a listener listening for database insert events. The way this plug-in works is that it allows the plug-in to access the application through well-defined extension points.

Most software claims to be plug-in and either implements one of these components or has some code that implements them. To become fully plug-in, software must envisage the idea that all business logic is implemented in both ways. This means that the whole software is made up of lots of little plug-ins, like lego.

Plug-in system

An important design principle runs through all of ZStack’s components: Each component should be designed to be minimally informative, self-contained, and independent of other components. For example, in order to create a virtual machine, allocating disks, providing DHCP, and setting up SNAT are all necessary steps, and managing the components that create the VM should be very clear. But does it really need to know so much? Why can’t this component simply allocate CPU/ memory for the VM, then send a startup request to the host, and let other components like storage and networking care about their own business? You’ve probably already guessed the answer: No, in ZStack, components don’t need to know that much, right! It can be that simple. We are fully aware of the fact that the more information your components know, the more tightly coupled your application, and you end up with complex software that is difficult to modify. So we provide the following plug-in form to ensure that our architecture is loosely coupled and to make it easy to add new features to a complete cloud solution.

1. Policy mode plug-ins

Plug-ins in IaaS software are typically drivers that integrate different physical resources. For example, NFS primary storage, ISCSI primary storage, VLAN based L2 network, Open vSwitch based L2 network. These plug-ins are all forms of the policy pattern we just mentioned. ZStack has abstracted cloud resources into: virtual machine manager, primary storage, backup storage, L2 network, L3 network, and so on. Each resource has an associated driver as a separate plug-in. To add a new driver, the developer only needs to implement three components: a type, a factory, and a concrete resource implementation, all packaged in a single plug-in, usually built as a JAR file. Using Open vSwitch as an example, let’s assume that we will create a new L2 network using Open vSwitch as the background, and then the developer needs to:

1.1 Define an Open vSwitch L2 network that will automatically register with the ZStack L2 network type system.

public static L2NetworkType type = new L2NetworkType("Openvswitch");

/* once the type is declared as above, there will be a new L2 network type called 'Openvswitch' that can be retrieved by API */(Once the type is declared, a new L2 network type called "Openvswitch" can be retrieved by the API.)Copy the code

1.2 Create an L2 network factory responsible for returning a concrete implementation to L2 network services.

public class OpenvswitchL2NetworkFactory implements L2NetworkFactory {
    @Override
    public L2NetworkType getType(a) {
        Return type defined in 1.1 */
        return type;
    }

    @Override
    public L2NetworkInventory createL2Network(L2NetworkVO vo, APICreateL2NetworkMsg msg) {
        /* * new resource will normally have own creational API APICreateOpenvswitchL2NetworkMsg that * usually inherits APICreateL2NetworkMsg, and own database object OpenvswitchL2NetworkVO that * usually inherits L2NetworkVO, and a java bean OpenvswitchL2NetworkInventory that usually inherits * L2NetworkInventory representing all properties of Openvswitch L2 network. */(the new resources will usually have a create API APICreateOpenvswitchL2NetworkMsg, usually inherited from APICreateL2NetworkMsg, will also have their own database objects, OpenvswitchL2NetworkVO, Usually inherited from L2NetworkVO, and a Java bean OpenvswitchL2NetworkInventory, Usually said in L2NetworkInventory Openvswitch all attributes of the L2 network) APICreateOpenvswitchL2NetworkMsg CMSG = (APICreateOpenvswitchL2NetworkMsg)APICreateL2NetworkMsg; OpenvswitchL2NetworkVO cvo =new OpenvswitchL2NetworkVO(vo);
         evaluate_OpenvswitchL2NetworkVO_with_parameters_in_API(cvo, cmsg);
         save_to_database(cvo);
         return OpenvswitchL2NetworkInventory.valueOf(cvo);
    }

    @Override
    public L2Network getL2Network(L2NetworkVO vo) {
        /* return the concrete implementation defined in 1.3 */
        return newOpenvswitchL2Network(vo); }}Copy the code

1.3 Create a concrete Open vSwitch L2 network implementation to interact with the background Open vSwitch controller.

public class OpenvswitchL2Network extends L2NoVlanNetwork {
    public OpenvswitchL2Network(L2NetworkVO self) {
        super(self);
    }

    @Override
    public void handleMessage(Message msg) {
        /* handle Openvswitch L2 network specific messages(both API and non API) and delegate * others to the base class L2NoVlanNetwork; so the implementation can focus on own business * logic and let the base class handle things like attaching cluster, detaching cluster; * of course, the implementation can override any message handler if it wants, for example, Override L2NetworkDeletionMsg to do some cleanup work before being deleted L2 network specific messages (API messages or messages that are not API messages) and pass the rest to L2NoVlanNetwork's base classes; So its implementation can focus on its own business logic and let the base classes handle things like binding clusters, unbinding clusters, etc. Of course, the implementation method can also override any message handler, for example, override L2NetworkDeletionMsg to do some cleanup before deleting. * /
        if (msg instanceof OpenvswitchL2NetworkSpecificMsg1) {
            handle((OpenvswitchL2NetworkSpecificMsg1)msg);
        } else if (msg instanceof OpenvswitchL2NetworkSpecificMsg2) {
            handle((OpenvswitchL2NetworkSpecificMsg2)msg);
        } else {
            super.handleMessage(msg); }}}Copy the code

By putting the three components together in a Maven module, adding some necessary Spring configuration files, and compiling them into JAR files, you create a new L2 network type in the ZStack. All Zstack resource drivers are implemented through the above steps (type, factory, concrete implementation). Once you’ve learned how to create drivers for one resource, you’ve learned how to do it for all resources. As we mentioned in “ZStack– In-process Microservices Architecture”, drivers can have their own apis and configuration methods.

2. Observer mode plugin

Policy mode plugins (drivers) allow you to extend the functionality of existing ZStack; However, in order for the architecture to be loosely coupled, plug-ins must be able to inject the application’s business logic, even that of other plug-ins; The key to looking at pattern plug-ins is extension points, which allow a piece of plug-in code to be called while a code flow is running. Currently, Zstack defines about 100 extension points, exposing a number of scenarios for plug-ins to receive events or change code behavior. Creating a new extension point defines a Java interface that components can easily create to allow other components to inject their own business logic. To see how it works, let’s continue with our Open vSwitch example; Assume that the Open vSwitch L2 network needs to hook into the process of creating a VM to prepare GRE tunnels before the VM is created. This plug-in implements the following:

PreVmInstantiateResourceExtensionPoint:
public class OpenvswitchL2NetworkCreateGRETunnel implements PreVmInstantiateResourceExtensionPoint {
    @Override
    public void preBeforeInstantiateVmResource(VmInstanceSpec spec) throws VmInstantiateResourceException {
       /* * you can do some check here; if any condition makes you think the VM should not be created/started, * you can throw VmInstantiateResourceException to stop it */
    }

    @Override
    public void preInstantiateVmResource(VmInstanceSpec spec, Completion completion) {
        /* create the GRE tunnel, you can get all necessary information about the VM from VmInstanceSpec */
        completion.success();
    }

    @Override
    public void preReleaseVmResource(VmInstanceSpec spec, Completion completion) {
       /* *in case VM fails to create/start for some reason, for cleanup, you can delete the prior created GRE tunnel here */completion.success(); }}Copy the code

When the ZStack connected to the KVM host, the Open vSwitch L2 network want to check on a host computer and start the Open vSwitch daemons, it realize KVMHostConnectExtensionPoint:

public class OpenvswitchL2NetworkKVMHostConnectedExtension implements KVMHostConnectExtensionPoint {
    @Override
    public void kvmHostConnected(KVMHostConnectedContext context) throws KVMHostConnectException {
        /* * You can use various methods like SSH login, HTTP call to KVM agent to check the Openvswitch daemon HTTP calls to the KVM proxy) check the Openvswitch daemon on the host using the information on KVMHostConnectedContext. If any state make you think that the host cannot provide Openvswitch L2 network method, you can throw KVMHostConnectExtensionPoint to prevent a socket connection. * on the host, using information in KVMHostConnectedContext. If any condition makes you think the * host cannot provide Openvswitch L2 network function, you can throw KVMHostConnectExtensionPoint to * stop the host from being connected. */}}Copy the code

Finally, you need to advertise that you have two components that implement these extension points, and ZStack’s plug-in system will ensure that the owner calls your component at an appropriate time. This notification is done in the plug-in’s Spring configuration file:

      
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:aop="http://www.springframework.org/schema/aop"
    xmlns:tx="http://www.springframework.org/schema/tx" xmlns:zstack="http://zstack.org/schema/zstack"
    xsi:schemaLocation="Http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.0.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-3.0.xsd http://zstack.org/schema/zstack http://zstack.org/schema/zstack/plugin.xsd"
    default-init-method="init" default-destroy-method="destroy">

    <bean id="OpenvswitchL2NetworkCreateGRETunnel" class="org.zstack.network.l2.ovs.OpenvswitchL2NetworkCreateGRETunnel">
        <zstack:plugin>
            <zstack:extension interface="org.zstack.header.vm.PreVmInstantiateResourceExtensionPoint" />
        </zstack:plugin>
    </bean>

    <bean id="OpenvswitchL2NetworkKVMHostConnectedExtension"
          class="org.zstack.network.l2.ovs.OpenvswitchL2NetworkKVMHostConnectedExtension">
        <zstack:plugin>
            <zstack:extension interface="org.zstack.kvm.KVMHostConnectExtensionPoint" />
        </zstack:plugin>
    </bean>

</beans>
Copy the code

That’s all you need to do. Create a new type of L2 network without changing even a line of code for any of the other ZStack components. This is the foundation on which ZStack keeps its core business processes stable.

No OSGI: Those familiar with Eclipse and OSGI may have noticed that our plug-in system is very similar to Eclipse’s and OSGI’s. One might ask why we don’t just use OSGI, which is specifically designed to create plug-in systems for Java applications. In fact, we spent quite a bit of time experimenting with OSGI; However, we feel that it is too hard. We don’t like having another container in our application, a separate class loader, and the complexity of creating plug-ins. It seems that OSGI is putting a lot of effort into keeping plug-ins separate from each other, but ZStack wants them to be flat. We’ve noticed that many projects introduce unnecessary restrictions in their code to make the overall architecture obviously hierarchical and isolated, but because of poorly designed interfaces, plug-ins have to write a lot of ugly code to overcome these restrictions, disrupting the real architecture. ZStack considers all plug-ins as part of its core and has the same privileges for the core business process. We’re not building a browser-like consumer application where users can install malicious plug-ins by mistake; We’re building enterprise-class software, and every corner needs to be rigorously tested. A flat plug-in system makes our code simple and robust.

3. Out-of-process services (plug-ins)

In addition to these two approaches, developers do have a third way to extend ZStack– out-of-process services. Although ZStack wraps all orchestration services into a single process, functions independent of business process services can be implemented as independent services that run on different processes or even different machines. The ZStack Web UI, a Python application that interacts with the RabbitMQ and ZStack orchestration services, is a good example. ZStack has a well-defined messaging specification and out-of-process services can be written in any language as long as they can interact with RabbitMQ. ZStack also has a mechanism called Canonical Event that exposes internal events to the bus, such as VM creation, VM stop, and disk creation. Software such as a billing system can build an out-of-process service by listening for these events. If a service to outside the process, but still need to visit some haven’t exposed core data structure of the business process, or need to access the database, it can use a hybrid approach, namely on the management node of a small plug-in is responsible for collecting data and sends them to the message broker, outside the process of service receives these data and complete your own thing.

conclusion

In this article, we show the ZStack plug-in architecture. While ZStack is not a complete cloud solution, it provides an architecture to build any future features into plug-ins (in-process or out-of-process), enabling rapid evolution into a full-fledged, complete cloud solution while keeping the core business processes stable.