Author: Zhou Xinyu

Review & proofread: White Japan, Jia Jia

Editing & Typesetting: Wen Yan

This article is compiled from the China Open Source annual conference speech

First of all, I would like to introduce myself. I am Zhou Xinyu, PMC member of RocketMQ. Currently, I am responsible for the product development of Alibaba Cloud RocketMQ and EventBridge. Today, my share mainly includes the following parts:

  • Messages and events, microservices and event-driven architectures

  • Aliyun EventBridge: Event-driven Architecture Practice

  • Build ali Cloud unified event hub based on RocketMQ kernel

  • New trend in cloud native era: Serverless+ event driven

  • The future of event-driven architecture

Messages and events, microservices and event-driven architectures

First, let’s talk about the difference between messages and events: You all know messages in RocketMQ, which is a very general concept, a more abstract concept than events. Because the content body of the message is a Byte array without any definition, it is a weak Data, so it is a very general abstraction.

In contrast, events can be more concrete. Typically, it has a Schema that describes exactly what fields an event has. CloudEvents, for example, has a clear Schema definition for events. Events also often represent the occurrence of something, a change in a state, so it is very concrete.

In terms of usage, messages tend to be used in asynchronous decoupled architectures for microservices. But in this case, event-driven and messaging are slightly similar. Message usage scenarios often occur within an organization, and the producers of the message know how the message will be processed. For example, in a team, the producer and sender of a message may be the same team and business and have a very strong convention on the content of the message. Events, by contrast, are more loosely coupled; for example, the event sender has no idea where the event will be delivered, who will consume it, who will be interested in it, and no expectation of how the event will be handled. So, event-based architecture is more decoupled. Messaging applications are often tied to the same business unit, even though some large companies involve cross-departmental collaboration at best. The use of messages is constrained by documents and events are constrained by schemas, so we consider events to be more thoroughly decoupled than messages.

Next, what is the difference between a microservices architecture and an EDA architecture?

The first is the microservice architecture, which evolved from a single application. For example, a single application is divided into many microservices, which are organized and connected through RPC. In the past, a business might have choreographed a bunch of functions locally and now string them together through a bunch of RPCS. For example, when a user places an order in the front end, there may be several micro-services in the background for order operation. One micro-service creates an order, another micro-service processes the order, and then another micro-service notifies the message that the order has been completed. This is a typical RPC architecture.

But the pure RPC architecture has many problems, such as the fact that all the business logic is coupled together, replacing local method calls with remote ones. When business growth reaches a certain stage, the capacity of each micro-service may be unequal. For example, SMS notification can be completed asynchronously but synchronously. As a result, the amount of traffic on the front end needs to be the same as that on the SMS notification. When the preparation resources are not sufficient and the upstream and downstream traffic is not equal, it is possible to cause a micro server to be hit, which affects the upstream, and then has an avalanche effect.

In this case, you would typically introduce message queues for asynchronous decoupling. This architecture is very close to event-driven architecture. Again, if the user creates an order, the event created by the order is sent to the event bus, Event Broker, and Event Bus, and the various downstream subscribers listen for the event.

The difference is that the message subscribers do message processing based on the SDK provided by the message middleware manufacturer, and the business often needs to be transformed and bound by the technology stack provided by the manufacturer. In event-driven architecture, subscribers are generalized subscriptions, that is, they are not required to be developed on any technology stack. They can be an HTTP gateway, a function, or even a legacy stock system. As long as the Event Broker is compatible with the business protocol, events can be pushed to different subscribers. As you can see, generalized subscriptions are more versatile, more decoupled, and have the lowest transformation costs.

Aliyun EventBridge: Event-driven Architecture Practice

Gartner has predicted that the EDA architecture will become the mainstream of microservices in the future. By 2022 it will be 60% of new digital business solutions and 50% of business organizations will be involved.

At the same time, the CNCF Foundation also proposed the CloudEvents specification, which aims to declare event communication using a uniform canonical format. EventBridge follows this standard. CloudEvents, as a community standard, addresses vendor lock-in concerns and improves interoperability between systems by agreeing to a common language. This is a key step.

Events have a uniform specification in the open source community, but in the cloud, many users buy many cloud products from cloud vendors, and these cloud products may have hundreds of millions of events occurring every day, which lie in the logs and internal implementations of different cloud services. Users can’t see, nor can they see, what happens to cloud product instances on the cloud. Different vendors have different definitions of events, and there is no same standard on the whole. Events between cloud services are isolated, that is, they are not connected, which is not conducive to mining the value of events. There is a similar problem when using open source products. Users often do not have a unified standard for data communication, and it is necessary to pay secondary development costs when trying to open up the ecosystem.

Finally, the status quo of event-driven applications in many scenarios is off-line, and fewer people are using EDA architectures for online scenarios. On the one hand, because there is no event-based middleware infrastructure, it is difficult for an event to be acquired in real time and pushed in real time, and the whole link can be traced by the business side. Therefore, the above is also why Ali Cloud to do this product background.

Therefore, we defined EventBridge as having several core values:

1. Unified event hub: Unify event interface, define event standard, and break the isolated island of cloud product events.

Event-driven engine: massive event source, millisecond level trigger ability, accelerate EDA/Serverless architecture upgrade.

Iii. Openness and integration: provide rich cross-product and cross-platform connectivity to promote the integration of cloud products, applications and SaaS services.

Let’s start with the basic Model of EventBridge, which has four main sections. The first part is event sources, which include cloud service events, custom applications, SaaS applications, and self-built data platforms.

The second part is the event bus, which is the storage entity, the event comes in, it has to exist somewhere for asynchronous decoupling. Similar to the concept of topic in RocketMQ, it has some storage but also provides asynchronous capability. Event bus covers two kinds, one is a default event bus, which is used to collect all the events of cloud products, and the other is a custom event bus, which is used to manage, define, and send and receive events by users themselves, which is used to practice EDA architecture concepts. The third part is rules. Rules are similar to RocketMQ’s consumer and subscription, but we give rules more computing power including filtering and transformation. The fourth part is the event target, the subscriber, which is interested in an event to create rules associated with the event, including function calculation, message service, HTTP gateway, and so on.

Here’s a look at the event rule, which is similar to a subscription, but has lightweight event handling capabilities. For example, when using a message, you may need to take the message locally and decide whether to consume it or not. However, based on rules, the message can be processed on the server.

Event rules support very complex event pattern filtering, including matching of specified values, such as prefix matching, suffix matching, numeric matching, array matching, and even combining these rules to form complex logical matching capabilities.

Another is the converter capability, the event target generalization definition, which accepts many event formats, but not necessarily downstream services. For example, if you want to push events to a pin, the pin API is already written and only accepts fixed formats. So, pushing the event to the past requires a transformation of the event. We offer the following:

  • Complete event: Post native CloudEvents directly without conversion.

  • Partial event: Extract partial content from CloudEvents and post it to the event target using JsonPath syntax.

  • Constants: The event only acts as a trigger, and the post is a constant.

  • Template Converter: Flexibly render custom content to event templates by defining templates.

  • Function: Custom function processing of the event by specifying a handler to deliver the return value to the event target.

EventBridge now integrates over 80 cloud products and over 800 event types. It was the first time that we opened up the messaging ecosystem. For example, RocketMQ is a micro-service ecosystem. RocketMQ events can be delivered directly to EventBridge, processed through event-driven architecture, and even integrated with MQTT, KafKa, and other message ecosystems. In addition to ali Cloud message products through, the next step will also open source some self-built message systems through. Another ecosystem is the ISV ecosystem. Why do ISVs need EventBridge? Take Nail 6.0, for example, which recently released connector capabilities. A lot of software needs to be installed in Dingduo, which may be provided by the authorities, ISV or third-party developers, resulting in poor data interoperability. So, we provide this capability to get ISV data flowing. Finally, there is event-driven ecology. We are currently able to reach more than 10 event goals and are still enriching them.

Events are more decoupled and discrete than messages, making event governance more difficult. So, we made the event center and provided three pieces of capability:

  • Event tracking: There is complete tracking of each event, where it was generated, when it was posted, when it was filtered, when it was posted to a target, and when it was successfully processed. Make the entire life cycle completely traceable.

  • Event insight & Analysis: From EDA programming perspective to user perspective, users can quickly understand what events are in EventBridge and perform visual analysis. EB achieves the nearest calculation and analysis, directly importing the business message into the event bus, and timely analyzing the message.

  • Event market: For cloud products, guide cloud products to define business events to make cloud products more open, thus providing market capabilities.

Build ali Cloud unified event hub based on RocketMQ kernel

EventBridge was built on top of cloud-native container services from the start. On top of this is the RocketMQ kernel, which plays two roles in this product. One is the event store, which is used as storage. The other is to take advantage of subscription capabilities and convert subscriptions into generalized subscriptions. On top of the RocketMQ kernel is the Connect cluster. One of the most important capabilities of EventBridge is to connect, so EventBridge needs to have the ability to send event sources and then save them. Its core is Connect cluster, and each Connect cluster has many workers. Each Worker is responsible for many things, including event intake, event filtering, event conversion, event playback, event tracking, etc. Meanwhile, Connect control surface is installed on Connect cluster to complete cluster governance, Worker scheduling, etc.

On the upper layer is the API Server, an event gateway. In the EventBridge world, there are two ways to consume events. One is through the Source Connector of Connect, which sends the active Source of the event. Another user or cloud product can post events via API Server via our SDK. There are several ways to post events, including OpenAPIS, official SDKS in multiple languages, as well as the CloudEvents community standards. EventBridge is also fully compatible with the community open source SDKS, and users can post events via Webhooks.

The advantages of this architecture are obvious:

(1) Reduce user development costs

  • Users do not need additional development for event handling

  • Write rules to filter and transform events

(2) Native CloudEvents support

  • Embrace the CNCF community and seamlessly connect to the community SDK

  • Standard protocol unified Ali Cloud event specification

(3) Event Schema support

  • Supports automatic event Schema detection and validation

  • Schema binding for Source and Target

(4) Random intercommunication of global events

  • A cross-regional and cross-account event network has been established

  • Supports cross-cloud and cross-data center event routing

New trend in cloud native era: Serverless+ event driven

We believe that Serverless plus event-driven is a new way of research and development. Different manufacturers have different emphasis on Serverless, but their landing ways are similar.

First, the Serverless infrastructure blocks out the underlying IaaS. The upper Serverless runtime is computation-hosted, hosting not just microservice applications, K8s containers, and not just functions.

EventBridge first connects the event sources of such drivers to trigger these runtimes. Because what Serverless needs most is the driver, the event driver gives him the ability to compute portals. EventBridge drives the Serverless runtime and then connects to the back-end service. Currently, EventBridge and Serverless combine mainly in loosely coupled scenarios, such as front-end applications, SaaS applets, and audio and video codec landing scenarios.

So, what is the development mode of Serverless EDA architecture? Take function calculation as an example. Firstly, developers need to transform from the perspective of application to the perspective of function, and implement each business logic in one function. A function represents a snippet of code that represents a specific business. When uploaded, the code becomes a function resource. EventBridge then drives the function through events, orchestrating the function into a specific application.

Function still needs to do a lot of things. As we all know, function has many disadvantages, and the most criticized one is cold start. Because Serverless requires scale to zero, it should receive 0 directly when there is no request or event to trigger, and 0~1 is a cold start. This cold start can sometimes take seconds, because it may involve downloading code, downloading images, building namespace, storage mount, root mount, many of these things, cloud vendors put a lot of effort to optimize this area. The price advantage of Serverless is obvious. Its resource utilization rate is particularly high. Due to payment by volume, it can achieve nearly 100% resource utilization rate without capacity planning.

A simple example is the minimalist programming paradigm based on Serverless plus EDA. Another specific example is the transformation of this business by EDA architecture in the new retail scenario. First of all, there are several key resources in the business, including API gateway and function calculation. First of all, we can open up some data, open up RDS and synchronize RDS data, compatible with some historical architectures, and trigger computing resources, functions and gateways. The overall architecture has obvious advantages, so it has extreme flexibility and does not need to reserve resources.

Event-driven vision of the future

We believe that the event-driven future has two parts. One is to do a good job of connectivity and integration within and across the cloud to make users’ diverse architectures more efficient. The second is the integration of the open source ecosystem. We can see that the open source ecosystem is becoming more and more vigorous, so we also need to integrate the data in the open source ecosystem. In addition, there are traditional IDC computing capacity, edge computing capacity these ecological need to have connectivity software to connect it.

EventBridge is the new computing driver in the cloud native era. This data can drive the computing power of the cloud and create more business value.

For more information, please scan the QR code below or search wechat (AlibabaCloud888) to add cloud native assistant! For more information!