Sentinel is a lightweight and highly available flow control component that is open source by Ali middleware team and oriented to distributed service architecture. It mainly takes traffic as the entry point and helps users protect the stability of services from multiple dimensions such as flow control, fuse downgrade and system load protection.
You may be asking: What are the similarities and differences between Sentinel and Netflix Hystrix, which is a common fuse downgrade library? The Sentinel website has a comparison article. Here’s a summary of the table, and a comparison can be found here.
| Compare the content of | Sentinel | Hystrix |
|---|---|---|
| Isolation strategy | Semaphore isolation | Thread pool isolation/semaphore isolation |
| Fuse downgrading strategy | Based on response time or failure rate | Based on failure ratio |
| Real-time indicator realization | The sliding window | Sliding Windows (based on RxJava) |
| Rule configuration | Support for multiple data sources | Support for multiple data sources |
| scalability | Multiple extension points | Plug-in form |
| Annotation-based support | support | support |
| Current limiting | Based on QPS, traffic limiting based on call relationships is supported | Does not support |
| Traffic shaping | Support slow start and constant speed mode | Does not support |
| System load protection | support | Does not support |
| The console | Out of the box, you can configure rules, view second-level monitoring, machine discovery, etc | imperfect |
| Common framework adaptation | Servlet, Spring Cloud, Dubbo, gRPC, etc | Servlet, Spring Cloud Netflix |
It can be seen from the comparison table that Sentinel is more powerful than Hystrix in terms of functionality. In this article, let’s take a look at the source code of Sentinel and uncover the mystery of Sentinel.
The project structure
Fork the Sentinel source code into your Github repository, clone the source code locally, and start reading the source code.
First, let’s take a look at the Sentinel project’s overall structure:
-
Sentinel-core core modules, current limiting, degradation, system protection and so on are implemented here
-
Sentinel – Dashboard console module, can be connected to the Sentinel client to achieve visual management
-
The Sentinel-Transport transport module provides basic monitoring server and client APIS, as well as some implementations based on different libraries
-
Sentinel -extension module, mainly for DataSource partial extension implementation
-
Sentinel-adapter adapter module, mainly to achieve some common framework adaptation
-
How to use Sentinel for traffic limiting and degradation
-
The Sentinel-benchmark module provides benchmarks for the accuracy of core code
Run the sample
Almost every framework comes with sample modules, some called Example, some called demo, and Sentinel is no exception.
Let’s take an example from the Sentinel demo and run it to see the general situation. As mentioned above, the main core function of Sentinel is to do flow limiting, downgrading and system protection. Let’s start with the implementation principle of Sentinel from “flow limiting”.
We can see that there are many different examples in sentinel-Demo module. We can find the flow package in basic module. Below this package is the corresponding flow limiting sample, but there are many types of limiting.
public class FlowQpsDemo {
private static final String KEY = "abc";
private static AtomicInteger pass = new AtomicInteger();
private static AtomicInteger block = new AtomicInteger();
private static AtomicInteger total = new AtomicInteger();
private static volatile boolean stop = false;
private static final int threadCount = 32;
private static int seconds = 30;
public static void main(String[] args) throws Exception {
initFlowQpsRule();
tick();
// first make the system run on a very low condition
simulateTraffic();
System.out.println("===== begin to do flow control");
System.out.println("only 20 requests per second can pass");
}
private static void initFlowQpsRule() {
List<FlowRule> rules = new ArrayList<FlowRule>();
FlowRule rule1 = new FlowRule();
rule1.setResource(KEY);
// set limit qps to 20
rule1.setCount(20);
// Set the traffic limiting type according to QPS
rule1.setGrade(RuleConstant.FLOW_GRADE_QPS);
rule1.setLimitApp("default");
rules.add(rule1);
// Load the rules for limiting traffic
FlowRuleManager.loadRules(rules);
}
private static void simulateTraffic() {
for (int i = 0; i < threadCount; i++) {
Thread t = new Thread(new RunTask());
t.setName("simulate-traffic-Task");
t.start();
}
}
private static void tick() {
Thread timer = new Thread(new TimerTask());
timer.setName("sentinel-timer-task");
timer.start();
}
static class TimerTask implements Runnable {
@Override
public void run() {
long start = System.currentTimeMillis();
System.out.println("begin to statistic!!!" );
long oldTotal = 0;
long oldPass = 0;
long oldBlock = 0;
while (! stop) {
try {
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e) {
}
long globalTotal = total.get();
long oneSecondTotal = globalTotal - oldTotal;
oldTotal = globalTotal;
long globalPass = pass.get();
long oneSecondPass = globalPass - oldPass;
oldPass = globalPass;
long globalBlock = block.get();
long oneSecondBlock = globalBlock - oldBlock;
oldBlock = globalBlock;
System.out.println(seconds + " send qps is: " + oneSecondTotal);
System.out.println(TimeUtil.currentTimeMillis() + ", total:" + oneSecondTotal
+ ", pass:" + oneSecondPass
+ ", block:" + oneSecondBlock);
if (seconds-- <= 0) {
stop = true;
}
}
long cost = System.currentTimeMillis() - start;
System.out.println("time cost: " + cost + " ms");
System.out.println("total:" + total.get() + ", pass:" + pass.get()
+ ", block:" + block.get());
System.exit(0);
}
}
static class RunTask implements Runnable {
@Override
public void run() {
while (! stop) {
Entry entry = null;
try {
entry = SphU.entry(KEY);
// token acquired, means pass
pass.addAndGet(1);
} catch (BlockException e1) {
block.incrementAndGet();
} catch (Exception e2) {
// biz exception
} finally {
total.incrementAndGet();
if (entry ! = null) {
entry.exit();
}
}
Random random2 = new Random();
try {
TimeUnit.MILLISECONDS.sleep(random2.nextInt(50));
} catch (InterruptedException e) {
// ignore
}
}
}
}
}
Copy the codeAfter executing the above code, print the following result:
As you can see, the number of passes in the result above is different from our expectation. We expected to allow 20 pass requests per second, but currently there are many passes with more than 20 requests.
The reason is that the code we are testing here uses multiple threads. Notice the value of threadCount. There are 32 threads to simulate. The number of passes will be higher than 20.
The following model can be used to describe a TimeTicker thread doing statistics every 1 second. There are N RunTask threads impersonating requests, the accessed Business code is protected by the resource key, and by rule only 20 requests per second are allowed to pass.
Because counters such as pass, block, and total are shared globally, and multiple RunTask threads do not have internal lock protection when they apply for and obtain entry through SphU. Entry, the number of passes exceeds the set threshold.
In order to prove the correctness and reliability of the flow at the single-threaded lower limit, our model should be like this:
ThreadCount = 1; threadCount = 1; threadCount = 1;
You can see that the number of passes is basically staying at 20, but the first count is still over 20. What causes this?
In fact, if you look closely at the code in the Demo, you can see that one thread is used to simulate the request, and another thread is used to count the results, and the statistics thread is used to count the results every 1 second. There is a time error between the two threads. As can be seen from the timestamp printed by the TimeTicker thread, although statistics are collected every second, the current printing time and the last time are still different, not exactly 1000ms interval.
In order to really verify the 20 requests per second limit and keep the data accurate, you need to do benchmarks.That’s not the focus of this article, but if you’re interested, you can check out the JMH.
In-depth principle
Using a simple example program, sentinel can limit the flow of requests. In addition to limiting the flow, sentinel also has degradation and system protection. Now let’s dig deep into the source code to see how Sentinel works.
Start with the entry: sphu.entry (). This method will apply for an entry. If the application succeeds, the flow is not restricted. Otherwise, a BlockException will be thrown, indicating that the flow is restricted.
Following the sphu.entry () method will enter sph.Entry (). The default Sph implementation class is CtSph, Will perform to the entry in the CtSph ResourceWrapperresourceWrapper intcount, Object… Args)throwsBlockException this method.
Let’s look at an implementation of this method:
public Entry entry(ResourceWrapper resourceWrapper, int count, Object... args) throws BlockException {
Context context = ContextUtil.getContext();
if (context instanceof NullContext) {
// Init the entry only. No rule checking will occur.
return new CtEntry(resourceWrapper, null, context);
}
if (context == null) {
context = MyContextUtil.myEnter(Constants.CONTEXT_DEFAULT_NAME, "", resourceWrapper.getType());
}
// Global switch is close, no rule checking will do.
if (! Constants.ON) {
return new CtEntry(resourceWrapper, null, context);
}
// Obtain the SlotChain corresponding to the resource
ProcessorSlot<Object> chain = lookProcessChain(resourceWrapper);
/ *
* Means processor cache size exceeds {@link Constants.MAX_SLOT_CHAIN_SIZE}, so no
* rule checking will be done.
* /
if (chain == null) {
return new CtEntry(resourceWrapper, null, context);
}
Entry e = new CtEntry(resourceWrapper, chain, context);
try {
// Executes the Slot entry method
chain.entry(context, resourceWrapper, null, count, args);
} catch (BlockException e1) {
e.exit(count, args);
/ / throw BlockExecption
throw e1;
} catch (Throwable e1) {
RecordLog.info("Sentinel unexpected exception", e1);
}
return e;
}
Copy the codeThis method can be divided into the following parts:
-
1. Check parameters and global configuration items. If the parameters do not meet the requirements, a CtEntry object is returned.
-
2. Obtain SlotChain based on the packaged resource object
-
3. Run the Entry method of SlotChain
-
3.1. If a BlockException is thrown by SlotChain’s Entry method, the exception is thrown upwards
-
3.2. If the Entry method of SlotChain is executed properly, the entry object is returned
-
4. If the upper method catches a BlockException, the request is curbed; otherwise, the request will execute normally
The two most important steps are step 2 and step 3. Let’s break these two steps down.
Create SlotChain
First look at the lookProcessChain method implementation:
private ProcessorSlot<Object> lookProcessChain(ResourceWrapper resourceWrapper) {
ProcessorSlotChain chain = chainMap.get(resourceWrapper);
if (chain == null) {
synchronized (LOCK) {
chain = chainMap.get(resourceWrapper);
if (chain == null) {
// Entry size limit.
if (chainMap.size() >= Constants.MAX_SLOT_CHAIN_SIZE) {
return null;
}
// Specify the chain construction method
chain = Env.slotsChainbuilder.build();
Map<ResourceWrapper, ProcessorSlotChain> newMap = new HashMap<ResourceWrapper, ProcessorSlotChain>(chainMap.size() + 1);
newMap.putAll(chainMap);
newMap.put(resourceWrapper, chain);
chainMap = newMap;
}
}
}
return chain;
}
Copy the codeThis method is cached using a HashMap where the key is the resource object. I’ve got a lock here, and I’ve got a doublecheck. Concrete structure chain method is through: Env. SlotsChainbuilder. Build () this code to create. Enter this method.
public ProcessorSlotChain build() {
ProcessorSlotChain chain = new DefaultProcessorSlotChain();
chain.addLast(new NodeSelectorSlot());
chain.addLast(new ClusterBuilderSlot());
chain.addLast(new LogSlot());
chain.addLast(new StatisticSlot());
chain.addLast(new SystemSlot());
chain.addLast(new AuthoritySlot());
chain.addLast(new FlowSlot());
chain.addLast(new DegradeSlot());
return chain;
}
Copy the codeProcessorSlotChain is a list of slots added to ProcessorSlotChain. The realization of the specific need to go to the DefaultProcessorSlotChain.
public class DefaultProcessorSlotChain extends ProcessorSlotChain {
AbstractLinkedProcessorSlot<? > first = new AbstractLinkedProcessorSlot<Object>() {
@Override
public void entry(Context context, ResourceWrapper resourceWrapper, Object t, int count, Object... args)
throws Throwable {
super.fireEntry(context, resourceWrapper, t, count, args);
}
@Override
public void exit(Context context, ResourceWrapper resourceWrapper, int count, Object... args) {
super.fireExit(context, resourceWrapper, count, args);
}
};
AbstractLinkedProcessorSlot<? > end = first;
@Override
public void addFirst(AbstractLinkedProcessorSlot<? > protocolProcessor) {
protocolProcessor.setNext(first.getNext());
first.setNext(protocolProcessor);
if (end == first) {
end = protocolProcessor;
}
}
@Override
public void addLast(AbstractLinkedProcessorSlot<? > protocolProcessor) {
end.setNext(protocolProcessor);
end = protocolProcessor;
}
}
Copy the codeDefaultProcessorSlotChain have two AbstractLinkedProcessorSlot types of variables: the first and the end, this is the head of the linked list node and end node.
Create DefaultProcessorSlotChain object, the first to create the first node, then the first node to the end node, can be expressed in below:
After the first node is added to the list, the structure of the entire list looks like the following:
After all the nodes are added to the linked list, the structure of the entire linked list becomes as shown below:
So will all the Slot objects added to the list, each Slot is inherited from AbstractLinkedProcessorSlot. And AbstractLinkedProcessorSlot is a chain of responsibility design, each object has a next attribute, point to an another AbstractLinkedProcessorSlot object. In fact, the chain of responsibility mode is in many frameworks, such as Netty through pipeline to achieve.
Now that you know how SlotChain is created, it’s time to look at how the Slot Entry method is executed.
Run the Entry method of SlotChain
Instances of ProcessorSlotChain lookProcessChain method is DefaultProcessorSlotChain, then execute chain. Entry method, Will perform DefaultProcessorSlotChain entry method, while DefaultProcessorSlotChain entry method is like this:
@Override
public void entry(Context context, ResourceWrapper resourceWrapper, Object t, int count, Object... args)
throws Throwable {
first.transformEntry(context, resourceWrapper, t, count, args);
}
Copy the codeThat is to say, the entry of DefaultProcessorSlotChain actual transformEntry method is executed first attribute.
And transformEntry method performs the entry of the current node, the first node in the DefaultProcessorSlotChain rewrote the entry method, specific as follows:
@Override
public void entry(Context context, ResourceWrapper resourceWrapper, Object t, int count, Object... args)
throws Throwable {
super.fireEntry(context, resourceWrapper, t, count, args);
}
Copy the codeThe entry method of the first node is actually the fireEntry method of the super, so we continue to focus on the fireEntry method, as follows:
@Override
public void fireEntry(Context context, ResourceWrapper resourceWrapper, Object obj, int count, Object... args)
throws Throwable {
if (next ! = null) {
next.transformEntry(context, resourceWrapper, obj, count, args);
}
}
Copy the codeAs you can see, the execution entry is passed from the fireEntry method, which executes the next node transformEntry method of the current node. As we’ve already analyzed, the transformEntry method triggers the entry of the current node. In other words, the fireEntry method actually triggers the entry method of the next node. The specific process is shown in the figure below:
As you can see from the figure, the initial entry() method calling the Chain is changed to the entry() method calling the slots in the SlotChain. From the above analysis, the first Slot node in SlotChain is NodeSelectorSlot.
Execute the Slot entry method
Now we can look at the entry method of the first node in SlotChain, NodeSelectorSlot, as follows:
@Override
public void entry(Context context, ResourceWrapper resourceWrapper, Object obj, int count, Object... args)
throws Throwable {
DefaultNode node = map.get(context.getName());
if (node == null) {
synchronized (this) {
node = map.get(context.getName());
if (node == null) {
node = Env.nodeBuilder.buildTreeNode(resourceWrapper, null);
HashMap<String, DefaultNode> cacheMap = new HashMap<String, DefaultNode>(map.size());
cacheMap.putAll(map);
cacheMap.put(context.getName(), node);
map = cacheMap;
}
// Build invocation tree
((DefaultNode)context.getLastNode()).addChild(node);
}
}
context.setCurNode(node);
// This triggers the entry method for the next node
fireEntry(context, resourceWrapper, node, count, args);
}
Copy the codeNodeSelectorSlot NodeSelectorSlot NodeSelectorSlot NodeSelectorSlot NodeSelectorSlot NodeSelectorSlot NodeSelectorSlot NodeSelectorSlot
-
NodeSelectorSlot Collects the paths of resources and stores the call paths of these resources in a tree structure for traffic degradation based on the call paths.
-
ClusterBuilderSlot is used to store resource statistics and caller information, such as RT, QPS and Thread Count of the resource. These information will be used as the basis for multi-dimensional traffic limiting and degradation.
-
StatistcSlot is used to record and count runtime information of different latitudes.
-
FlowSlot is used to limit traffic according to preset flow limiting rules and slot statistics.
-
AuthorizationSlot is used to control the blacklist and whitelist based on the blacklist and whitelist.
-
Degrades a pack with statistics and preset rules.
-
SystemSlot controls the total incoming traffic based on the system state, such as load1.
After performing the business logic processing, the fireEntry() method is called, which triggers the entry method of the next node. This is how the sentinel chain of responsibility is passed: After each Slot node completes its business, fireEntry is called to trigger the entry method of the next node.
So you can complete the above diagram as follows:
At this point, the entry() method of each node is invoked through SlotChain. Each node performs its own logical processing according to the created rules. When the statistical result reaches the set threshold, events such as traffic limiting and degradation are triggered, specifically, BlockException is thrown.
conclusion
Sentinel is a linked list based on seven different slots. Each Slot does its job and passes the request to the next Slot until a BlockException is thrown in one of the slots.
The first three slots collect statistics, and the latter slots control whether the request is blocked or allowed based on the statistics results and configured rules.
There are also many options for the type of control: by QPS, thread count, cold start, and so on.
Then, based on this core method, many other functions were derived:
-
1. Dashboard console, which can visually control each sentinel client connected (by sending heartbeat messages) and communicate with the client via HTTP protocol.
-
2. Rule persistence: By implementing the DataSource interface, you can persist the configured rules in different ways. The default rule is in memory
-
3. Adapt mainstream frameworks, including Servlet, Dubbo, rRpc, etc
The Dashboard console
Sentinel-dashboard is a standalone application that is powered by Spring-Boot and provides a lightweight console that provides machine discovery, real-time monitoring of stand-alone resources, cluster resource summary, and rule management.
We only need to configure the application to use these features.
1 Starting the Console
1.1 Download the code and compile the console
-
Download Console Engineering
-
Package the code into a fat JAR using the following command: MVN cleanPackage
1.2 start
Start the compiled console with the following command:
$ java -Dserver.port=8080 -Dcsp.sentinel.dashboard.server=localhost:8080 -jar target/sentinel-dashboard.jar
Copy the codePort =8080 is used to specify the Spring Boot port 8080.
2 Connect the client to the console
After the console starts, perform the following steps to connect the client to the console.
2.1 Importing the Client JAR Package
Importing jar packages via POM.xml:
<dependency>
<groupId>com.alibaba.csp</groupId>
<artifactId>sentinel-transport-simple-http</artifactId>
<version>x.y.z</version>
</dependency>
Copy the code2.2 Setting Startup Parameters
Started to join the JVM parameter – Dcsp. Sentinel. Dashboard. Server = consoleIp: specified console port address and port. If multiple applications are started, you need to run -dcsp.sentinel.api. port= XXXX to specify the port of the client monitoring API (8719 by default).
In addition to modifying JVM parameters, the same effect can be achieved through configuration files. For more information, see startup Configuration items.
2.3 Triggering Client Initialization
To ensure traffic to the client, Sentinel initializes the first time the client calls and starts sending heartbeat packets to the console.
Sentinel-dashboard is a standalone Web application that accepts connections from clients and communicates with them using HTTP. Their relationship is shown below:
dashboard
Dashboard starts will wait for the client connection, one of the specific practice is in MachineRegistryController receiveHeartBeat method, the client sends a heartbeat message, is this method through HTTP request.
After dashboard receives the heartbeat message from the client, it encapsulates the IP and port information transmitted by the client into a MachineInfo object. The object is then added to a ConcurrentHashMap via the addMachine method of the MachineDiscovery interface to save it.
This is a problem because the client information is stored in Dashboard memory, so when the Dashboard application restarts, the client information that has been sent before will be lost.
client
During startup, the client selects a CommandCenterInitFunc and only one CommandCenter.
CommandHandler implementation classes are scanned and obtained by spi before startup, and all CommandHandlers are registered in a HashMap for later use.
PS: Consider why CommandHandler does not need to be persisted and is stored directly in memory.
After registering CommandHandler, CommandCenter is launched, which has two implementation classes:
-
SimpleHttpCommandCenter starts a server with ServerSocket and accepts socket connections
-
NettyHttpCommandCenter starts a server over Netty that accepts channel connections
After CommandCenter is started, it waits for the Dashboard to send a message. After receiving a message, it processes the message through a specific CommandHandler and returns the result to the Dashboard.
Note here that Dashboard sends messages to clients via asynchronous httpClient, in the HttpHelper class.
But the weird thing is, since the message is sent asynchronously, and the CountDownLatch is used to wait for the message to return and then get the result, there is no sense of asynchron. The specific code is as follows:
private String httpGetContent(String url) {
final HttpGet httpGet = new HttpGet(url);
final CountDownLatch latch = new CountDownLatch(1);
final AtomicReference<String> reference = new AtomicReference<>();
httpclient.execute(httpGet, new FutureCallback<HttpResponse>() {
@Override
public void completed(final HttpResponse response) {
try {
reference.set(getBody(response));
} catch (Exception e) {
logger.info("httpGetContent " + url + " error:", e);
} finally {
latch.countDown();
}
}
@Override
public void failed(final Exception ex) {
latch.countDown();
logger.info("httpGetContent " + url + " failed:", ex);
}
@Override
public void cancelled() {
latch.countDown();
}
});
try {
latch.await(5, TimeUnit.SECONDS);
} catch (Exception e) {
logger.info("wait http client error:", e);
}
return reference.get();
}
Copy the codeAdaptation of mainstream frameworks
Sentinel also ADAPTS some mainstream frameworks so that sentinel can enjoy the protection even when using mainstream frameworks. Currently supported adapters include the following:
-
Web Servlet
-
Dubbo
-
Spring Boot / Spring Cloud
-
gRPC
-
Apache RocketMQ
In fact, the adaptation is done through the extension points of the major frameworks, and then the Sentinel limiting degradation code is added to the extension points. Take a look at the Servlet adaptation code, the specific code is:
public class CommonFilter implements Filter {
@Override
public void init(FilterConfig filterConfig) {
}
@Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
throws IOException, ServletException {
HttpServletRequest sRequest = (HttpServletRequest)request;
Entry entry = null;
try {
// Resources generated on request
String target = FilterUtil.filterTarget(sRequest);
target = WebCallbackManager.getUrlCleaner().clean(target);
// "Apply" for the resource
ContextUtil.enter(target);
entry = SphU.entry(target, EntryType.IN);
// If the resource can be applied successfully, the traffic is not restricted
// Will request release
chain.doFilter(request, response);
} catch (BlockException e) {
// Otherwise, if BlockException is caught, the request is restricted
// Redirects the request to a default page
HttpServletResponse sResponse = (HttpServletResponse)response;
WebCallbackManager.getUrlBlockHandler().blocked(sRequest, sResponse);
} catch (IOException e2) {
// Omit some code
} finally {
if (entry ! = null) {
entry.exit();
}
ContextUtil.exit();
}
}
@Override
public void destroy() {
}
}
Copy the codeThe Servlet Filter is extended to implement a Filter, and then the request is restricted in the doFilter method. If the request is restricted, the request is redirected to a default page, otherwise the request is released to the next Filter.
Rules are persistent and dynamic
The idea behind Sentinel is that developers only need to focus on the definition of the resource, and when the resource definition is successful, various flow control degradation rules can be dynamically added.
Sentinel provides two ways to change rules:
-
Modify directly via API (loadRules)
-
Modify data sources using a DataSource
You can use the following apis to modify different rules:
FlowRuleManager.loadRules(List<FlowRule> rules); // Modify the flow control rule
DegradeRuleManager.loadRules(List<DegradeRule> rules); // Modify the reversion rule
SystemRuleManager.loadRules(List<SystemRule> rules); // Modify system rules
Copy the codeThe DataSource extension
The loadRules() method above only accepts in-memory rule objects, but the in-memory rules are lost after the application restarts. More often, rules are better stored in files, databases, or configuration centers.
The DataSource interface gives us the ability to connect to any configuration source. Implementing the DataSource interface is a much more reliable way to modify rules directly through the API.
It is recommended that you set rules on the console and push the rules to a unified rule center. You only need to implement the DataSource interface to monitor the rule changes in the rule center and obtain the changed rules in real time.
DataSource extensions are commonly implemented in the following ways:
-
Pull mode: The client periodically polls the pull rules to a rule management center, such as SQL, files, or VCS. The way to do this is simple, but the disadvantage is that changes cannot be captured in a timely manner;
-
Push mode: the rule center pushes the rules uniformly, and the client monitors the changes constantly by registering listeners, such as using Nacos and Zookeeper configuration centers. This method has better real-time performance and consistency assurance.
So far, the basic situation of Sentinel has been analyzed, and you can continue to read the source code for more details.
I am Hou Yi, if the article is helpful to you, welcome you to like and pay attention, and welcome you to pay attention to my public account:
