Click for PDF manual V2.0 – > interview | springboot | springcloud | programming ideas

Recommended: www.xttblog.com/?p=5302 ##

preface

Let’s consider the solutions for the following business scenarios:

  • The payment system runs at 1:00 a.m. every day for one-day settlement, and the last month’s settlement will be carried out on the first day of every month
  • E-commerce shopping hour, commodity prices start at 8 o ‘clock discount
  • 12306 ticket purchase system, more than 30 minutes without successful payment order, recycling processing
  • After a product is successfully delivered, an SMS notification is sent to the customer

❝

There are many similar business scenarios. How do we solve them?

❞

Many business scenarios require us to do a task at a certain time, and timed tasks solve this business scenario. In general, the system can replace some of the scheduled tasks with messaging, which has many similarities and can be interchangeable scenarios.

For example, in the above business scenario of sending SMS notifications to customers after successful delivery, we can send MQ messages to the queue after successful delivery, then consume MQ messages and send SMS messages.

But it’s not interchangeable in certain situations:

  • Time-driven/event-driven: Internal systems can generally be time-driven, but when it comes to external systems, only time-driven can be used. For fear of taking the price of external websites, climb once every hour
  • Batch/line-by-line processing: Batch processing of stacked data is more efficient and has advantages over message-oriented middleware without the need for real time. And some business logic can only be processed in batches. Such as mobile monthly settlement of our phone bill
  • Real-time/non-real-time: Message-oriented middleware can process data in real time, but there are situations where real-time is not required, such as VIP upgrades
  • Intra-system/system decoupling: Scheduled task scheduling is typically within the system, while message-oriented middleware can be used between two systems

Java has a framework for scheduling tasks

stand-alone

  • Timer: indicates a timer class that can be configured for a specified scheduled task. The TimerTask class is a scheduled task class that implements the Runnable interface
  • ScheduledExecutorService: a relatively delayed or periodic task is scheduled. The disadvantage is that there is no absolute date or time
  • Spring timing framework: It is simple to configure and has many functions. If the system uses a single machine, Spring timers are preferred

distributed

  • Quartz: Java’s de facto timed task standard. But Quartz focuses on timed tasks rather than data, and there is no process that can be customized for data processing. Although Quartz can implement high availability of database-based jobs, it lacks distributed parallel scheduling capabilities
  • TBSchedule: Alibaba’s early open source distributed task scheduling system. The code is slightly older and uses timers instead of thread pools to perform task scheduling. Timers are known to be defective in handling exceptions. In addition, the TBSchedule job type is relatively simple and can only be a mode of data acquisition/processing. There is also a serious lack of documentation
  • Elastice-job: A flexible, distributed task scheduling system developed by Dangdang that uses ZooKeeper for distributed coordination, high availability, and sharding. Currently available in version 2.15, it supports cloud development.
  • Saturn: A distributed scheduling platform for scheduled tasks independently developed by Vipshop. Developed based on Dangdang’s Elastice-Job version 1, Saturn can be deployed to Docker containers.
  • Xxl-job: A distributed task scheduling platform released in 2015 by Xu Xueli, an employee of Dianping, is a lightweight distributed task scheduling framework. Its core design goal is rapid development, simple learning, lightweight and easy to expand.

Comparison of distributed task scheduling systems

Available system solutions for comparison are as follows: Elastic — Job (e-job) and XXX -Job (X-job)

Project background and community strength

X – Job:

  • Xu Xueli, an employee of Dianping, and 3 contributors;
  • Github has 2470Star, 1015fork;
  • 6 QQ discussion groups;
  • There are more than 40 companies registered in use;
  • The document is complete

E – the Job:

  • Dangdang open source, 17 contributors;
  • Github has 2524Star, 1015fork;
  • QQ discussion group 1, source discussion group 1;
  • There are more than 50 companies registered in use;
  • Complete documentation;
  • Have a clear development plan

Cluster Deployment

  • X-job: The only requirements for cluster deployment are as follows: Ensure that the db and login accounts of all cluster nodes are the same. The scheduling center uses DB configurations to differentiate clusters.

The actuator supports cluster deployment, improving scheduling system availability and task processing capability. The only requirements for cluster deployment are as follows: Ensure that the configuration items xxl.job.admin.addresses/ scheduling center addresses of each actuator in the cluster are the same, and that actuators perform automatic registration based on the configuration.

  • E-job: Rewrite Quartz’s database-based distributed functionality to implement a registry using Zookeeper

Job registry: Global job registry control center based on Zookeeper and its client Curator implementation. Used to register, control, and coordinate distributed job execution.

When multiple nodes are deployed, the task cannot be repeated

  • X-job: Uses Quartz’s database-based distributed capabilities
  • E-job: after a task is divided into N tasks, each server executes the assigned tasks. If a new server is added to the cluster or an existing server goes offline, extice-job triggers job re-sharding before the next job starts without changing the execution of this one.

Log traceability

  • X-job: supports log query
  • E-job: processes important events in the scheduling process by subscribing to events. E-job is used for query, statistics collection, and monitoring. Currently, Elastic-Job provides two event subscription methods for recording events based on a relational database.

Monitoring alarm

  • X-job: when scheduling fails, failure alarms are triggered, such as sending alarm emails.

Multiple email addresses can be configured. Use commas (,) to separate multiple email addresses

  • E-job: this mode can be implemented automatically by subscribing to events

Monitoring job running status, monitoring job server survival, monitoring recent data processing success, and data flow type jobs (You can judge whether the job traffic is normal by monitoring the number of recent data processing success. If it is less than the threshold for normal operation, you can choose to alarm.) And monitor the failure of recent data processing (the result of job processing can be judged by the number of recent data processing failures monitored. If the number is greater than 0, you can choose to alarm.)

Elastic capacity expansion and reduction

  • X-job: With Quartz’s database-based distributed capabilities, exceeding a certain number of servers puts a certain amount of strain on the database
  • E-job: Uses ZK to register, control, and coordinate services

Support parallel scheduling

  • X-job: Multiple threads in the scheduling system (10 threads by default) trigger the scheduling to ensure that the scheduling is executed accurately and not blocked.
  • E-job: implements task fragmentation. A task is divided into N independent task items, which are executed by distributed servers in parallel.

High availability policy

  • X-job: The scheduling center uses DB locks to ensure the consistency of distributed scheduling in the cluster. A task scheduling task triggers only one execution.
  • E-job: High availability of the Scheduler is achieved by running several instances of Elastice-job-cloud-Scheduler that point to the same ZooKeeper cluster. ZooKeeper is used to perform a leader election if the current master elastice-job-Cloud-Scheduler instance fails. The cluster is formed by at least two instances of the scheduler, in which only one instance of the scheduler provides services and the other instances are in “standby” state. When this instance fails, the cluster elects one of the remaining instances to continue providing services.

Failure Handling Strategy

  • X-job: indicates the processing policy for scheduling failures. The policies include failure alarm (default) and retry.
  • E-job: Indicates that elastic capacity expansion or reduction is performed before the next Job is run. However, during this Job execution, jobs assigned by offline servers will not be re-assigned. Failover can be performed in this job run with an idle server fetching orphan job fragments. Fail-over also compromises performance.

Dynamic Sharding strategy

  • X-job: Fragment broadcast tasks are fragmented based on actuators. Dynamic expansion of actuator clusters is supported to dynamically increase the number of fragments and coordinate service processing. It can significantly improve the task processing capability and speed when performing large data operations.

When the task routing policy is fragment broadcast, a task scheduling policy triggers all the actuators in the cluster to execute a task and transmits fragment parameters. Sharding tasks can be developed according to sharding parameters;

  • E-job: supports multiple sharding policies and can be customized

The default sharding policy supports three types of sharding policies: sharding policy based on the average assignment algorithm, sharding policy based on the IP address ascending or descending algorithm based on the hash value of the Job name, and sharding policy that rotates the Job instance list based on the hash value of the Job name. You can customize the sharding policy

Elastice-job sharding is implemented using ZooKeeper. Shards are allocated by the master node. The shard algorithm on the master node is triggered in the following three situations: a. New Job instances are added to the cluster. B.

Compare that to the Quartz framework

  1. Call the API in a way to operate tasks, not humanized;
  2. The need to persist the business QuartzJobBean into the underlying data table is quite intrusive.
  3. Scheduling logic and QuartzJobBean are coupled in the same project, which will lead to a problem. When the number of scheduling tasks gradually increases and the scheduling task logic gradually increases, the performance of the scheduling system will be greatly limited by services.
  4. Quartz focuses on timed tasks rather than data, and has no process that is customized for data processing. Although Quartz can implement high availability of database-based jobs, it lacks distributed parallel scheduling capabilities.

Comprehensive comparison

Technology selection of distributed timed task scheme

Summary and conclusion

Thing in common:

E-job and X-Job have a wide user base and complete technical documents, which meet the basic function requirements of scheduled tasks.

Difference:

X-job focuses on easy service implementation, easy management, easy learning cost, and rich failure policies and routing policies. It is recommended to use it when the user base is relatively small and the number of servers is within a certain range.

E-job focuses on data, adding flexible capacity expansion and data fragmentation to maximize the utilization of distributed server resources. However, the learning cost is high. You are advised to use this command when a large amount of data is required and a large number of servers are deployed.

Other plans with scheduled tasks

The system can automatically confirm receipt of goods when the goods have not been received for more than 10 days after delivery.

  • Every day at midnight screening the next day can automatically confirm the receipt of orders, and then the next day every 10 minutes to perform the confirmation of the receipt of goods is not too expensive, it is relatively accurate time
  • If the status of automatic confirmation of receipt is only for the client to see, the next time the user online, do a calculation can be.
  • Delayed and timed message delivery
  • ActiveMQ provides a broker side message scheduling mechanism. The broker does not want messages to be sent to consumers immediately, but 60 seconds later. The broker does not want messages to be sent to consumers immediately
  • RabbitMQ can set x-message-tt for Queue and Message to control the lifetime of messages. If a timeout occurs, the Message will become a dead letter. With DLX, when a message becomes dead letter in one queue, it can be republished to another Exchange. The message can then be consumed again.

Well, that’s it for today. Hope to help you!

Recommend 3 original Springboot +Vue projects, with complete video explanation and documentation and source code:

Build a complete project from Springboot+ ElasticSearch + Canal

  • Video tutorial: www.bilibili.com/video/BV1Jq…
  • A complete development documents: www.zhuawaba.com/post/124
  • Online demos: www.zhuawaba.com/dailyhub

【VueAdmin】 hand to hand teach you to develop SpringBoot+Jwt+Vue back-end separation management system

  • Full 800 – minute video tutorial: www.bilibili.com/video/BV1af…
  • Complete development document front end: www.zhuawaba.com/post/18
  • Full development documentation backend: www.zhuawaba.com/post/19
  • Online demos: www.markerhub.com/vueadmin

【VueBlog】 Based on SpringBoot+Vue development of the front and back end separation blog project complete teaching

  • Full 200 – minute video tutorial: www.bilibili.com/video/BV1PQ…
  • Full development documentation: www.zhuawaba.com/post/17
  • Online demos: www.markerhub.com:8084/blogs

If you have any questions, please come to my official account [Java Q&A Society] and ask me