Author: Huang Xiaomeng (Xueren)
Scheduled tasks are common requirements of every business, such as scanning overdue orders every minute, clearing historical database data every hour, collecting data of the previous day and generating reports every day, and so on.
Built-in solutions in Java
Use the Timer
Create a java.util.TimerTask to implement the business logic in the run method. Scheduling is performed through java.util.Timer, which supports fixed frequency execution. All TimerTasks are executed sequentially in the same thread, affecting each other. In other words, for multiple TimerTasks in the same Timer, if one TimerTask is in execution, other TimerTasks can only queue up and wait even if they reach the execution time. If an exception occurs, the thread exits and the entire scheduled task fails.
import java.util.Timer;
import java.util.TimerTask;
public class TestTimerTask {
public static void main(String[] args) {
TimerTask timerTask = new TimerTask() {
@Override
public void run() {
System.out.println("hell world"); }}; Timer timer = new Timer(); timer.schedule(timerTask, 10.3000); }}Copy the codeUsing ScheduledExecutorService
Timed task solution based on thread pool design, each scheduled task will be allocated to a thread in the thread pool to execute, solve the problem that Timer timers cannot be executed concurrently, support fixedRate and fixedDelay.
import java.util.Timer;
import java.util.TimerTask;
public class TestTimerTask {
public static void main(String[] args) {
TimerTask timerTask = new TimerTask() {
@Override
public void run() {
System.out.println("hell world"); }}; Timer timer = new Timer(); timer.schedule(timerTask, 10.3000); }}Copy the codeA built-in solution in Spring
Springboot provides a set of lightweight scheduled Task tools Spring Task, annotations can be easily configured, support cron expression, fixedRate, fixedDelay.
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
public class TestTimerTask {
public static void main(String[] args) {
ScheduledExecutorService ses = Executors.newScheduledThreadPool(5); // Execute according to fixed frequency, every interval5Run ses once every second.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
System.out.println("hello fixedRate"); }},0.5, TimeUnit.SECONDS); // Execute according to fixed delay3Ses seconds to run.scheduleWithFixedDelay(new Runnable() {
@Override
public void run() {
System.out.println("hello fixedDelay"); }},0.3, TimeUnit.SECONDS); }}Copy the codeThe biggest advantage Spring Task has over the two solutions mentioned above is its support for CRon expressions that can handle business execution on a fixed cycle of standard time, such as at what time of day.
Business idempotent solutions
Today’s applications are basically distributed, and the code on all machines is the same. The Java and Spring solutions described above are process-level, with each machine performing scheduled tasks at the same point in time. This may cause problems for scheduled tasks that require idempotent services. For example, messages are pushed to users for multiple times every month.
A natural solution for many applications is to use distributed locks. That is, before each scheduled task is executed, grab the lock first. If the lock is captured, the task will be executed. If the lock is not captured, the task will not be executed. How to grab the lock, is multifarious, such as the use of DB, Zookeeper, Redis.
Use DB or Zookeeper to snatch locks
The architecture of DB or Zookeeper is similar, and the principles are as follows:
- When the time is up, in the callback method, grab the lock first.
- If the lock is captured, the method continues to execute and returns directly if the lock is not captured.
- After executing the method, release the lock.
Example code is as follows:
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
@Component
@EnableScheduling
public class MyTask {
/** * run once at the 30th second of every minute */
@Scheduled(cron = "30 * * * *?")
public void task1() throws InterruptedException {
System.out.println("hello cron");
}
/** * run every 5 seconds */
@Scheduled(fixedRate = 5000)
public void task2() throws InterruptedException {
System.out.println("hello fixedRate");
}
/** * Run every 3 seconds after the last run */
@Scheduled(fixedDelay = 3000)
public void task3() throws InterruptedException {
System.out.println("hello fixedDelay"); }}Copy the codeThe current design, if you are more careful, may actually lead to repeated execution of the task. For example, when the task is executed very fast, machine A grabs the lock and releases the lock soon after completing the task. B: This machine grabs the lock, it still grabs the lock and performs the task again.
Use Redis to grab locks
Using Redis lock grab is similar to DB/ ZooKeeper in terms of architecture, but redis lock grab supports an expiration time without actively releasing the lock, and can make full use of the expiration time to solve the problem of repeated task execution caused by releasing the lock too fast. The architecture is as follows:
Example code is as follows:
@Component
@EnableScheduling
public class MyTask {
/** * run once at the 30th second of every minute */
@Scheduled(cron = "30 * * * *?")
public void task1() throws InterruptedException {
String lockName = "task1";
if (tryLock(lockName, 30)) {
System.out.println("hello cron");
releaseLock(lockName);
} else {
return;
}
}
private boolean tryLock(String lockName, long expiredTime) {
//TODO
return true;
}
private void releaseLock(String lockName) {
//TODO
}
}
Copy the codeSeeing this, some students may have questions. Is adding an expiration time still not rigorous enough, or may the task be repeated?
Yes, it is still possible to schedule a Task 30 seconds later if a machine is suddenly fullgc for a long time, or if the previous Task is not finished (Spring Task and ScheduledExecutorService are essentially thread pools for processing tasks).
The use of Quartz
Quartz[1] is a set of lightweight task scheduling framework that only needs to define Job (task), Trigger (Trigger) and Scheduler (Scheduler) to achieve a scheduled scheduling capability. Support database – based cluster mode, can achieve idempotent task execution.
Quartz supports idempotent execution of tasks. In theory, it is still possible to grab DB locks.
QRTZ_LOCKS is the QRTZ_LOCKS table for Quartz cluster synchronization. The table structure is as follows:
--QRTZ_LOCKS creates the table structureTABLE `QRTZ_LOCKS` (
`LOCK_NAME` varchar(40) NOT NULL, PRIMARY KEY (`LOCK_NAME`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; - QRTZ_LOCKS record + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | LOCK_NAME | + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | CALENDAR_ACCESS | | JOB_ACCESS | | MISFIRE_ACCESS | | STATE_ACCESS | | TRIGGER_ACCESS | +-----------------+Copy the codeYou can see that QRTZ_LOCKS has five records representing five locks that are used to synchronize Job, Trigger, and Calendar access from multiple Quartz nodes.
Open source task scheduling middleware
All the solutions mentioned above have a problem in architecture, that is, lock snatching is required for each scheduling. Especially when DB and Zookeeper lock snatching is used, the performance will be poor. Once the number of tasks increases to a certain amount, there will be obvious scheduling delay. Another pain point is when the business wants to change the scheduling configuration, or add a task, it has to change the code and redistribute the application.
Therefore, a bunch of task scheduling middleware emerged in the open source community to create, modify and schedule tasks through the task scheduling system, among which xxl-Job and ElasticJob are the most popular in China.
ElasticJob
ElasticJob[2] is a lightweight, decentralized distributed task scheduling framework developed based on Quartz and relying on Zookeeper as a registry. It has been open source through Apache.
ElasticJob’s biggest functional difference from Quartz is that it supports sharding and can distribute a task sharding parameter to different machines for execution. The biggest difference in architecture is that Zookeeper is used as the registry, and different tasks are assigned to different nodes for scheduling, which does not need to be triggered by snatch lock. The performance is much stronger than Quartz. The architecture diagram is as follows:
Development is also relatively simple, and springBoot is better combined, you can define tasks in the configuration file as follows:
elasticjob:
regCenter:
serverLists: localhost:2181
namespace: elasticjob-lite-springboot
jobs:
simpleJob:
elasticJobClass: org.apache.shardingsphere.elasticjob.lite.example.job.SpringBootSimpleJob
cron: 0/5* * * *? timeZone: GMT+08:00
shardingTotalCount: 3
shardingItemParameters: 0=Beijing,1=Shanghai,2=Guangzhou
scriptJob:
elasticJobType: SCRIPT
cron: 0/10* * * *? shardingTotalCount:3
props:
script.command.line: "echo SCRIPT Job: "
manualScriptJob:
elasticJobType: SCRIPT
jobBootstrapBeanName: manualScriptJobBean
shardingTotalCount: 9
props:
script.command.line: "echo Manual SCRIPT Job: "
Copy the codeThe task interface is as follows:
@Component
public class SpringBootShardingJob implements SimpleJob {
@Override
public void execute(ShardingContext context) {
System.out.println(" Total number of fragments ="+context.getShardingTotalCount() + ", shard number ="+context.getShardingItem() + ", fragment argument ="+context.getShardingParameter());
}
Copy the codeThe running results are as follows:
Total number of fragments =3, the fragment number =0, fragment parameter =Beijing Total number of fragments =3, the fragment number =1, shard parameter =Shanghai Shard total =3, the fragment number =2, the fragment parameter =GuangzhouCopy the codeElasticJob also provides a simple UI that allows you to view the task list and modify, trigger, stop, take effect, or fail the task.
Unfortunately, ElasticJob does not support dynamic task creation.
XXL-JOB
Xxl-job [3] is a lightweight distributed task scheduling system out of the box. Its core design goal is rapid development, simple learning, lightweight and easy to expand, which is widely popular in the open source community.
Xxl-job is a master-slave architecture in which the Master is responsible for task scheduling and the Slave is responsible for task execution. The architecture is shown as follows:
The @xxlJob annotation is used to define JobHandler instead of ElasticJob.
@Component
public class SampleXxlJob {
private static Logger logger = LoggerFactory.getLogger(SampleXxlJob.class);
/** * 1, simple task example (Bean mode) */
@XxlJob("demoJobHandler")
public ReturnT<String> demoJobHandler(String param) throws Exception {
XxlJobLogger.log("XXL-JOB, Hello World.");
for (int i = 0; i < 5; i++) {
XxlJobLogger.log("beat at:" + i); TimeUnit.SECONDS.sleep(2); } return ReturnT.SUCCESS; } /** * 1; /** * 2;shardingJobHandler") public ReturnT
shardingJobHandler(String param) throws Exception {// Sharding parameter shardingutil. ShardingVO ShardingVO = ShardingUtil.getShardingVo(); XxlJobLogger.log("
Fragment parameters: Current fragment number = {}, total number of slices = {}", shardingVO.getIndex(), shardingVO.getTotal()); Business logic for (inti = 0; i < shardingVO.getTotal(a);i++) {
if (i == shardingVO.getIndex()) {
XxlJobLogger.log(" {} slice, hit fragment start processing ",i);
} else {
XxlJobLogger.log(" {} piece, ignore ",i);
}
}
return ReturnT.SUCCESS; }}Copy the codeXxl-job provides more functions than ElasticJob. Xxl-job supports dynamic task creation, log scheduling, and report running on the console.
Xxl-job history records, running reports, and scheduling logs are implemented based on the database:
It can be seen that all functions of XXL-job depend on the database, and the scheduling center does not support the distributed architecture. In the case of a large number of tasks and schedules, performance bottlenecks will occur. However, xxL-Job can meet the requirements of scheduled tasks if the task size, high availability, monitoring, alarm, and visualization are not too high.
Enterprise-level solutions
Open source software can only provide basic scheduling capabilities and is generally weak in regulatory control. For logging services, the industry tends to use ELK solutions; SMS alarm, need to have SMS platform; To monitor the market, the dominant solution is Prometheus; And so on. To have these capabilities requires not only additional development costs but also expensive resource costs.
In addition, the use of open source software also comes with the risk of stability, that is, if there is a problem, no one can deal with it, and they want to report back to the community to deal with it, the link is too long, and the failure has already occurred.
Ali Cloud Task scheduling SchedulerX[4] is a one-stop task scheduling platform developed by Alibaba based on Akka architecture, compatible with open source XXL-Job, ElasticJob and Quartz (under planning), and supports Cron timing, one-time task, task scheduling and distributed batch running. With high availability, visualization, operable maintenance, low delay and other capabilities, with enterprise monitoring, log services, SMS alarm and other services.
advantage
Safety protection
• Multi-level security protection: Supports HTTPS and VPC access, and aliyun provides multi-level security protection against malicious attacks. • Multi-tenant isolation: Supports isolation at multiple regions, namespaces, and application levels. • Permission control: Manages console read and write permissions and authenticates client access.
Employing high availability, enterprise high availability SchedulerX2.0 task more backup mechanism, through alibaba for many years a double tenth disaster, drills, can do any room hung up, task scheduling will not be affected.
Commercial grade alarm operation and maintenance• Alarm: support email, nail, SMS, telephone, (other alarm methods are in the planning). Support task failure, timeout, no machine alarm. The reason for the failure of the task can be directly seen from the alarm content. Take the nail robot as an example.
• Operation and maintenance operations: re-run in place, re-brush data, mark success, view stack, stop tasks, specify machines, etc.
Rich Visualization SchedulerX has rich visualization capabilities, such as:
• User market
• View historical task execution records
• View task run logs
• View the task run stack
• View task operation records
Open Source Schedulerx Compatible with open source XXL-Job, ElasticJob, Quartz (under planning), services do not need to change a line of code, that is, tasks can be hosted on the Schedulerx scheduling platform, enjoying enterprise-level visualization and alarm capabilities.
Spring’s native SchedulerX supports dynamic task creation through the console and API, as well as Spring’s declarative task definition. A task configuration can be launched in any environment with one click, as follows:
Spring: Schedulerx2: endpoint: acm.aliyun.com433d8b23-06e9-xxxx-xxxx-90AppName: myTest groupId: mytest. group # Create a unique appKey in the same namespace: myTest123@alibaba # regionId: public # regionId aliyunAccessKey: AliyunSecretKey: XXXXXXX # aliyunSecretKey: XXXXXXX # aliyunSecretKey: SK alarmChannel: SMS, DING # alarmChannel: SMS and pin jobs: simpleJob: jobModel: standalone className: com.aliyun.schedulerx.example.processor.SimpleJob cron:0/30* * * *? # cron jobParameter: hello overwrite: true shardingJob: jobModel: sharding className: ccom.aliyun.schedulerx.example.processor.ShardingJob oneTime:2022-06-02 12:00:00# jobParameter:0=Beijing,1=Shanghai,2=Guangzhou Overwrite: true broadcastJob: com.aliyun.schedulerx.example.processor.BroadcastJob jobParameter: hello overwrite: true mapReduceJob: jobModel: mapreduce className: com.aliyun.schedulerx.example.processor.MapReduceJob cron:0* * * *? jobParameter:100Overwrite: true alarmUsers: # Alarm contact user1: userName: zhang 3 userPhone:12345678900User2: userName: li si ding: https://oapi.dingtalk.com/robot/send?access_token=xxxxxCopy the codeDistributed batch running
SchedulerX provides a rich distributed model that can handle a wide variety of distributed business scenarios. Including single machine, broadcast, sharding, MapReduce[5], etc., architecture is as follows:
SchedulerX’s MapReduce model can distribute massive tasks to multiple machines with just a few lines of code. Compared with big data batch, SchedulerX’s MapReduce model has the characteristics of fast speed, data security, low cost and simplicity to learn.
Task scheduling
SchedulerX arranges tasks through workflows and provides a visual interface that is easy to configure by dragging and dropping a workflow. Detailed task status diagram can clearly see why downstream tasks do not run, which is convenient to locate problems.
Preemptable task priority queue
A common scenario is offline report services at night. For example, many report tasks start at 1 or 2 o ‘clock in the evening. The maximum number of concurrent tasks must be controlled (otherwise, the service cannot handle the tasks). At the same time, THE KPI report must be published before 9 am. You can set the KPI task with a higher priority to preempt tasks with a lower priority.
SchedulerX supports preemptable task priority queues that can be dynamically configured on the console:
Q&A
- Can Kubernetes applications access SchedulerX?
SchedulerX can be connected to physical machines, containers, and Kubernetes Pods. 2. My application is not on Ali Cloud, can I use SchedulerX? SchedulerX can be connected to any cloud platform or local machine that has access to the Internet.
Release the latest information of cloud native technology, collect the most complete content of cloud native technology, hold cloud native activities and live broadcast regularly, and release ali products and user best practices. Explore the cloud native technology with you and share the cloud native content you need.
Pay attention to [Alibaba Cloud native] public account, get more cloud native real-time information!