XXL-JOB is a distributed task scheduling framework, the core design goal is to develop quickly, learning simple, lightweight, easy to expand. Is now open source and access to a number of companies online product line, download and use it now.
- 1.Simple: support through the Web page on the task CRUD operation, simple operation, a minute to get started;
- 2.Dynamic: support dynamic modification of task status, pause / resume tasks, and termination of running tasks,immediate effect;
- 3.Dispatch center HA (center type): Dispatch with central design, "dispatch center" based on the cluster of Quartz implementation, can guarantee the scheduling - center HA;
- 6.Consistency: "Dispatch Center" through the DB lock to ensure the consistency of cluster distributed scheduling,one task excuted for once;
- 7.Custom task parameters: support online configuration scheduling tasks into the parameters, immediate effect;
- 8.Scheduling thread pool: scheduling system multi-threaded trigger scheduling operation, to ensure accurate scheduling, not blocked;
- 9.Elastic expansion capacity: once the new executor machine on the line or off the assembly line, the next time scheduling will be re-assigned tasks;
- 10.Mail alarm: the task fails to support e-mail alarm, support configuring multiple email addresses to send bulk alert messages;
- 11.Status monitoring: support real-time monitoring of the progress of the task;
- 12.Rolling execution log: support online view scheduling results, and support Rolling real-time view of the executer output of the complete implementation of the log;
- 13.GLUE: provide Web IDE, support online development task logic code, dynamic release, real-time compiler effective, omit the deployment of the on-line process. Supports historical versions of 30 versions back;
- 14.Data Encryption: The communication between the dispatching center and the executor is used for data encryption, Enhancing the security of dispatching information;
- 15.Task Dependency: Support configuration subtask dependencies, When the parent task executed end and after the success of the implementation will take the initiative to trigger a second task execution, multiple sub tasks are separated by commas;
- 16.Push the Maven central warehouse: The latest stable version will be sent to the Maven central warehouse to facilitate user access and use;
- 17.Task registration: The executor automatically registers tasks periodically, and the dispatch center automatically finds the registered tasks and triggers execution. It also supports manual input of executor address;
- 18.Router strategy: A rich routing strategy is provided when the executor cluster is deployed, these include: first, last, poll, random, consistent HASH, least frequently used, least recently used, failover, busy over, sharding broadcast,etc.;
- 19.Report monitor: Support real-time view of running data, such as the number of tasks, the number of dispatch, the number of executors, etc .; and scheduling reports, such as scheduling date distribution, scheduling success map;
- 20.Script task: Support the development and operation of script tasks in GLUE mode, including shell, Python and other types of script;
- 21.Blocking handling strategy: The scheduling is too dense and the executor is too late to handle. The strategy includes: single machine serial (default), discarding the following scheduling, and Override the previous scheduling;
- 22.Failure handling strategy:Handling strategy when scheduling fails, the strategy includes: failure alarm (default), failure retry;
- 23.Sharding broadcast task: When an executor cluster is deployed, task routing strategy select "sharding broadcast", a task schedule will broadcast all the actuators in the cluster to perform it once, you can develop sharding tasks based on sharding parameters;
- 24.Dynamic sharding: The sharding broadcast task is sharded by the executors to support the dynamic expansion of the executor cluster to dynamically increase the number of shardings and cooperate with the business handle; In the large amount of data operations can significantly improve the task processing capacity and speed.
- 25、Event trigger：In addition to "Cron" and "Task Dependency" to trigger tasks, support event-based triggering tasks. The dispatch center provides API service that triggers a single execution of the task, it can be triggered flexibly according to business events.
In 2015, I created the XXL-JOB project repository on github and submitted the first commit, followed by the system structure design, UI selection, interactive design ...
In 2015 - November, XXL-JOB finally RELEASE the first big version of V1.0, then I will be released to OSCHINA, XXL-JOB OSCHINA won the popular recommendation of @红薯, the same period reached OSCHINA's " Popular move "ranked first and git.oschina open source software monthly heat ranked first, especially thanks for @红薯, thank you for the attention and support.
In 2015 - December, I will XXL-JOB published to our internal knowledge base, and get internal colleagues recognized.
In 2016 - 01 months, my company started XXL-JOB internal access and custom work, in this thank Yuan and Yin two colleagues contribution, but also to thank the internal other attention and support colleagues.
In 2017-05-13, the link of "let the code run" in "[the 62nd source of open source China Genesis](https://www.oschina.net/event/2236961)" held in Shanghai,, I stepped on and made a speech about the XXL-JOB, five hundred spectators in the audience reacted enthusiastically ([pictorial review](https://www.oschina.net/question/2686220_2242120)).
> Our company have access to XXL-JOB, internal alias "Ferrari" (Ferrari based on XXL-JOB V1.1 version customization, new access application recommended to upgrade the latest version).
According to the latest statistics, from 2016-01-21 to 2017-07-07 period, the system has been scheduled about 600,000 times, outstanding performance. New access applications recommend the latest version, because after several major updates, the system's task model, UI interaction model and the underlying scheduling communication model has a greater optimization and upgrading, the core function more stable and efficient.
So far, XXL-JOB has access to a number of companies online product line, access to scenes such as electronic commerce, O2O business and large data operations, as of 2016-07-19, XXL-JOB has access to the company But not limited to:
> The company that access and use this product is welcome to register at the [address](https://github.com/xuxueli/xxl-job/issues/1 ), only for product promotion.
Please download project source code，get db scripts and execute, it will generate 16 tables if succeed.
The relative path of db scripts is as follows:
/xxl-job/doc/db/tables_xxl_job.sql
The xxl-job-admin can be deployed as a cluster,all nodes of the cluster must connect to the same mysql instance.
If mysql instances is deployed in master-slave mode,all nodes of the cluster must connect to master instace.
### 2.2 Compile
Source code is organized by maven,unzip it and structure is as follows:
xxl-job-admin：schedule admin center
xxl-job-core：public common dependent library
xxl-job-executor：executor Sample(Select appropriate version of executor,Can be used directly,You can also refer to it and transform existing projects into executors）
：xxl-job-executor-sample-spring：Spring version，executors managed by Spring，general and recommend;
：xxl-job-executor-sample-springboot：Springboot version，executors managed by Springboot;
：xxl-job-executor-sample-jfinal：JFinal version，executors managed by JFinal;
### 2.3 Configure and delploy "Schedule Center"
schedule center project:xxl-job-admin
target:Centralized management、Schedule and trigger task
#### Step 1:Configure Schedule Center
Configure file’s path of schedule center is as follows:
If you has finished step 1,then you can compile the project in maven and deploy the war package to tomcat.
the url to visit is :http://localhost:8080/xxl-job-admin (this address will be used by executor and use it as callback url),the index page after login in is as follow
xxl-job-admin can be deployed as a cluster to improve system availability.
Prerequisites for cluster is to keep all node configuration(db and login account info) consistent with each other. Different xxl-job-admin cluster distinguish with each other by db configuration.
xxl-job-admin can be visited through nginx proxy and configure a domain for nginx,and the domain url can be configured as the executor’s callback url.
### 2.4 Configur and Deploy "xxl-job-executor-example"
Executor Project:xxl-job-executor-example (if you want to create new executor project you can refer this demo);
Target:receive xxl-job-admin’s schedule command and execute it;
#### Step 1:import maven dependence
Pleast confirm import xxl-job-core jar in pom.xml;
#### Step 2:Executor Configuration
Relative path of the executor configuration file is as follows:
The concret content of configuration file as follows:
### xxl-job admin address list：xxl-job-admin address list: Multiple addresses are separated by commas,this address is used for "heart beat and register" and "task execution result callback" between the executor and xxl-job-admin.
### xxl.job.executor.appname is used to group by executors
xxl.job.executor.appname=xxl-job-executor-sample
### xxl.job.executor.ip :1,used to register with xxl-job-admin;2,xxl-job-admin dispatch task to executor through it;3,if it is blank executor will get ip automatically, multi network card need to be configured.
xxl.job.executor.ip=
### xxl.job.executor.port :the port of the executor runned by,if multiple executor instance run on the same computer the port must different with each other
xxl.job.executor.port=9999
### xxl-job log path：runtime log path of the job instance
You can compile and package the project If have done all the steps above successfully,the project supply two executor demo projects,you can choose any one to deploy:
xxl-job-executor-sample-spring:compile and package in WAR,can be deployed to tomcat;
xxl-job-executor-sample-springboot:compile and package in JAR,and run in springboot mode;
Now you have deployed the executor project.
#### Step 5:executor cluster(optional)
In order to improve system availability and job process capacity,executor project can be deployed as cluster.
Prerequisites:keep all node’s configuration item "xxl.job.admin.addresses" exactly the same with each other,all executors can be register automatically.
### 2.5 Start first job "Hello World"
Now let’s create a "GLUE模式(Java)" job,if you want to learn more about it , please see “chapter 3：Task details”。( "GLUE模式(Java)"'s code is maintained online through xxl-job-admin,compare with "Bean模式任务" it’s not need to develop, deploy the code on the executor and it’s not need to restart the executor, so it’s lightweight）
#### Prerequisites:please confirm xxl-job-admin and executor project has been deployed successfully.
#### Step 1:Create new job
Login in xxl-job-admin,click on the"新建任务" button, configure the job params as follows and click "保存" button to save the job info.
Click “GLUE” button on the right of the job to go to GLUE editor view as shown below。“GLUE模式(Java)” mode task has been inited with default task code for printing Hello World。 （ “GLUE模式(Java)” mode task is a java code fragment implements IJobHandler interface,it will be executed in executor,you can use @Resource/@Autowire to inject other java bean instance,if you want to see more info please go to chapter 3）
If you want to run the job manually please click "执行" button on the right of the job(usually we trigger job by Cron expression)
#### Step 4:view log
Click “日志” button on the right side of the task you will go to the task log list ,you will see the schedule history records of the task and the schedule detail info,execution info and execution params.If you click the “执行日志” button on the right side of the task log record,you will go to log console and view the execute log in the course of task execution.
On the log console,you can view task execution log on the executor immediately after it dump to log file,so you can monitor the task execution process by Rolling way.
- 执行器：the container where job executed in,it will be discovered automaticly if it has registered success when job was scheduled,and the job will be executed automaticly through this way.On the other side all tasks was grouped by this way.Tasks must be binded to a executor and it can be configured on "执行器管理" page;
- 描述：the decription of task
- 路由策略：when executors deployed as a cluster,it can configure multi route policys,include:
FIRST（第一个）：default select the first executor;
LAST（最后一个）：default select the last executor;
ROUND（轮询）：round select the executor;；
RANDOM（随机）：random select the executor;
CONSISTENT_HASH（一致性HASH）：all jobs was evenly scheduled on different machines,make sure load balance of executors under the same group and the same job will be scheduled to the same machine.
LEAST_FREQUENTLY_USED（最不经常使用）：default select the least often used executor.
LEAST_RECENTLY_USED（最近最久未使用）：defalut select the longest not used executor.
FAILOVER（故障转移）：beat with the executor in order and select the first beat success executor as target executor.
BUSYOVER（忙碌转移）：check the executor busy or not in order,the first executor checked not busy is to be select as the target scheduled executor.
SHARDING_BROADCAST(分片广播)：broadcast all executor nodes under the same executor group execute the job, slice number will be transferred at the same time,shard task will be executed accordate with the shard number.
- Cron：Cron expression used to trigger job execution;
- 运行模式：
BEAN模式：job was maintained on the side of executor by as JobHandler instance,it will be executed accordate with "JobHandler" properties.
GLUE模式(Java)：task source code is maintened in the schedule center,it must implement IJobHandler and explain by "groovy" in the executor instance,inject other bean instace by annotation @Resource/@Autowire.
GLUE模式(Shell)：it’s source code is a shell script and maintained in the schedule center.
GLUE模式(Python)：it’s source code is a python script and maintained in the schedule center.
- 子任务Key：every task has a unique key (task Key can acquire from task list)，when main task is done successfully it’s child task stand for by this key will be scheduled.
- 阻塞处理策略：the stategy handle the task when this task is scheduled too frequently and the task is block to wait for cpu time.
单机串行（默认）：task schedule request go into the FIFO queue and execute serially.
丢弃后续调度：the schedule request will be discarded and marked as fail when the same task’s instance scheduled befor is running in the target executor.
覆盖之前调度：the schedule request will be executed and clear before task queue when the same task’s instance scheduled befor is running in the target executor.
- 失败处理策略:handle policy for schedule fail
失败告警（默认）：it will trigger alarm such as send alarm mail when it’s scheduled fail.
失败重试：it will try another time when it’s scheduled fai,if try fail it will trigger alarm for fail.every time it will trigger a new schedule request.
- 执行参数：the params needed in the run time of the task, multiple values are separated by commas,it will be passed to task instace as an array when task is scheduled.
- 报警邮件：the email used to receive the alarm mail when task is scheduled fail or execute fail, multiple values are separated by commas.
- 负责人：The person name response for the task.
### 3.1 BEAN模式
The task logic exist in the executor project as JobHandler,the develop steps as shown below:
#### Step 1:develp obHandler in the executor project
- 1, create new java class implent com.xxl.job.core.handler.IJobHandler;
- 2, if you add @Component annotation on the top of the class name it’s will be managed as a bean instance by spring container;
- 3, add “@JobHandler(value=" customize jobhandler name")” annotation，the value stand for JobHandler name,it will be used as JobHandler property when create a new task in the schedule center.
If you want learn more about configure item please go and sedd “Description of configuration item”，select "BEAN模式" as run mode，property JobHandler please fill in the value defined by @JobHande.
Task source code is maintained in the schedule center and can be updated by Web IDE online, it will be compiled and effective real-time,didn’t need to assign JobHandler,develop flow shown as below:
#### Step 1:create task in schedule center
If you want learn more about configure item please go and sedd “Description of configuration item”，select "GLUE模式(Java)" as run mode.
Select the task record and click “GLUE” button on the righe of it,it will go to GLUE task’s WEB IDE page,on this page yo can edit you task code(also can edit in other IDE tools,copy and paste into this page).
Version backtrack（support 30 versions while backtrack）：on the WEB IDE page of GLUE task,on upper right corner drop down box please select “版本回溯”,it will display GLUE updated history,select the version you want it will display the source code of this version,it will backtrace the version while click save button.
If you want learn more about configure item please go and sedd “Description of configuration item”，select "GLUE模式(Shell)"as run mode.
#### Step 2:develop task source code
Select the task record and click “GLUE” button on the righe of it,it will go to GLUE task’s WEB IDE page,on this page yo can edit you task code(also can edit in other IDE tools,copy and paste into this page).
If you want learn more about configure item please go and sedd “Description of configuration item”，select "GLUE模式(Python)"as run mode.
#### Step 2:develop task source code
Select the task record and click “GLUE” button on the righe of it,it will go to GLUE task’s WEB IDE page,on this page yo can edit you task code(also can edit in other IDE tools,copy and paste into this page).
1,"调度中心OnLine”:display schedule center machine list,when task is scheduled it will callback schedule center for notify the execution result in failover mode, so that it can avoid a single point scheduler;
2,"执行器列表" :display all nodes under this executor group.
If you want to create a new executor,please click "+新增执行器" button:
Go to task management list page,click “新增任务” button on the upper right corner，on the pop-up window“新增任务”page configure task property and save.learn more info please go and see "3,task details".
Go to task management list page and choose the task you want to edit ,click”编辑”button on the right side of the task,on the pop-up window “编辑任务”page edit task property and save.
choose the task you want to edit and click” GLUE”button on the right side of the task, it will go to the Web IDE page of GLUE task,then you can edit task source code on this page.you can read "3.2 GLUE模式(Java)" for more info.
You can pause or recover task but it just fit to follow up schedule trigger and won’t affect scheduled tasks,if you want to stop tasks which has been triggered,please go and see “4.8 stop the running task”
You can view task’s history schedule log by click “日志” button,on the history schedule log list page you can view every time of task’s schedule result,execution result and so on,click “执行日志” button can view the task’s full execute log.
Click the “执行日志” button on the right side of the record,you can go to the execution log page,you can view the full execution log of the logic business code, shown as below:
Just fit to running tasks,on the task log list page,click “终止任务” button on the right side of the record, it will send stop command to the executor where the task was executed,finally the task was killed and the task instance execute queue of this task will be clear.
It is implemented by interrupt execute thread, it will trigger InterruptedException.so if JobHandler catch this execuption and handle this exception this function is unavailable.
So if you want stop the running task ,the JobHandler need to handle InterruptedException separately by throw this exception.the right logic is as shown below:
On the task log list page, after you select executor and task, you can click"删除" button on the right side and it will pop-up "日志清理" window,on the pop-up window you can choose different log delete policy,choose the policy you want to execute and click "确定" button it will delele relative logs:
- XXL_JOB_QRTZ_TRIGGER_GROUP:executor basic table, maintain the info about the executor;
- XXL_JOB_QRTZ_TRIGGER_REGISTRY:executor register table, maintain addressed of online executors and schedule center machines.
- XXL_JOB_QRTZ_TRIGGER_INFO:schedule extend table,it is used to save XXL-JOB schedule extended info,such as task group,task name,machine address,executor,input params of task and alarm email and so on.
- XXL_JOB_QRTZ_TRIGGER_LOG:schedule log table,it is used to save XXL-JOB task’s histry schedule info,such as :schedule result,execution result,input param of scheduled task,scheduled machine and executor and so on.
- XXL_JOB_QRTZ_TRIGGER_LOGGLUE:schedule log table,it is used to save XXL-JOB task’s histry schedule info,such as :schedule result,execution result,input param of scheduled task,scheduled machine and executor and so on.
All schedule behavior has been abstracted into “schedule center” common platform , it dosen’t include business logic and just responsible for starting schedule requests.
All tasks was abstracted into separate JobHandler and was managed by executors, executor is responsible for receiving schedule request and execute the relative JobHandler business.
So schedule and task can be decoupled from each other, by the way it can improve the overall stability and scalability of the system.
it is responsible for manage schedule info,send schedule request accord task configuration and it is not include an business code.schedule system decouple with the task, improve the overall stability and scalability of the system, at the same time schedule system performance is no longer limited to task modules.
Support visualization, simple and dynamic management schedule information, include create,update,delete, GLUE develop and task alarm and so on, All of the above operations will take effect in real time，support monitor schedule result and execution log and executor failover.
it is responsible for receive schedule request and execute task logic,task module focuses on the execution of the task, Development and maintenance is simpler and more efficient.
Receive execution request, end request and log request from schedule center.
Quartz is a good open source project and was often as the first choice for job schedule.Tasks was managed by api in quartz cluster so it can avoid some disadvantages of single quartz instance,but it also has some disadvantage as shown below:
- problem 1:it is not humane while operate task by call apill.
- problem 2:it is need to store business QuartzJobBean into database, System Invasion is quite serious.
- problem 3:schedule logic and couple with QuartzJobBean in the same project,it will lead a problem in case that if schedule tasks gradually increased and task logic gradually increased,under this situation the performance of the schedule system will be greatly limited by business.
Under Quartz develop,task logic often was maintained by QuartzJobBean, couple is very serious.in XXL-JOB"Schedule module" and "task module" are completely decoupled,all scheduled tasks in schedule module use the same QuartzJobBean called RemoteHttpJobBean.the params of the tasks was maintained in the extended tables,when trigger RemoteHttpJobBean,it will parse different params and start remote cal l and it wil call relative remote executor.
This call module is like RPC,RemoteHttpJobBean provide call proxy functionality,the executor is provided as remote service.
It is based on Quartz cluster，databse use Mysql；while QUARTZ task schedule is used in Clustered Distributed Concurrent Environment,all nodes will report task info and store into database.it will fetch trigger from database while execute task,if trigger name and execute time is the same only one node will execute the task.
business logic was executed on remote executor in XXL-JOB,schedule center just start one schedule request at every schedule time,executor will inqueue the request and response schedule center immediately. There is a huge difference from run business logic in quartz’s QuartzJobBean directly，just as Elephants and feathers；
the logic of task in XXL-JOB schedule center is very light and single job average run time alaways under 100ms,（most is network time consume）.so it can use limited threads to support a large mount of job run concurrently, 10 threads configured above can support at least 100 JOB normal execution.
This annotation is not used default by the schedule center of XXL-JOB schedule module, it use concurrent policy default,because RemoteHttpJobBean is common QuartzJobBean,so it greatly improve the capacity of schedule system and decrease the blocked chance of schedule module in the case of multi-threaded schedule.
Every schedule module was scheduled and executed parallel in XXL-JOB,but tasks in executor is executed serially and support stop task.
The handle policy when miss the job’s trigger time.
he reason may be:restart service,schedule thread was blocked by QuartzJobBean, threads was exhausted,some task enable @DisallowConcurrentExecution，the last schedule was blocked and next schedule was missed.
The default value of misfire in quartz.properties as shown below, unit in milliseconds:
When schedule center of the schedule module was deployed as web service, on one side it play as schedule center, on the other side it also provide api service for executor.
If executor project was deployed as cluster schedule center will known all online executor nodes,such as:“127.0.0.1:9997, 127.0.0.1:9998, 127.0.0.1:9999”.
When "路由策略" select "故障转移(FAILOVER)",it will send heart beat check request in order while schedule center start schedule request. The first alive checked executor node will be selected and send schedule request to it.
“调度备注” can be viewed on the monitor page when schedule success. As shown below:
“调度备注” will display local schedule route path、executor’s "注册方式"、"地址列表" and task’s "路由策略"。Under "故障转移(FAILOVER)" policy, schedule center take first address to do heartbeat detection, heat beat fail will automatically skip, the second address heart beat fail…… until the third address “127.0.0.1:9999” heart beat success, it was selected as target executor, then send schedule request to target executor, now the schedule process is end wait for the executor’s callback execution result.
- 任务信息:include executor address、JobHandler and executor params，accord these parameters it can locate specific machine and task code that the task will be executed.
- 调度信息:include schedule time、schedule result and schedule log and so on，accord these parameters you can understand some task schedule info of schedule center.
- 执行信息:include execute time、execute result and execute log and so on, accord these parameters you can understand the task execution info in the executor.
principle:every task has a task key in XXL-JOB, every task can configure property “child task Key”,it can match task dependency relationship through task key.
When parent task end execute and success, it will match child task dependency accord child task key, it will trigger child task execute once if it matched child task.
On the task log page ,you can see matched child task and triggered child task’s log info when you “查看”button of “执行备注”,otherwise the child task didin’t execute, as shown beleow:
principle: every Bean mode task is a Spring Bean instance and it is maintained in executor project’s Spring container. task class nedd to add “@JobHandler(value="name")” annotation, because executor identify task bean instance in spring container through annotation. Task class nedd to implements interface IJobHandler, task logic code in method execute(), the task logic in execute() method will be executed when executor received a schedule request from schedule center.
Principle : every "GLUE模式(Java)" task code is a class implemets interface IJobHandler, when executor received schedule request from schedule center these code will be loaded by Groovy classloader and instantiate into a Java object and inject spring bean service declared in this code at the same time（please confirm service and class reference in Glue code exist in executor project）, then call the object’s execute() method and execute task logic.
principle:the source code of script task is maintained in schedule center and script logic will be executed in executor. when script task was triggered, executor will load script source code and generate a script file on the machine where executor was deployed, the script will be called by java code, the script output log will be written to the task log file in real time so that we can monitor script execution in real time through schedule center, the return code 0 stands for success other for fail.
Executor will identify Bean mode task in spring container through @JobHandler When project start, it will be managed use the value of annotation as key.
When executor received schedule request from schedule center, if task type is “Bean模式” it will match bean mode task in Spring container and call it’s execute() method and execute task logic. if task type is “GLUE模式”, it will load Glue code, instantiate a Java object and inject other spring service（notice: the spring service injected in Glue code must exist in the same executor project）, then call execute() method and execute task logic.
XXL-JOB will generate a log file for every schedule request, the log info will be recorded by XxlJobLogger.log() method, the log file will be loaded when view log info through schedule center.
(history version is implemented by overriding LOG4J’s Appender so it exists dependency restrictions, The way has been discraded in the new version)
The location of log file can be specified in executor configuration file, default pattern is : /data/applogs/xxl-job/jobhandler/formatted date/primary key for database scheduling log records.log”.
When start child thread in JobHandler, child thread will print log in parent JobHandler thread’s execute log in order to trace execute log.
- 1,schedule center send http request to executor, and the service in executor in fact is a jetty server with default port 9999.
- 2,executor execute task logic.
- 3,executor http callback with schedule center for schedule result, the service in schedule center used to receive callback request from executor is a set of api opended to executor.
When scheduler center send request to executor, it will use RequestModel and ResponseModel object to encapsulate schedule request parameters and response data, these two object will be serialized before communication, data protocol and time stamp will be checked so that achieve data encryption target.
Task executor machine property has been canceled from v1.5, instead of task register and auto discovery, get remote machine address dynamic.
AppName: unique identify of executor cluster, executor is minimal unite of task register, every task recognize machine addresses under the executor on which it was binded.
Beat: heartbeat cycle of task register, default is 15s, and the time executor usedto register is twice the time, the time used to auto task discover is twice the beat time, the invalid time of register is twice the Beat time.
registry table: see XXL_JOB_QRTZ_TRIGGER_REGISTRY table, it will maintain a register record periodically while task register, such as the bind relationship between machine address and AppName, so that schedule center can recognize machine list by AppName dynamicly.
To ensure system lightweight and reduce learning costs, it did not use Zookeeper as register center, Use DB as register center to do task registration.
Since v1.6.2, the task execute result is recognized through ReturnT of IJobHandler, it executes success when return value meets the condition "ReturnT.code == ReturnT.SUCCESS_CODE" , or it executes fail, and it can callback error message info to schedule center through ReturnT.msg, so it can control task execute results in the task logic.
When “分片广播” is selected as route policy in executor cluster, one task schedule will broadcast all executor node in cluster to trigger task execute in every executor, pass slice parameter at the same time, so we can develop slice task by slice parameters.
"分片广播" break the task by the dimensions of executor, support dynamic extend executor cluster so that it can add slice number dynamically to do business process, In case of large amount of data process can significantly improve task processing capacity and speed.
The develop process of "分片广播" is the same as general task, The difference is that you can get slice parameters，code as shown below（go and see ShardingJobHandler in execuotr example ):
To improve system security it is need to check security between schedule center and executor, just allow communication between them when AccessToken of each other matched.
The AccessToken of scheduler center and executor can be configured by xxl.job.accessToken.
There are only two settings when communication between scheduler center and executor just:
The scheduling center provides API services for executors and business parties to choose to use, and the currently available API services are available.
1. Job result callback service;
2. Executor registration service;
3. Executor registration remove services;
4. Triggers a single execution service, and support the task to be triggered according to the business event;
The scheduling center API service location: com.xxl.job.core.biz.AdminBiz.java
- 1、simple:support CRUD operation through Web page, simple and one minute to get started;
- 2、dynamic:support dynamic update task status,pause/recover task and effective in real time;
- 3、service HA:task info stored in mysql, Job service support cluster to make sure service HA;
- 4、task HA:when some Job services hangs up, tasks will be assigned to some other alive machines, if all nodes of the cluster hangs up, it will compensate for the execution of lost task when restart;
- 5、one task instance will only be executed on one executor;
- 2、dicard underlying communication type servlet, JETTY was recommended, schedule and callback bidirectional communication, rebuild the communication logic;
-GLUE mode executor:every executor corresponds to a piece of code，edited and maintained online by Web, Dynamic compile and takes effect in real time, executor is responsible for loading GLUE code and executing;
- /xxl-job-admin -------------------- 【schedule center】:Responsible for managing schedule information，send schedule request according to schedule configuration;
- 1、Task dependency: it is implemented by trigger event, it will automatically trigger a child task schedule after Task execute success and callback, multiple child tasks are separated by commas;
- 2、executor source code has been reconstructed, optimize underlying db script;
- 3、optimize task thread group logic of executor, before it is group by executor’s JobHandler so when multiple task reuse Jobhanlder will cause block with each other. Now it is grouped by task of schedule center so tasks are isolated from task execution.
- 4、optimize communication scheme between executor and schedule center, a simple RPC protocol was implemented through Hex + HC, optimize the maintenance and analysis process of communication parameters.
- 5、schedule center, create/edit task, page attribute adjustment:
- 5.1、the property JobName was removed from task add/edit page and it is changed to automatically generate by system: this field before is used to identify a task in schedule center and did not use in other scenes, so remove it to simplify the task creation;
- 5.2、adjust "GLUE模式" property in task add/edit page to near JobHandler input box;
- 5.3、"报警阈值" property was removed from task add/edit page;
- 5.4、"子任务Key" property was removed from task add/edit page, the key of task can be acquired from task list page, child task will be triggered by child task key when main task execute success.
- 6、bug fix:
- 6.1、optimize jetty executor shutdown, solve one problem may cause jetty could not shutdown.
- 6.2、optimize callback of executor task queue when task execute finish. Solve a problem which may cause task could not callback.
- 6.3、Optimize Page List Parameters of Schedule Center, solve one problem which may be caused by post length limit of server.
- 6.4、optmize executor Jobhandler annotation, solve a problem that container could not load the JobHandler caused by the transaction proxy.
- 6.5、optimize remote schedule, disable retry policy, solve a problem may caused repeat call;
Tips: V1.3.x release has been published , enter the maintenance phase, branch address is [V1.3](https://github.com/xuxueli/xxl-job/tree/v1.3) .New features will be updated continuously in the master branch.
- 2、To adapt to the rules of central warehouse, groupId has been changed from com.xxl to com.xuxueli.
- 3、to resolve the problem that sub-modules can not be compiled separately, system version is not maintained in the project root pom, each sub-module is configured separately for version configuration;
- 4、optimize data byte length statistics rule of RPC communication it may reduce 50% of data traffic;
- 5、IJobHandler cancel task return value, before the execution status is judged by the return value, now it instead of task was executed successfully by default only when exception was caught the task execution was judged failed.
- 6、optimize system public pop-up box as a plugin;
- 7、optimize table structure and the table name now is upper case;
- 8、modify ContentType of JSON response from exception handler of schedule center to fix the bug that it is could not recognized by browser.
- 1、task register: executor registers the task automatically, schedule center will automatically discover the registered task and trigger execution.
- 2、add parameter AppName for executor: AppName is the unique identifier of each executor cluster, register periodically and automatically with AppName.
- 3、add column executor management in schedule center : manage online executors, automatically discover registered executors via the property AppName。Only managed executors are allowed to be used;
- 4、change Task group attribute to executor : each task needs to be bound to the specified exector, schedule address is obtained by binded executor;
- 5、discard property task machine: by the way of binding task with executor, automatically discovers registered remote executor address and triggers schedule request.
- 6、add DBGlueLoader in public dependency, it implement GLUE source code calssloader based on native jdbc, Reduce third party reliance (mybatis,spring-orm etc); simplify and optimize executor configuration (for GLUE task), Reduce the difficulty of getting started;
- 7、adjust table structure, reconstruct the project;
- 8、schedule center automatically registered and found, failover: schedule center periodically registered automatically, task callback can recognize all online schedule center addresses, task callback support failover so that it can avoid single point of risk.
- 1、optimize IP tools class which used to gets IP address，IP static cache;
- 2、both executor and schedule center support customize registered IP address;Solve problem when machine has multiple network card and get the wrong card;
- 3、solve the problem that it will generate multiple log files when executed across days;
- 4、the non-sensitive log level is adjusted to debug;
- 5、Upgrade the database connection pool to c3p0;
- 6、optimize log4j property of executor，remove invalid attribute;
- 7、reconstruct underlying code and optimize logic and Clean Code;
- 4、unified thread model and thread destruction scheme (by the way of listener or stop() method，Destroy the thread when container is destroyed;Daemon is sometimes not ideal);
- 5、unified system configuration data，Unified managed by configuration files;
- 6、CleanCode，Clean up invalid historical parameters;
- 8、new created task defaults to a non-running state;
- 9、optimize update logic of GLUE mode task instance , The original update is based on the timeout value and now is updated according to the version number，version number plus one while source changed;
- 1、execution report:support view run time data in real time, such as task number, total schedule number, executor number etc., include schedule report , such as scheduled distribution graph on date, scheduled success distribution graph etc;
- 2、JobHandler support set return value for tasks, it is easy to control task execute result in task logic;
- 3、the problem could not view exception info when resource path include space or chinese word casused resource file could not be loaded;
- 4、optimize route policy:fix problems that Loop and LFU routing policy counters are no limit and first route is focused on the first machine;
- 1、unified write and read code of execute log as UTF-8，solve log garbled problem under windows environment;
- 2、communication timeout period is limited to 10s，To avoid schedule thread is occupied under abnormal situation;
- 3、adjust executor , server stat, destroy and register logic;
- 4、optimize Jetty Server shutdown logic, repair port occupation caused by executor could not be closed normally and frequent printe c3p0 log probleam;
- 5、start child thread in JobHandler，support child thread print execute log and view by Rolling;
- 1、block handle policy:the policy when schedule is too frequently and the executor it too late to handle, include multiple strategies:single machine serially execute（default）、discard subsequent schedule、override before schedule;
- 2、fail handle policy:handle policy when scheduled fail, include :failure alarm（default）、failed to retry;
- 3、The communication timeout is adjusted to 180s;
- 4、executor and database are completely decoupled，But the executor needs to configure schedule center cluster address。schedule center provides APIs for executor callbacks and heartbeat registration services，cancel jetty inner schedule center, heartbeat cycle is adjusted to 30s，heartbeat failure is triple heartbeat;
- 5、fix executor parameters lost bug when edit;
- 6、add task test Demo to make task logic test easier;
- 1、optimize update logic of task Cron，instead of rescheduleJob，at the same time preventing set cron repeatedly;
- 2、optimize API callback service failed status code，facilitate troubleshooting;
- 3、XxlJobLogger support multi-parameter;
- 4、route policy add "忙碌转移" mode:Perform idle detection in sequence，The first idle test successfully machine is selected as the target executor and trigger schedule;
- 5、reconstruct route policy code;
- 6、fix executor repeat registration problem;
- 7、Task thread will be destroyed after 30 times idle turn, reduce the inefficient thread consumption of low frequency tasks;
- 1、slice broadcast task:When slice broadcast is selected as route policy in executor cluster, one task schedule will broadcast all executor node in cluster to trigger task execute in every executor, pass slice parameter at the same time, so we can develop slice task by slice parameters;
- 2、dynamic slice: break the task by the dimensions of executor, support dynamic extend executor cluster so that it can add slice number dynamically to do business process, In case of large amount of data process can significantly improve task processing capacity and speed;
- 3、executor JobHandler disables name conflicts;
- 4、executor cluster address list for natural sorting;
- 5、add test cases and optimize DAO layer code for Scheduling center;
- 6、schedule Center API service change to self-study RPC framework to u nify communication model;
- 7、add schedule center API service test Demo, convenient in dispatch center API extension and testing;
- 8、Task list page interaction optimization，The task list is automatically refreshed when the executor group is replaced，create new job defaults to locate current executor position;
- 9、access Token:To improve system security，it is used for safety check between schedule center and executor, communication allowed just when Both Access Token matched;
- 10、upgrade springboot version to 1.5.6.RELEASE of executor;
- 1,Task privilege management:control privilege on executor, check privilege on core operations;
- 2,Task slice routing:using consistent Hash algorithm to calculate slice order as stable as possible, even if there is fluctuation in the registration machine will not cause large fluctuations in the order of slice. Currently using IP natural sorting can meet the demand，to be determined;
- 3,Failure retry optimization:The current failure to retry logic is execute the request logic once again after the scheduled request fails。The optimization point is retry for both scheduling and execution failures, retry a full schedule when retrying，This may lead schedule failure to an infinite loop，to be determined;
- 4,write file when callback failed，read the log when viewing the log，callback confirm after rebooting;
- 5,Task dependency，flow chart，child task + aggregation task，log of each node;
- 6,Scheduled task priority;
- 7,Remove quartz dependencies and rewrite scheduld module:insert the next execution record into delayqueue when add or resume task, schedule center cluster compete distributed lock，successful nodes bulk load expired delayqueue data and batch execution;
- 8,springboot and docker image，and push docker image to the central warehouse，further realize product out of the box;
- 9,globalization:schedule center interface and Official documents，add English version;
- 10,executor removal:notify schedule center and remove the corresponding execute node when executor is destroyed, improve the timeliness of executor state recognized;
Contributions are welcome! Open a pull request to fix a bug, or open an [Issue](https://github.com/xuxueli/xxl-job/issues/) to discuss a new feature or change.
This product is open source and free, and will continue to provide free community technical support. Individual or enterprise users are free to access and use.
