In an application service, timing for less demanding business scenarios, we don’t need to wait until all the tasks performed to return as a result, such as user registration scenario, save the user account password, you can return immediately, subsequent account activation email, you can use a form of asynchronous to deal with, this kind of asynchronous operations can use the queue service. Otherwise, seconds may pass if you wait for the email to be sent.

What is Celery?

Celery is a distributed queue service implemented by Python language. Besides supporting immediate tasks, Celery also supports scheduled tasks. Celery has five core roles.

Task

Task (Task) that you want to do, such as a registered process there are many tasks, verify the mail is sent to the user is a Task, the time consuming Task to the hands of the Celery, and a Task is timing Task, such as the enrollment time statistics web site every day, this also can be dealt with to Celery periodically.

Broker

Broker is a person who serves as an intermediary between buyers and sellers in the market. In Celery, this role acts like a queue in the data structure and acts as a broker between the producer and consumer. For example, in a Web system, the producer is the main program, which produces tasks and sends them to the Broker, and the consumer is the Worker, which is the background service dedicated to performing the tasks. Celery tasks do not provide queue services. Redis or RabbitMQ are used to implement queue services.

Worker

The Worker is the person who always executes tasks in the background, and also becomes the consumer of tasks. It will monitor whether there are any tasks in the queue in real time, and if there are any tasks, it will immediately take them out for execution.

Beat

Beat is a scheduled task scheduler that sends tasks to the Broker at a specified time and waits for the Worker to consume them.

Backend

Backend stores the results of tasks. Every task has a return value. For example, the service that sent an email will tell us if it was sent successfully.

If you want to add tasks like Celery tasks, you can add tasks like Celery tasks.

Quick start

With anything new, nothing learns faster than actually doing it. Assuming we choose Redis as broker, you need to have Redis installed and the Redis service started (use your own search engine for this step).

pip install -U "celery[redis]"
Copy the code

1, create Celery instance

# tasks.py
from celery import Celery

app = Celery('tasks', broker='redis://localhost:6379/0')
Copy the code

2. Create tasks

Let’s say the task of sending the email takes five seconds to complete

# tasks.py
@app.task
def send_mail(email):
    print("send mail to ", email)
    import time
    time.sleep(5)
    return "success"
Copy the code

If no tasks are added, the program is executed sequentially and each step needs to wait for the previous step to complete.

1. Insert records into database 2. Send emails 3Copy the code

We can put 2 in a task and give it to tasks to execute, so we don’t need to wait for email to complete, you just need to arrange tasks to deal with and help me complete. The code becomes

1. Insert record into database 2. Registered successfullyCopy the code

The second step is very fast, it just puts the task into the queue and doesn’t wait for the task to actually execute. This is perfectly true of life, for example, where we don’t do a lot of things ourselves, but rather hand over a less important or less immediate task to someone else.

3. Start Worker

Start Worker, listen for tasks in Broker, command: celery Worker, you may need to specify parameters

celery -A tasks worker --loglevel=info
Copy the code

-a: specifies the module in which the tasks instance resides. In this example, the tasks instance is in the tasks.py file

App. task () {return app.task ();

4. Call the task

The task is invoked in the main program and sent to the Broker instead of actually executing it

# user.py
from tasks import send_mail

def register(a):
    import time
    start = time.time()
    print("1. Insert record into database")
    print(2. Celery send email for me)
    send_mail.delay("[email protected]")
    print("3. Tell the user that the registration is successful")
    print("耗时：%s 秒 " % (time.time() - start))

if __name__ == '__main__':
    register()

Copy the code

In the main program, the.delay method of the function is called

Directory structure:

├─ celery_test ├── task.py ├─ key.pyCopy the code

Run Python user.py to start the application

1. Insert record into database 2. Tell the user the successful registration time: 0.22688984870910645 secondsCopy the code

The program took less than 0.23 seconds to complete, or at least 5 seconds if you follow the normal synchronization logic, because the email task took 5 seconds.

View log information in the worker service window

Note:

If user root is not selected, set environment variables:

$ export C_FORCE_ROOT='true'
Copy the code

2. Celery4.x is no longer supported on Windows. If you need to develop on Windows, please use 3.x version.

Use RabbitMQ or Redis as Broker and never use a relational database in production

4. Do not use complex objects as parameters to task functions

# Good
@app.task
def my_task(user_id):
    user = User.objects.get(id=user_id)
    print(user.name)
    #...
Copy the code
# Bad
@app.task
def my_task(user):
    print(user.name)
    #...
Copy the code

summary

To learn Celery, first you need to know how to use them, then the common roles in Celery, and finally follow the steps to get a feel for how they run.

Reference links:

  • Funhacks.net/2016/12/13/…
  • Celery. Readthedocs. IO/en/latest/u…
  • celerytaskschecklist.com/

Blog at foofish.net