Main    /    Blog    /    uWSGI Spooler

←   →

uWSGI Spooler

4 May 2017 4 min read
Tags:

The best way to implement a scalable system that communicates with multiple external components—whether it uses an external API, sends email, or converts videos—is by using an asynchronous task queue, which mediates the interactions between all system components.

The most popular queue for Python is Celery, which offers a wide range of task-management options and abilities.
Unfortunately, Celery-based systems aren’t the easiest to keep up and running, and when something doesn’t go quite right, the issue usually isn’t that easy to find. You can ask any DevOps engineer about their experience with Celery, but be prepared for some harsh words.

Luckily, there’s an alternative: uWSGI Spooler, and that’s what we’ll be talking about in today’s article.

The main difference between uWSGI Spooler and Celery is that the former doesn’t require any additional components (Celery may need a storage system like Redis, for example), which cuts the number of points of failure in half. Tasks can be stored in a directory, external directory, or network pool.

We often use uWSGI ourselves to manage Python programs.

Why?

Because it’s easy to set up, reliable, flexible, and meets the bulk of our requirements.

In addition to ensuring Python code is continually accessible to web applications, uWSGI includes a Spooler, which implements the queuing system.
Spooler has quite a few features, but its documentation is fairly lacking.

uWSGI Spooler is super easy to use; however, there are a few nuances.

Nuances

The uWSGI module can’t be imported from code and therefore can’t be tested from the console. You’ll have to launch a uWSGI worker each time, and this requires you create a config file:

[uwsgi]
socket = /var/run/mysite.sock
master = True
processes = 4
project_dir = /home/myuser/mysite
chdir = %(project_dir)
spooler = /var/uwsgi_spools/mysite_spool
spooler-import = path.to.spool.package # (package to import spool file)
spooler-frequency = 10 # Frequency for scanning spool
max-requests = 5000
module = wsgi:application
touch-reload = wsgi.py

The worker file:

from uwsgidecorators import spool, uwsgi
 
@spool
def my_func(args):
    print(args)
    # do some job

Setting a task from your code:

import uwsgi_spools.mysite_spool as mysite_spool
 
mysite_spool.my_func.spool(test=True)

As you can see from this example, the learning curve is extremely short.

The task contains an argument, which itself contains a dictionary consisting of three service keys (the function name: ud_spool_func, task name: spooler_task_name, and task status: ud_spool_ret) and all of the parameters that were given when it was created; in our example, this is the test key.

The task can return three statuses:

  • -2 (SPOOL_OK) – task complete and will be deleted from queue
  • -1 (SPOOL_RETRY) – something didn’t go as planned and the task will be called again
  • 0 (SPOOL_IGNORE) – ignore the task

All other values will be interpreted as -1 (SPOOL_RETRY).

Note: The @spool decorator will run only once (and returns SPOOL_OK) unless the function fails with an exception.
Life cycles can be managed using @spoolraw.

Keys

When you create a task, you can include special keys:

  • spooler — the absolute path of the spooler that will run the task
  • at — the unix time when a task should be run (or more accurately: the absolute earliest a task will be run)
  • priority — shows the subdirectory in the spooler directory (multiple workers can be assigned to one subfolder) where –spooler-ordered can set up priorities
  • body — this key is used for values over 64 KB and is available to the task as a serial

In addition to the @spool decorator, there is also the @timer decorator, which takes the number of seconds and uses it as an argument to run decorated functions at a given interval.

@timer(30)
def my_func(args):
    print(args)
    # do some job every 30 sec

Like @timer, there is also @spoolforever, which repeatedly launches a function (tasks are completed with the status SPOOL_RETRY).

@spoolforever
def my_func(args):
    print(args)
    # do some job and repeat

Configuring Workers

To configure a worker for your network, you have to add the address it will be accessible at to the ini file:

socket = 127.0.0.1:10001

When we create a task, we give the address of the target:

uwsgi.send_message(“127.0.0.1:10001”, 17, 0, test=True, 5)
# or
uwsgi.spool(test=True, spooler=“127.0.0.1:10001”)

Conclusions

Today we saw how uWSGI Spooler can be used as a replacement queue. If you feel uWSGI Spooler doesn’t offer all the functions you need or you’re just looking for something a bit sweeter, uswgi-tasks fills in all of those gaps.

4 May 2017 Tags:

Popular articles