Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
328 views
in Technique[技术] by (71.8m points)

python - How to return data from a CherryPy BackgroundTask running as fast as possible

I'm building a web service for iterative batch processing of data using CherryPy. The ideal workflow is as follows:

  1. Users POST data to the service for processing
  2. When the processing job is free, it collects the queued data and starts another iteration
  3. While the job is processing, users are POSTing more data to the queue for the next iteration
  4. Once the current iteration is finished, the results are passed back so that users can GET them using the same API.
  5. The job starts again with the next batch of queued data.

The key consideration here is that the processing should run as fast as possible with each iteration starting as soon as the previous one finishes, regardless of the amount of data in the queue. There's no upper bound on how long each iteration can take so I can't create a fixed schedule for it to run on.

There are a few examples of using BackgroundTask (like this one) but I've yet to find one that deals with returning data, or one that deals with tasks running as fast as possible as opposed to on a fixed schedule.

I'm not wedded to the BackgroundTask solution so if anyone can offer an alternative one I'd be more than happy. It feels like there's a solution within the framework though.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

Don't run a background task using the BackgroundTask solution, because it will run in a thread and, due to the GIL, cherrypy won't be able to answer new requests. Use a queue solution that runs your background tasks in a different process, like Celery or RQ.

I'm going to develop in detail an example using RQ. RQ uses Redis as a message broker, so first of all you need to install and start Redis.

Then create a module (mytask in my example) with the long time running background methods:

import time
def long_running_task(value):
    time.sleep(15)
    return len(value)

Start one (or more than one if you want to run tasks in parallel) RQ workers, it's important that the python that is running your workers has access to your mytask module (export the PYTHONPATH before running the worker if your module it's not already in the path):

# rq worker

Above you have a very simple cherrypy webapp that shows how to use the RQ queue:

import cherrypy
from redis import Redis
from rq import Queue    
from mytask import long_running_task


class BackgroundTasksWeb(object):

    def __init__(self):
        self.queue = Queue(connection=Redis())
        self.jobs = []

    @cherrypy.expose
    def index(self):
        html =  ['<html>', '<body>']
        html += ['<form action="job">', '<input name="q" type="text" />', '<input type="submit" />', "</form>"]
        html += ['<iframe width="100%" src="/results" />']
        html += ['</body>', '</html>']
        return '
'.join(html)

    @cherrypy.expose
    def results(self):
        html = ['<html>', '<head>', '<meta http-equiv="refresh" content="2" >', '</head>', '<body>']
        html += ['<ul>']
        html += ['<li>job:{} status:{} result:{} input:{}</li>'.format(j.get_id(), j.get_status(), j.result, j.args[0]) for j in self.jobs]
        html += ['</ul>']
        html += ['</body>', '</html>']
        return '
'.join(html)

    @cherrypy.expose
    def job(self, q):
        job = self.queue.enqueue(long_running_task, q)
        self.jobs.append(job)
        raise cherrypy.HTTPRedirect("/")


cherrypy.quickstart(BackgroundTasksWeb())

In a production webapp I would use jinja2 template engine to generate the html, and most likely websockets to update the job status in the web browser.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...