Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
450 views
in Technique[技术] by (71.8m points)

python - Run a chord callback even if the main tasks fail

Is it possible to run a chord callback even if the main tasks failed?

I've created a chord which I added a bunch of tasks and registered a callback to it. My problem, is that if one of the tasks fail the callback is not triggered, but I would like the callback to be triggered either way.

I've tried to register the callback with si() (immutability)

callback = tasks.run_delete_rule.si([timestamp])
header = [tasks.run_update_rule.s(i, timestamp) for i in item_ids]
result = chord(header)(callback)

I also tried to add the param ignore_result=True to both tasks decorator, but no success.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

From the github issue #1881 if the callback has the link_error option set, which takes a list of tasks names, then when a task of the chord fails the link_error tasks are going to be executed.

@task(name='super_task.good')
def good():
    return True

@task(name='super_task.raise_exception')
def raise_exception():
    raise ValueError('error')

@task(name='super_task.callback')
def callback(*args, **kwargs):
    logger.info('callback')
    logger.info(args)
    logger.info(kwargs)
    return 'finished'

@task(name='super_task.error_callback')
def error_callback(*args, **kwargs):
    logger.info('error_callback')
    logger.info(args)
    logger.info(kwargs)
    return 'error'

>>> c = chord(
        [raise_exception.s(), good.s(), raise_exception.s()], 
        callback.s().set(link_error=['super_task.error_callback'])
    )
>>> result = c()

This will execute the chord and in your celery log, you'll see the raise_exception task fail, and the execution of error_callback which will receive in it's args the task_id of callback.

At this point the value of result will contain the AsyncResultinstance of callback, and because in a chord the errors propagate to the callback doing result.get() will raise the exception of the tasks and result.traceback gives you the traceback.

If you want to have a single callback, just pass the name of the chord callback to link_error

callback.s().set(link_error='super_task.callback')

NOTE

Another options it's to set CELERY_CHORD_PROPAGATES = False which will revert to the pre celery 3.1 behavior and always execute the callback.

But this is not a recommended approach because as you can find in the github issue #1349

Celery 3.1 defines how chord errors are handled, the previous behavior was never documented and more of an accident since it was never the intention to work that way.

We couldn't change the behavior in a bugfix release so a setting had to be used instead, but it was never the intention that someone would deliberately disable the new behavior.

The new behavior is there to protect against this sort of issue happening, and the backward compatible setting may be removed. I suggest you find some other way to handle errors here (and I wouldn't mind a proposal if you can invent a nice api for it)


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...