Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
507 views
in Technique[技术] by (71.8m points)

python - Celery is rerunning long running completed tasks over and over

I've a python celery-redis queue processing uploads and downloads worth gigs and gigs of data at a time.

Few of the uploads takes upto few hours. However once such a task finishes, I'm witnessing this bizarre celery behaviour that the celery scheduler is rerunning the just concluded task again by sending it again to the worker (I'm running a single worker) And it just happened 2times on the same task!

Can someone help me know why is this happening and how can I prevent it?

The tasks are definitely finishing cleanly with no errors reported just that these are extremely long running tasks.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

I recently ran into this issue, and eventually figured out that tasks were running multiple times because of a combination of task prefetching and tasks exceeded the visibility timeout. Tasks are acknowledged right before they're executed (unless you set ACKS_LATE=True), and by default 4 tasks are prefetched per process. The first task will be acknowledged before execution, but if it takes over an hour to execute then the other prefetched tasks will be delivered to another worker where it will be executed an additional time (or in your case, executed an additional time by the same worker).

You can solve by increasing the visibility timeout to something longer than the longest possible runtime of your tasks:

BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 3600*10}  # 10 hours

You could also set PREFETCH_MULTIPLIER=1 to disable prefetching so that long running tasks don't keep other tasks from being acknowledged.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...