python - Daemonizing celery

Question

Following instructions found here, I copied the script from github into /etc/init.d/celeryd, then made it executable;

$ ll /etc/init.d/celeryd
-rwxr-xr-x 1 root root 9481 Feb 19 11:27 /etc/init.d/celeryd*

I created config file /etc/default/celeryd as per instructions:

# Names of nodes to start
#   most will only start one node:
#CELERYD_NODES="worker1"
#   but you can also start multiple and configure settings
#   for each in CELERYD_OPTS (see `celery multi --help` for examples).
CELERYD_NODES="worker1 worker2 worker3"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"

# App instance to use
# comment out this line if you don't use an app
#CELERY_APP="proj"
# or fully qualified:
#CELERY_APP="proj.tasks:app"

# Where to chdir at start.
CELERYD_CHDIR="/path/to/folder/containing/tasks/"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=3000 --concurrency=3 --config=celeryconfig"

# %N will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"

# Workers should run as an unprivileged user.
#   You need to create this user manually (or you can choose
#   a user/group combination that already exists, e.g. nobody).
CELERYD_USER="celery"
CELERYD_GROUP="celery"

# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1

Note: I added the --config=celeryconfig part in CELERYD_OPTS.

I created a new user celery with

sudo useradd -N -M --system -s /bin/false celery

and then created group celery and added user to it:

$ id celery
uid=999(celery) gid=1005(celery) groups=1005(celery)

I used chown celery:celery for folders:

/var/run/celery/

/var/log/celery/

When I try to start the service, I get no indication of error:

$ sudo service celeryd start
celery init v10.0.
Using config script: /etc/default/celeryd

but status gives me "no pids were found":

$ sudo service celeryd status
celery init v10.0.
Using config script: /etc/default/celeryd
celeryd is stopped: no pids were found

and indeed, ps -ef gives no results and no PID file or log files are created:

$ ll /var/run/celery
total 0
drwxr-sr-x  2 celery celery  40 Feb 19 14:13 ./
drwxr-xr-x 16 root   root   620 Feb 19 12:35 ../

$ ll /var/log/celery
total 8
drwxr-sr-x  2 celery celery 4096 Feb 19 14:13 ./
drwxr-xr-x 13 root   root   4096 Feb 19 11:37 ../

Additional info:

$ pip freeze | grep celery
celery==3.1.9

What am I missing? Where should I be looking for more clues as to why celeryd is not starting?

Answer 1

I replicated your steps and it resulted in the same issue. The problem was the celery user not having a shell.

sudo useradd -N -M --system -s /bin/false celery

Changing -s /bin/false to -s /bin/bash fixed the issue. The reason being is that the celeryd init script uses the celery user's shell to execute celery commands. Without a shell, the su command below exits silently.

_chuid () {
    su "$CELERYD_USER" -c "$CELERYD_MULTI $*"
}