docker - How to fix "process apparently never started in ..." error in Jenkins pipeline?

Question

I am getting the strange error below in my Jenkins pipeline

[Pipeline] withDockerContainer
acp-ci-ubuntu-test does not seem to be running inside a container
$ docker run -t -d -u 1002:1006 -u ubuntu --net=host -v /var/run/docker.sock:/var/run/docker.sock -v /home/ubuntu/.docker:/home/ubuntu/.docker -w /home/ubuntu/workspace/CD-acp-cassandra -v /home/ubuntu/workspace/CD-acp-cassandra:/home/ubuntu/workspace/CD-acp-cassandra:rw,z -v /home/ubuntu/workspace/CD-acp-cassandra@tmp:/home/ubuntu/workspace/CD-acp-cassandra@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** quay.io/arubadevops/acp-build:ut-build cat
$ docker top 83d04d0a3a3f9785bdde3932f55dee36c079147eb655c1ee9d14f5b542f8fb44 -eo pid,comm
[Pipeline] {
[Pipeline] sh
process apparently never started in /home/ubuntu/workspace/CD-acp-cassandra@tmp/durable-70b242d1
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
$ docker stop --time=1 83d04d0a3a3f9785bdde3932f55dee36c079147eb655c1ee9d14f5b542f8fb44
$ docker rm -f 83d04d0a3a3f9785bdde3932f55dee36c079147eb655c1ee9d14f5b542f8fb44
[Pipeline] // withDockerContainer

The corresponding stage in Jenkins pipeline is

    stage("Build docker containers & coreupdate packages") {
        agent {
            docker {
                image "quay.io/arubadevops/acp-build:ut-build"
                label "acp-ci-ubuntu"
                args "-u ubuntu --net=host -v /var/run/docker.sock:/var/run/docker.sock -v $HOME/.docker:/home/ubuntu/.docker"
              }
          }
          steps {
              script {
                 try {
                    sh "export CI_BUILD_NUMBER=${currentBuild.number}; cd docker; ./build.sh; cd ../test; ./build.sh;"
                    ciBuildStatus="PASSED"
                 } catch (err) {
                    ciBuildStatus="FAILED"
                 }
              }
          }
      }

What could be the reasons why the process is not getting started within the docker container? Any pointers on how to debug further are also helpful.

Answer 1

This error means the Jenkins process is stuck on some command.

Some suggestions:

• Upgrade all of your plugins and re-try.
• Make sure you've the right number of executors and jobs aren't stuck in the queue.
• If you're pulling the image (not your local), try adding alwaysPull true (next line to image).
• When using agent inside stage, remove the outer agent. See: JENKINS-63449.
• Execute org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true in Jenkins's Script Console to debug.
• When the process is stuck, SSH to Jenkins VM and run docker ps to see which command is running.
• Run docker ps -a to see the latest failed runs. In my case it tried to run cat next to custom CMD command set by container (e.g. ansible-playbook cat), which was the invalid command. The cat command is used by design. To change entrypoint, please read JENKINS-51307.
• If your container is still running, you can login to your Docker container by docker exec -it -u0 $(docker ps -ql) bash and run ps wuax to see what's doing.
• Try removing some global variables (could be a bug), see: parallel jobs not starting with docker workflow.