express - Nginx Reverse Proxy to Docker 502 Bad Gateway

Question

Spent all week on this one and tried every related stackoverflow post. Thanks for being here.

I have an Ubuntu VM running nginx with reverse proxies pointing to various docker daemons concurrently running on different ports. All my static sites work flawlessly. However, I have one container running an expressjs app.

I get responses after restarting the server for about an hour. Then I get 502 Bad Gateway. A refresh brings the site back up for approx 5 seconds until it permanently goes down. This is reproducible.

• The docker container has express listening on 0.0.0.0:8090 inside the container

• The container is running

  02e1917991e6   docker/express-site       "docker-entrypoint.s…"   About an hour ago   Up About an hour   127.0.0.1:8090->8090/tcp   express-site
  

• The 8090 port is EXPOSEd in the Dockerfile.

• I tried other ports.

• When down, I can curl the site from within the container when inspecting.

• When down, curling the site from within the VM yields

  curl: (52) Empty reply from server
  

• Memory and CPU usage within the container and within the VM barely reach 5%.

• Site usually has SSL but tried http as well.

• Tried various nginx proxy settings (see config below)

• Using out-of-the box nginx.conf

• Considering that it might be related to a timeout or docker network settings...

My site-available config file looks like:

server {
   server_name example.com www.example.com;

   location / {
        proxy_pass http://127.0.0.1:8090;
        #proxy_set_header Host $host;
        #proxy_buffering off;
        #proxy_buffer_size 16k;
        #proxy_busy_buffers_size 24k;
        #proxy_buffers 64 4k;
   }
    listen 80;
    listen [::]:80;
    #listen 443 ssl; # managed by Certbot
    #ssl_certificate /etc/letsencrypt/live/www.example.com/fullchain.pem; # managed by Certbot
    #ssl_certificate_key /etc/letsencrypt/live/www.example.com/privkey.pem; # managed by Certbot
    #include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    #ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

Nginx Error Log shows:

2021/01/02 23:50:00 [error] 13901#13901: *46 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: ***.**.**.***, server: example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8090/favicon.ico", host: "www.example.com", referrer: "http://www.example.com"

Anyone else have ideas?

question from:https://stackoverflow.com/questions/65545516/nginx-reverse-proxy-to-docker-502-bad-gateway

Answer 1

Didn't get much feedback, but I did more research and the issue is now stable so I wanted to post my findings.

I have isolated the issue with the docker container. Nginx works fine with the same app running on the VM directly.

I updated my docker container image from node:12-alpine to node:14-alpine. The site has been up for 42 hours without issue.

If it randomly fails again, then it's probably due to load.

I hope this solves someone's issue.