kubernetes - Nginx reports "upstream connection timeout"

Question

Discription: The k8s nginx-ingress-controllers are exposed in loadbalancer type(implemented by metallb) with ip addr 192.168.1.254. Another nginx cluster is in front of k8s cluster and it has only one upstream which is 192.168.1.254(lb ip addr).The request flow route:client -> nginx clusters -> nginx-ingress-controllers-> services.

Question: Sometimes nginx cluster reports very tiny little "upstream(192.168.1.254) time out" and finally the client got 504 timeout from nginx.

But When I dropped the nginx cluster and switch request flow to : client -> nginx-ingress-controllers-> services.It goes well and client didn't get 504 timeout any more.I am sure the network between nginx cluster and nginx ingress controller works well.

Most of requests can be handled by nginx cluster and return status 200.I have no idea why few requests report "upstream time out" and return status 504.

system architecture

nginx cluster reports timeout

tcpdump package track

question from:https://stackoverflow.com/questions/65838808/nginx-reports-upstream-connection-timeout

Answer 1

That's most likely slow file uploads (the requests you've showed are all POSTs), something that can't fit into the limit.

You can set a greater timeout value for application paths where uploads can be possible. If you are using and ingress controller you'd better create a separate ingress object for that. You can manage timeouts with these annotations, for example:

  annotations:
    nginx.ingress.kubernetes.io/proxy-send-timeout: 300s
    nginx.ingress.kubernetes.io/proxy-read-timeout: 300s

These two annotations define the maximum upload time to 5 minutes.

If you are configuring nginx manually, you can set limits with proxy_read_timeout and proxy_send_timeout.