flask - Service+deployment on GKE hangs up after 30 minutes

Question

I tried to look up for solutions, but I am probably not asking the correct question.

I have a service performing long running tasks deployed inside a GKE cluster (a flask app served through gunicorn). When I make a request, Postman returns a Error: socket hang up after 30 minutes. What am I not considering? Is the behaviour due to the service in front of the deployment?

My chain of services is as follows:

• Consumer: Postman, setting for timeout=0 - i.e. infinite - and header "Connection":"keep-alive" is passed
• loadBalancer (exposed) proxy: node+express, timeout=7200000 (ms)
• clusterIP (internal) service: python+flask, gunicorn timeout=7200 (s)

Service is started through CMD exec gunicorn --bind 0.0.0.0:$PORT start:app --workers $NO_WORKERS --threads $NO_THREADS --timeout $TIMEOUT

The service has been deployed with the following yaml file by kubectl apply -f:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: layers-api
  labels: 
    app: layers-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: layers-api
      tier: backend
  template:
    metadata:
      labels:
        app: layers-api
        tier: backend
    spec:
      containers:
      - env:
        - name: PORT
          value: '4000'
        - name: NO_WORKERS
          value: '2'
        - name: NO_THREADS
          value: '2'
        - name: TIMEOUT
          value: '7200'
        # ...other env vars... 
        name: layers-api
        image: [DOMAIN]/[PROJECT]/[IMAGE]
        imagePullPolicy: Always
        ports:
          - containerPort: 4000
        readinessProbe:
          httpGet:
            path: /api/v1/healthz
            port: 4000
          initialDelaySeconds: 30
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /api/v1/healthz
            port: 4000
---
apiVersion: v1 
kind: Service
metadata:
  name: layers-api
  labels:
    app: layers-api
    tier: backend
spec:
  selector:
    app: layers-api
    tier: backend
  ports:
  - port: 80
    targetPort: 4000
    protocol: TCP
    name: http
  - port: 443
    targetPort: 4000
    protocol: TCP
    name: https
---

Thank you for spending your time to help.

C

EDIT: this the log from the proxy server. Moreover, I use this proxy for another routing (to a managed cloud run service - beta timeout of 3600 seconds) and this does not hang up after 30 minutes).

question from:https://stackoverflow.com/questions/65928332/servicedeployment-on-gke-hangs-up-after-30-minutes

Answer 1

Waitting for answers