Airflow webserver doesn't find scheduler in docker-compose

Question

I'm currently trying to run a minimal Airflow ennvironment to test a few DAGs on my local machine. The setup is quite basic: I am using docker-compose to spin an Airflow webserver and a scheduler after initialising the local sqlite db.

My docker-compose.yml file looks like the following:

version: '3'

services:
  webserver:
    image: apache/airflow
    container_name: airflow_webserver
    restart: on-failure
    volumes:
      - ./dags:/opt/airflow/dags
      - ./plugins:/opt/airflow/plugins
      - ./scripts:/scripts
    ports:
      - 8080:8080
    environment:
      AIRFLOW_RUNAS_WEBSERVER: 1
    depends_on: 
      - scheduler
    entrypoint:  /scripts/wait-for-it.sh scheduler:8793 -t 40 -- /scripts/docker-entrypoint.sh

  scheduler:
    image: apache/airflow
    container_name: airflow_scheduler
    restart: on-failure
    ports:
      - 8793:8793
    volumes:
      - ./dags:/opt/airflow/dags
      - ./plugins:/opt/airflow/plugins
      - ./scripts:/scripts
    environment: 
      AIRFLOW_RUNAS_SCHEDULER: 1
    entrypoint: /scripts/docker-entrypoint.sh

Where my docker-entrypoint.sh is like:

#!/bin/bash

AIRFLOW_USERNAME=test
AIRFLOW_PASSWORD=test

if [ "$AIRFLOW_RUNAS_SCHEDULER" = "1" ]; then
    echo "initializing airflow database"
    airflow db init
    echo "running airflow scheduler"
    while true
    do
        airflow scheduler
        echo "restarting airflow scheduler"
        sleep 1
    done
elif [ "$AIRFLOW_RUNAS_WEBSERVER" = "1" ]; then
    echo "running airflow webserver"
    airflow db upgrade
    airflow users create 
        --username=$AIRFLOW_USERNAME 
        --password=$AIRFLOW_PASSWORD 
        --firstname=test 
        --lastname=test 
        [email protected] 
        --role=Admin
    exec airflow webserver -p 8080
elif [ "$AIRFLOW_RUNAS_WORKER" = "1" ]; then
    echo "running airflow worker"
    exec airflow worker
elif [ "$AIRFLOW_RUNAS_FLOWER" = "1" ]; then
    echo "running airflow flower"
    exec airflow flower
else
    echo "ERROR: no AIRFLOW_RUNAS_* variable set"
    exit 1
fi

Now, from the logs I can clearly see that running docker-compose up both airflow webserver and airflow scheduler spin up their services correctly even though it seems Airflow webserver is having an hard time finding the scheduler. On the UI I am greeted with a warm:

The scheduler does not appear to be running. The DAGs list may not update, and new tasks will not be scheduled.

How should I go at finding out why the two services don't talk to each other?

Update:

Strangely enough everything works well if I adapt my docker-entrypoint.sh to run all the steps within the same service with a trivial

exec airflow webserver -p 8080 & exec airflow scheduler

This makes following complicated as everything gets logged by the same container.

question from:https://stackoverflow.com/questions/65881570/airflow-webserver-doesnt-find-scheduler-in-docker-compose

Answer 1

Waitting for answers