How to output Airflow's scheduler log to stdout or S3 / GCS

Question

We're running Airflow cluster using puckel/airflow docker image with docker-compose. Airflow's scheduler container outputs its logs to /usr/local/airflow/logs/scheduler.

The problem is that the log files are not rotated and disk usage increases until the disk gets full. Dag for cleaning up the log directory is available but the DAG run on worker node and log directory on scheduler container is not cleaned up.

I'm looking for the way to output scheduler log to stdout or S3/GCS bucket but unable to find out. Is there any to output the scheduler log to stdout or S3/GCS bucket?

question from:https://stackoverflow.com/questions/65651301/how-to-output-airflows-scheduler-log-to-stdout-or-s3-gcs

Answer 1

remote_logging=True in airflow.cfg is the key. Please check the thread here for detailed steps.