python - Unable to start airflow worker due to `Address already in use`

Question

I try to set up airflow distributed mode (suing celeroy executor), but fails to start airflow worker. I get the following error. It seems there's another process (airflow serve_logs) using the same port as airflow worker. So no matter what value I set worker_log_server_port in airflow.cfg, they would always use the same port which cause the port conflict. Can anyone help me figure out what's wrong here ? Thanks

Traceback (most recent call last):
  File "/usr/lib/anaconda/envs/airflow/bin/airflow", line 37, in <module>
    args.func(args)
  File "/usr/lib/anaconda/envs/airflow/lib/python3.8/site-packages/airflow/utils/cli.py", line 76, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/anaconda/envs/airflow/lib/python3.8/site-packages/airflow/bin/cli.py", line 1240, in serve_logs
    flask_app.run(host='0.0.0.0', port=worker_log_server_port)
  File "/usr/lib/anaconda/envs/airflow/lib/python3.8/site-packages/flask/app.py", line 990, in run
    run_simple(host, port, self, **options)
  File "/usr/lib/anaconda/envs/airflow/lib/python3.8/site-packages/werkzeug/serving.py", line 1012, in run_simple
    inner()
  File "/usr/lib/anaconda/envs/airflow/lib/python3.8/site-packages/werkzeug/serving.py", line 956, in inner
    srv = make_server(
  File "/usr/lib/anaconda/envs/airflow/lib/python3.8/site-packages/werkzeug/serving.py", line 807, in make_server
    return ThreadedWSGIServer(
  File "/usr/lib/anaconda/envs/airflow/lib/python3.8/site-packages/werkzeug/serving.py", line 701, in __init__
    HTTPServer.__init__(self, server_address, handler)
  File "/usr/lib/anaconda/envs/airflow/lib/python3.8/socketserver.py", line 452, in __init__
    self.server_bind()
  File "/usr/lib/anaconda/envs/airflow/lib/python3.8/http/server.py", line 138, in server_bind
    socketserver.TCPServer.server_bind(self)
  File "/usr/lib/anaconda/envs/airflow/lib/python3.8/socketserver.py", line 466, in server_bind
    self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use

question from:https://stackoverflow.com/questions/65882087/unable-to-start-airflow-worker-due-to-address-already-in-use

Answer 1

Waitting for answers