python - How to run Airflow on Windows

Question

The usual instructions for running Airflow do not apply on a Windows environment:

# airflow needs a home, ~/airflow is the default,
# but you can lay foundation somewhere else if you prefer
# (optional)
export AIRFLOW_HOME=~/airflow

# install from pypi using pip
pip install airflow

# initialize the database
airflow initdb

# start the web server, default port is 8080
airflow webserver -p 8080

The Airflow utility is not available in the command line and I can't find it elsewhere to be manually added. How can Airflow run on Windows?

Answer 1

Three Basic Options

I went through a few iterations of this problem and documented them as I went along. The three things I tried were:

1. Install Airflow directly into Windows 10 - This attempt failed.
2. Install Airflow into Windows 10 WSL with Ubuntu - This worked great. Note that WSL is Windows Subsystem for Linux, which you can get for free in the Windows store.
3. Install Airflow into Windows 10 via Docker + Centos - This worked great as well.

Note that if you want to get it running as a Linux service, it is not possible for option number 2. It is possible for option number 3, but I didn't do it as it requires activating privileged containers in docker (which I wan't aware of when I started). Also, running a service in Docker is kind of against paradigm as each container should be a single process/unit of responsibility anyway.

Detailed Description of #2 - WSL Option

If you're gong for option 2, the basic steps are:

• Get WSL Ubuntu installed and opened up.
• Verify it comes with python 3.6.5 or so (python3 -version).
• Assuming it still does, add these packages so that installing PIP will work.
  • sudo apt-get install software-properties-common
  • sudo apt-add-repository universe
  • sudo apt-get update
• Install pip with:
  • sudo apt-get install python-pip (or python3-pip for Python 3)
• Run the following 2 commands to install airflow:
  • export SLUGIFY_USES_TEXT_UNIDECODE=yes
  • pip install apache-airflow (or pip3 for Python 3)
• Open a new terminal (I was surprised, but this seemed to be required).
• Init the airflow DB:
  • airflow initdb

After this, you should be good to go! The blog has more detail on many of these steps and rough timelines for how long setting up WSL takes, etc - so if you have a hard time dive in there some more.