python - What is the best way to store login credentials on Airflow?

Question

I found out there are lot of ways to store it as variables, hooks and other ways using encryption. I would like to know what's the best way to do it.

Answer 1

Currently there 2 ways of storing secrests:

1) Airflow Variables: Value of a variable will be hidden if the key contains any words in (‘password’, ‘secret’, ‘passwd’, ‘authorization’, ‘api_key’, ‘apikey’, ‘access_token’) by default, but can be configured to show in clear-text as shown in the image below.

However, there is a known-bug where anyone with an access to UI can export all the variables which will expose the secrets.

2) Airflow Connections:

You can use the Passwords field in Airflow connections which will encrypt that field if you had installed the crypto package (pip install apache-airflow[crypto]). The password field would just appear as blank in the UI as shown in the screenshot.

More on Securing connections: https://airflow.apache.org/howto/secure-connections.html

I recommend the 2nd approach as even if someone gets access to the UI, he/she won't be able to get your secrets. Keep in mind though that you need to install the crypto package for this.

You can then access the secrets as below:

from airflow.hooks.base_hook import BaseHook

connection = BaseHook.get_connection(CONN_ID)
slack_token = connection.password

You can set the CONN_ID as the name of your connection.