Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
318 views
in Technique[技术] by (71.8m points)

python - How to run bash script file in Airflow

I have a bash script that creates a file (if it does not exist) that I want to run in Airflow, but when I try it fails. How do I do this?

#!/bin/bash
#create_file.sh

file=filename.txt

if [ ! -e "$file" ] ; then
    touch "$file"
fi

if [ ! -w "$file" ] ; then
    echo cannot write to $file
    exit 1
fi

and here's how I'm calling it in Airflow:

create_command = """
 ./scripts/create_file.sh
"""
t1 = BashOperator(
        task_id= 'create_file',
        bash_command=create_command,
        dag=dag
)

lib/python2.7/site-packages/airflow/operators/bash_operator.py", line 83, in execute
    raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed
See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

From the tutorial this is OK:

t2 = BashOperator(
    task_id='sleep',
    bash_command='sleep 5',
    retries=3,
    dag=dag)

But you're passing a multi-line command to it

create_command = """
 ./scripts/create_file.sh
"""

should be

create_command = "./scripts/create_file.sh "

Moreover, you also have to make sure that you are in the correct directory to avoid cryptic errors. Do it like this for example:

create_command = "./scripts/create_file.sh "
if os.path.exists(create_command):
   t1 = BashOperator(
        task_id= 'create_file',
        bash_command=create_command,
        dag=dag
   )
else:
    raise Exception("Cannot locate {}".format(create_command))

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...