python - Apache airflow unable to locate AWS credentials when using boto3 inside a DAG

Question

Running an instance of Airflow on ECS Fargate. The problem is I cannot run the code to call an existing Glue Job within the DAG. Below is the DAG script.

import boto3
import os
import logging
import time
import sys
import botocore
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.operators.bash_operator import BashOperator
from airflow.providers.amazon.aws.sensors.s3_key import S3KeySensor

default_args = {
    'owner': 'test',
    'start_date': datetime(2021, 1, 4),
    'depends_on_past': False,
    'provide_context': True
}

dag = DAG('run_glue_job',
          description='Executes Glue job.',
          schedule_interval=None,
          catchup=False,
          default_args=default_args)

def task_1(**kwargs):
    print('recieved trigger')

    glue = boto3.client('glue', 'us-east-1')
    response = glue.start_job_run(JobName='airflow-dev-job')
    print(response['JobRunId'])

    return response['JobRunId']

def task_2(**kwargs):
    print('send email message')

    return 'passed' 

t1 = PythonOperator(
    task_id = 'execute_glue_job',
    dag = dag,
    python_callable = task_2
)

t2 = PythonOperator(
    task_id = 'send_email_notification',
    dag = dag,
    python_callable = task_3
)

t1 >> t2

I am returning a credential error.

File "/usr/local/airflow/.local/lib/python3.7/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials

I tried setting the task_role_arn in the aws_default under the connections; however, I am still not able to get around the issue.

question from:https://stackoverflow.com/questions/65617277/apache-airflow-unable-to-locate-aws-credentials-when-using-boto3-inside-a-dag

Answer 1

Waitting for answers