python - Getting a google.auth.exceptions.RefreshError when trying to read a file from cloud storage into Pandas

Question

I am trying to read a cloud storage file into a Pandas dataframe locally and then load it into a Big Query table using Python but I get the following error:

google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', '{ "error": "invalid_grant", "error_description": "Bad Request" }')

My code is:

import pandas as pd
from google.cloud import storage
from google.oauth2 import service_account
import gcsfs

scopes=["https://www.googleapis.com/auth/bigquery","https://www.googleapis.com/auth/cloud-platform"]

file='<<filename>>.json'
credentials = service_account.Credentials.from_service_account_file(file, scopes=scopes)

def run(file):
    client = storage.Client.from_service_account_json(file)
    bucket = client.bucket('my_bucket')
    blobs = bucket.list_blobs()
    list_temp_raw = []
    for file in blobs:
        filename = file.name
        temp = pd.read_csv('gs://<<>my_bucket>>/' + filename, encoding='utf-8')
        list_temp_raw.append(temp)
    df = pd.concat(list_temp_raw, encoding='utf-8')
    df.to_gbq('<<dataset>>.<<table>>',if_exists='append',credentials=credentials)

run(file)

It is the temp = pd.read_csv('gs://<<>my_bucket>>/' + filename, encoding='utf-8') causing the error:

Traceback (most recent call last):
  File "test-load-bq.py", line 23, in <module>
    run(file)
  File "test-load-bq.py", line 18, in run
    temp = pd.read_csv('gs://project-test/' + filename, encoding='utf-8')
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/pandas/io/parsers.py", line 676, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/pandas/io/parsers.py", line 431, in _read
    filepath_or_buffer, encoding, compression
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/pandas/io/common.py", line 192, in get_filepath_or_buffer
    filepath_or_buffer, encoding=encoding, compression=compression, mode=mode
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/pandas/io/gcs.py", line 17, in get_filepath_or_buffer
    filepath_or_buffer = fs.open(filepath_or_buffer, mode)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/fsspec/spec.py", line 936, in open
    **kwargs
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/gcsfs/core.py", line 1287, in _open
    **kwargs,
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/gcsfs/core.py", line 1412, in __init__
    **kwargs,
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/fsspec/spec.py", line 1257, in __init__
    self.details = fs.info(path)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/fsspec/asyn.py", line 121, in wrapper
    return maybe_sync(func, self, *args, **kwargs)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/fsspec/asyn.py", line 100, in maybe_sync
    return sync(loop, func, *args, **kwargs)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/fsspec/asyn.py", line 71, in sync
    raise exc.with_traceback(tb)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/fsspec/asyn.py", line 55, in f
    result[0] = await future
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/gcsfs/core.py", line 825, in _info
    return await self._get_object(path)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/gcsfs/core.py", line 607, in _get_object
    res = await self._call("GET", "b/{}/o/{}", bucket, key, json_out=True)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/gcsfs/core.py", line 503, in _call
    self.maybe_refresh()
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/gcsfs/core.py", line 404, in maybe_refresh
    self.credentials.refresh(req)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/google/oauth2/credentials.py", line 198, in refresh
    self._scopes,
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/google/oauth2/_client.py", line 248, in refresh_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/google/oauth2/_client.py", line 124, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', '{
  "error": "invalid_grant",
  "error_description": "Bad Request"
}')

I have given my service account the following roles: Storage Admin & BigQuery Admin

Any ideas why the error is happening?

question from:https://stackoverflow.com/questions/65928734/getting-a-google-auth-exceptions-refresherror-when-trying-to-read-a-file-from-cl

Answer 1

Waitting for answers