google cloud platform - Cannot access BigQuery from Compute Engine

Question

I have setup a standard Debian Linux VM via Compute Engine on GCP. The VM does not have an external IP address. I can connect to it via ssh by using the browser. I allowed incomming ssh (port 22) traffic and all outgoing traffic. I have tested BigQuery by executing queries via the browser interface and it works. I have configured BigQuery to be enabled for the VM via settings -> Cloud API access scopes. Now I would like to do a simple thing as the following:

bq show bigquery-public-data:samples.shakespeare

But nothing happens. I tried to do the following to get more info:

bq --apilog=stdout show bigquery-public-data:samples.shakespeare

Output is the following:

I0106 15:29:47.271125 140258687915840 bigquery_client.py:1205] Requesting discovery document from https://www.googleapis.com/discovery/v1/apis/bigquery/v2/rest
I0106 15:29:47.271456 140258687915840 transport.py:158] Attempting refresh to obtain initial access_token

Nothing more happens. Any ideas what the issue could be?

After reading the documentation it seems to me that the connection via the BigQuery command line tool should work by itself.

Answer 1

Firstly, why it doesn't work. In fact, when you use bq CLI, it's only a wrapper that call the BigQuery apis: https://bigquery.googleapis.com. The domain name is public. The Compute Engine try to resolve it on the public internet. But the compute engine doesn't have public IP and can't go on internet (internet server doesn't know how to route back the answer, because the VM is not reachable!)

Then, how to solve. 2 solutions:

• You can set a Cloud NAT on your Compute Engine and thus, grant it a shareable public IP, only used to initiate outgoing traffic
• You can use a not well known trick: activate the Google private API access in your subnet. For this, note the subnet of your Compute engine. Then go to VPC and select this subnet. Edit it and set to ON the private Google access.