Google Bigquery

Our Google Bigquery connector offers multiple ways of fetching data, depending on the number of records that need to be fetched.

Small amount of records (<100k records)

The easiest way to fetch data is to use the Do Select Query endpoint. This allows you to enter a select query to fetch data from Google Bigquery. The standard batch size in which records will be returned is 200. This can be changed in the connector under Settings > Page Size. Increasing this number will also increase the speed of the endpoint, but responses can't be bigger than 10MB.

Big amount of records (>100k records)

To fetch a larger amount of records in an efficient way, a different approach is needed. This can be done with the combination of the endpoints Create Job, Get Job and List Data. This will be more performant than the Do Select Query endpoint but requires a bit more configuration.

1. Create Job Enter your dataset id, project id, and select query.
2. Get Job Enter the job id and project id. They are both returned by the Create Job endpoint under Create Job > Job Reference.
3. List Data Enter the project id, dataset id, and table id. These are all returned by the Get Job endpoint under Get Job > Configuration > Query > Destination Table. Data returned by the List Data endpoint will only contain the values from the fetched records (without the keys). The values are returned in the same order as they were specified in the query from the Create Job endpoint.