Google Bigquery
Our Google Bigquery connector offers multiple ways of fetching data, depending on the number of records that need to be fetched.
Small amount of records (<100k records)
The easiest way to fetch data is to use the Do Select Query
endpoint. This allows you to enter a select query to fetch data from Google Bigquery. The standard batch size in which records will be returned is 200. This can be changed in the connector under Settings > Page Size. Increasing this number will also increase the speed of the endpoint, but responses can't be bigger than 10MB.
Big amount of records (>100k records)
To fetch a larger amount of records in an efficient way, a different approach is needed. This can be done with the combination of the endpoints Create Job
, Get Job
and List Data
. This will be more performant than the Do Select Query
endpoint but requires a bit more configuration.
- Create Job Enter your dataset id, project id, and select query.
- Get Job
Enter the job id and project id. They are both returned by the
Create Job
endpoint underCreate Job > Job Reference
. - List Data
Enter the project id, dataset id, and table id. These are all returned by the
Get Job
endpoint underGet Job > Configuration > Query > Destination Table
. Data returned by theList Data
endpoint will only contain the values from the fetched records (without the keys). The values are returned in the same order as they were specified in the query from theCreate Job
endpoint.
Did this page help you?
If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!