Our Google Bigquery connector offers multiple ways of fetching data, depending on the number of records that need to be fetched.
The easiest way to fetch data is to use the
Do Select Query endpoint. This allows you to enter a select query to fetch data from Google Bigquery. The standard batch size in which records will be returned is 200. This can be changed in the connector under Settings > Page Size. Increasing this number will also increase the speed of the endpoint, but responses can't be bigger than 10MB.
To fetch a larger amount of records in an efficient way, a different approach is needed. This can be done with the combination of the endpoints
Get Job and
List Data. This will be more performant than the
Do Select Query endpoint but requires a bit more configuration.
- Create Job Enter your dataset id, project id, and select query.
- Get Job
Enter the job id and project id. They are both returned by the
Create Jobendpoint under
Create Job > Job Reference.
- List Data
Enter the project id, dataset id, and table id. These are all returned by the
Get Jobendpoint under
Get Job > Configuration > Query > Destination Table. Data returned by the
List Dataendpoint will only contain the values from the fetched records (without the keys). The values are returned in the same order as they were specified in the query from the