Making data request with synchronous, asynchronous, batch, and auto mode
You can make the following request types with Qlik Web Connectors:
For/next loops are often used to run a large number of API requests with Qlik Web Connectors. These modes let you extract the maximum performance from Qlik Web Connectors and minimize load time of the QlikView or Qlik Sense application.
For example, if you run a large number of text messages through one of the available sentiment analysis connectors, like Sentiment140, it can result in a long load time.
In the script example above, each message is processed sequentially. This means that the sentiment score for the next message cannot be calculated until the current request has completed. If each request takes 250 ms, then it will take 40 minutes to score the sentiment score of 10,000 Tweets.
Many APIs allow multiple requests to be made in parallel from the same client so we can use different request types to improve the performance of the Qlik Web Connectors.
These requests are handled by separate processes on the server or cloud, allowing more requests to be processed per second. The Asynchronous mode allows us to take advantage of this feature.
Some APIs also allow multiple requests to be combined into a single API call, for example sending 1000 text messages in a single request to be processed for sentiment. This is referred to as sending requests in a batch and it is the Batch mode that allows us to take advantage of this feature.
Synchronous requests are the default mechanism and all tables of all connectors support this. A request to Qlik Web Connectors results in one or more sequential requests to the underlying API and this is the script you receive when copying the standard generated script from Qlik Web Connectors.
For simple for/next loops, It is possible to simplify the script by using a temporary table.
Example: For/next loop
Example: Temporary Params table
Create a temporary table named Params that contains the same parameters that are passed in each iteration of the for/next loop above.
There are several important changes to the new script:
- An additional column named rowKey, which is set to the Tweet ID, so we can link it back into the data model when the results come back.
- The Params table is saved as a CSV file. It is important that the user account that is running the Qlik Web Connectors has read access to this file.
The script makes a request to Qlik Web Connectors but instead of passing the usual parameters, it passes the path of the previously created CSV file using the processParamsSync parameter. We also read back the rowKey and alias ID back to the Tweet ID.
You can apply this method to almost any Qlik Web Connectors request, and it is the basis of the Asynchronous, Batch and Auto modes.
To make asynchronous requests, you simply change the parameter name processParamsSync into processParamsAsync:
Qlik Web Connectors attempts to run the requests on a number of parallel threads and the above call only completes when all the requests have finished.
Specifying the number of threads
By default, Qlik Web Connectors use a maximum of 30 parallel threads.
You can set an alternative value, between 1 and 50, by including a maxThreads=VALUE parameter in the request.
To make batch requests, you simply change the parameter name processParamsSync into processParamsBatch:
Qlik Web Connectors attempt to run the requests using the Batch feature that the API provides, and the above call only completes when all the requests have finished.
If you want Qlik Web Connectors to decide the best method, you can change the parameter name processParamsSync into processParamsAuto:
When you use auto requests, Qlik Web Connectors uses the following rules:
- If the table supports batch mode, Qlik Web Connectors uses this method.
- If the table explicitly supports asynchronous mode, Qlik Web Connectors uses this method.
- Otherwise, Qlik Web Connectors uses the standard synchronous mode.
Using URLs instead of file paths
When using a processParamsxxx parameter, as well as passing a file location to it, you can also pass a URL beginning with http or https.
Certain parameters can be used by any Qlik Web Connectors connector. These parameters should not be placed in the Params CSV file but specified in the request URL for the table.
|table||The name of the table requested. Should only be included in the final request.|
|format||Tells Qlik Web Connectors what format to return the data in. As all the data for a given request must be returned in the same format this cannot appear in the Params file. In almost all cases this should be set to qvx.|
|appID||An identifier for your application passed to Qlik Web Connectors allowing it to be logged alongside any errors which may occur.|
|requestStamp||Allows you to specify an id for a request which you can then use to access follow on data from another table related to the request.|