Making data request with synchronous, asynchronous, batch, and auto mode

You can make the following request types with Qlik Web Connectors:

  • Synchronous
  • Asynchronous
  • Batch
  • Auto

For/next loops are often used to run a large number of API requests with Qlik Web Connectors. These modes let you extract the maximum performance from Qlik Web Connectors and minimize load time of the QlikView or Qlik Sense application.

For example, if you run a large number of text messages through one of the available sentiment analysis connectors, like Repustate or Sentiment140 it can result in a long load time.

Example:  

LET noRows = NoOfRows('Tweets');   for i=0 to $(noRows)-1   let id = peek('Search_id', $(i), 'Tweets'); let textEncoded = peek('text_urlEncoded', $(i), 'Tweets');   Sentiment: LOAD '$(id)' as Search_id, status as sentiment_status, score as sentiment_score FROM [http://localhost:5555/web/connector/TextAnalyserV2/?table=Sentiment_Sentiment140&text=$(textEncoded)&appID=$(vQWCAppId)format=qvx] (qvx); next

In the script example above, each message is processed sequentially. This means that the sentiment score for the next message cannot be calculated until the current request has completed. If each request takes 250 ms, then it will take 40 minutes to score the sentiment score of 10,000 Tweets.

Many APIs allow multiple requests to be made in parallel from the same client so we can use different request types to improve the performance of the Qlik Web Connectors.

These requests are handled by separate processes on the server or cloud, allowing more requests to be processed per second. The Asynchronous mode allows us to take advantage of this feature.

Some APIs also allow multiple requests to be combined into a single API call, for example sending 1000 text messages in a single request to be processed for sentiment. This is referred to as sending requests in a batch and it is the Batch mode that allows us to take advantage of this feature.

Synchronous requests

Synchronous requests are the default mechanism and all tables of all connectors support this. A request to Qlik Web Connectors results in one or more sequential requests to the underlying API and this is the script you receive when copying the standard generated script from Qlik Web Connectors.

For simple for/next loops, It is possible to simplify the script by using a temporary table.

Example: For/next loop

LET noRows = NoOfRows('Tweets');   for i=0 to $(noRows)-1   let id = peek('Search_id', $(i), 'Tweets'); let textEncoded = peek('text_urlEncoded', $(i), 'Tweets');   Sentiment: LOAD '$(id)' as Search_id, status as sentiment_status, score as sentiment_score FROM [http://localhost:5555/web/connector/TextAnalyserV2/?table=Sentiment_Sentiment140&text=$(textEncoded)&appID=$(vQVSourceAppId)format=qvx] (qvx); next

Example: Temporary Params table

let vQWCAppId = 'test_app'; let vWorkingFolder = 'c:\QlikWebConnectors\Temp\'; let vParamsFile = '$(vWorkingFolder)$(vQWCAppId)' & '_SentimentArgs.txt';   Params: LOAD Search_id as rowKey, Search_text as text resident Tweets;   store Params into '$(vParamsFile)' (txt); drop table Params; // Optional.   Sentiment: LOAD rowKey as Search_id, status as Sentiment_status, score as Sentiment_score, language as Sentiment_language FROM [http://localhost:5555/web/connector/TextAnalyserV2/?table=Sentiment_Sentiment140& processParamsSync=$(vParamsFile)&appID=$(vQWCAppId)&format=qvx] (qvx);

Create a temporary table named Params that contains the same parameters that are passed in each iteration of the for/next loop above.

There are several important changes to the new script:

  • An additional column named rowKey, which is set to the Tweet ID, so we can link it back into the data model when the results come back.
  • The Params table is saved as a CSV file. It is important that the user account that is running the Qlik Web Connectors has read access to this file.
Note: The parameters in the Params table should not be URL encoded.
Warning: There are certain parameters you cannot put in the Params table. In particular, the table or format parameters cannot be used. These should be in the final Qlik Web Connectors request. For a complete list of parameters, see Special parameters

The script makes a request to Qlik Web Connectors but instead of passing the usual parameters, it passes the path of the previously created CSV file using the processParamsSync parameter. We also read back the rowKey and alias ID back to the Tweet ID.

You can apply this method to almost any Qlik Web Connectors request, and it is the basis of the Asynchronous, Batch and Auto modes.

Note: You can use rowKey2 and rowKey3 parameters with processParamsSync.

Asynchronous requests

To make asynchronous requests, you simply change the parameter name processParamsSync into processParamsAsync:

Sentiment: LOAD rowKey as Search_id, status as Sentiment_status, score as Sentiment_score, language as Sentiment_language FROM [http://localhost:5555/web/connector/TextAnalyserV2/?table=Sentiment_Sentiment140& processParamsAsync=$(vParamsFile)&appID=$(vQWCAppId)&format=qvx] (qvx);

Qlik Web Connectors attempts to run the requests on a number of parallel threads and the above call only completes when all the requests have finished.

Note: You can use rowKey2 and rowKey3 parameters with processParamsAsynch.

Specifying the number of threads

By default, Qlik Web Connectors use a maximum of 30 parallel threads.

You can set an alternative value, between 1 and 50, by including a maxThreads=VALUE parameter in the request.

Batch requests

To make batch requests, you simply change the parameter name processParamsSync into processParamsBatch:

Sentiment: LOAD rowKey as Search_id, status as Sentiment_status, score as Sentiment_score, language as Sentiment_language FROM [http://localhost:5555/web/connector/TextAnalyserV2/?table=Sentiment_Sentiment140& processParamsBatch=$(vParamsFile)&appID=$(vQWCAppId)&format=qvx] (qvx);

Qlik Web Connectors attempt to run the requests using the Batch feature that the API provides, and the above call only completes when all the requests have finished.

Note: You can use rowKey2 and rowKey3 parameters with processParamsBatch.

Auto requests

If you want Qlik Web Connectors to decide the best method, you can change the parameter name processParamsSync into processParamsAuto:

Sentiment: LOAD rowKey as Search_id, status as Sentiment_status, score as Sentiment_score, language as Sentiment_language FROM [http://localhost:5555/web/connector/TextAnalyserV2/?table=Sentiment_Sentiment140& processParamsAuto=$(vParamsFile)&appID=$(vQWCAppId)&format=qvx] (qvx);

When you use auto requests, Qlik Web Connectors uses the following rules:

  • If the table supports batch mode, Qlik Web Connectors uses this method.
  • If the table explicitly supports asynchronous mode, Qlik Web Connectors uses this method.
  • Otherwise, Qlik Web Connectors uses the standard synchronous mode.
Note: You can use rowKey2 and rowKey3 parameters with processParamsAuto.
Note: Auto request mode is not recommended because only a few connectors explicitly support asynchronous mode, so Qlik Web Connectors would fall back on the slowest synchronous mode. We recommended that you look at each table you are using and decide whether to use synchronous, asynchronous or batch mode.

Using URLs instead of file paths

When using a processParamsxxx parameter, as well as passing a file location to it, you can also pass a URL beginning with http or https.

Special parameters

Certain parameters can be used by any Qlik Web Connectors connector. These parameters should not be placed in the Params CSV file but specified in the request URL for the table.

table The name of the table requested. Should only be included in the final request.
format Tells Qlik Web Connectors what format to return the data in. As all the data for a given request must be returned in the same format this cannot appear in the Params file. In almost all cases this should be set to qvx.
appID An identifier for your application passed to Qlik Web Connectors allowing it to be logged alongside any errors which may occur.
noCache Causes Qlik Web Connectors to ignore the cache.
requestStamp Allows you to specify an id for a request which you can then use to access follow on data from another table related to the request.

Did this information help you?

Thanks for letting us know. Is there anything you'd like to tell us about this topic?

Can you tell us why it did not help you and how we can improve it?