How connectors handle data, paging, rate limits and errors in API's

This article describes the internal operations of the Qlik Application Automation for OEM connectors.

The Qlik Application Automation for OEM connectors use the API's of the connected applications. Depending on the application, this could be a REST API, a SOAP Webservice, a GraphQL API etc. Each endpoint of the API becomes a building block in the automation editor.

For example, Salesforce has a REST API, with an endpoint to create a new contact. This means that you will find a create contact block in the Salesforce connector.

Data formats

Qlik Application Automation for OEM can handle different formats such as JSON and XML, but the response of an API will always be converted to JSON, which means that inside the automations you always work with JSON data.

Paging

The Qlik Application Automation for OEM connectors handle the paging in the connected API's. If you use a block list contacts, the response will be a list of all the contacts. Behind the scenes, the connector will perform paging to retrieve all the data from the API.

Iterators

Note that the connector uses iterators when working with lists. If you use a block list contacts and there are 1 million contacts, these contacts will not be fetched all at once. Instead, the data will be retrieved page by page, while the data is being processed in the automation.

500 errors and API downtime

If the API of the connected application throws a 500 error when reading data, the connector will automatically retry for up to 5 minutes with increasing pauses. The automation will keep running, the execution of the block (e.g. get contact or list contacts) will just take longer.

If the API still does not send a response after 5 minutes, the request will fail. The block will output an error in the automation.

The behaviour is the same when the API is down for less than 5 minutes. Note that these retries are only performed when reading data, NOT when writing data.

Rate limits

The Qlik Application Automation for OEM connectors automatically handle rate limits imposed by the API. The connector will automatically retry eternally until the automation times out. The automation will keep running, the execution of the block (e.g. get contact or list contacts) will just take longer.