Data Store overview

The Qlik Application Automation for OEM Data Store is a database that can be used to store any type of items, e.g. Contacts, Products from a webshop or Orders. It can be used as permanent data storage or as a temporary data cache (*).

A Data Store is especially useful if you want to process items, and you want to keep track of the status for each item. This is called the exactly once processing pattern.

There are other ways of processing items exactly once inQlik Application Automation for OEM. Most connectors have blocks to retrieve data incrementally, by using an internal pointer. The Data Store however, is more advanced, since it keeps a status of each individual item and allows retrying individual items.

Creating a Data Store

InQlik Application Automation for OEM, go to the menu in the top right corner and select My Data Stores. Now you can add a Data Store, change the settings of an existing Data Store, and see the items inside a Data Store.

Click the database icon to view items in a Data Store. Click the pencil icon to change the settings of a Data Store.

Once you have created a Data Store, you can add items to it in an automation, using the Data Store block Add item.

Creating a Data Store in an automation or Template

If you build a Template that uses a Data Store, you have to create the Data Store inside the Blend/Template. You can do this with the block Create Datastore. Simply put the block at the start of the Blend, under the Start block:

This block will be executed on each run of the Blend, if the Data Store already exists (based on the name), it will do nothing and return the details of the existing Data Store in the response. This way you can reference the id of the Data Store and use it throughout your Blend:

Notes:

• Do not put the block Create Datastore in the Setup flow of an automation, because in that case you will be unable to access the id of the Data Store when your automation is executed.
• Make sure the name of your Data Store is unique, especially if multiple instances of a Template are used inside one customer account. Add e.g. a unique identifier in the name (e.g. bundle guid) to make sure that two Template instances will each have their own Data Store.

Status of items

Each item in a Data Store has a status:

• Ready to process: a new item, not yet processed by an automation
• Processed: an item that has been processed by an automation
• Error: an item that has been processed but the processing failed. It will be retried on the next run of the Blend.
• Failed: an item that will not be retried for processing

Avoiding duplicates

The Data Store is a great way to avoid duplicates. Each item in the Data Store has a unique key. The unique key can be anything, but typically it is the id of the item, e.g. an order id when you are processing orders from a web shop.

For each Data Store, you can configure how the Data Store handles adding new items with the same unique key. See setting Action to perform on duplicate. You can choose between 3 settings:

• Update: when you add an item with a unique key that already exists in the Data Store, the existing item will be updated, and its status will NOT change. So if the existing item has status Processed, it will NOT be processed again after the update.
• Replace: when you add an item with a unique key that already exists in the Data Store, the existing item will be replaced with this new item, and the new item will have a new status Ready to process.
• Do nothing: when you add an item with a unique key that already exists in the Data Store, the new item will be ignored.

Example scenario: process orders exactly once

A typical scenario is that you have an automation that reads items from a connector, and writes them to the Data Store. For example, you may read orders from your webshop every day and store them in a Data Store for further processing.

The Data Store will automatically avoid creating duplicates, so in the automation you could simply retrieve ALL orders every time and add them to the Data Store over and over. There's no need to read them incrementally. In the Data Store, every order will only exist once.

For this scenario, make sure to set Action to perform on duplicate to "Do nothing".

A second automation can read orders from the Data Store, process them and set the status of each order to Processed.

Add items to a Data Store

Use the Data Store block Add item to add an item to a Data Store. Here's an example that retrieves attendees of an event from Eventbrite, and adds the attendees one by one to a Data Store called Event attendees. Note that the whole item (object) of each item is sent to the Data Store (using the input field Data), and that the attendee id from Eventbrite is used as unique key to avoid duplicates (input field Uniquekey).

Add items to a Data Store in bulk

Use the block Add items (as opposed to Add item) to add items in Bulk. This will dramatically increase the speed of inserting/updating a high number of records.

Use a block Loop batch to create batches from a list of records. Example:

The batch from the loop is the input of the block Add items:

Make sure that each item (object) in the list has a key with the name as defined under Uniquekey. Note that in the block Add items, the input Uniquekey is a fieldname, not a value. You can use dot-notation for nested properties, e.g. contact.id.

Existing records in the Data Store will be updated based on this field. In the above example, items will be updated in the Data Store based on the id field from each contact from Salesforce.

Viewing items in the Data Store

The above automation could be scheduled to run every hour, and it will not create duplicates thanks to the unique key. You can view the items in the Data Store by going to My Data Stores, selecting the Data Store Event attendees, and clicking on the Database icon:

You can click on one item to see the item in full:

Processing items from a Data Store

You can create an automation to process items from a Data Store. Use the block List items to process to get items from a Data Store with status Ready to process and items with status Error.

Note that the block List items to process will not include items in status Processed and Failed, but it will include items in status Error in order to retry processing the item.

In the following example, we are processing items and adding them to a Google Sheet. If the insert succeeded, we set the status of the item to Processed using the block Update item status:

On the next run of the Blend, the block List items to Process will output all items in the Data Store with status Ready to process AND status Error. This allows you to retry processing items.

You could implement a counter in the Data Store for each retry, and said the status of the item to Failed after a number of retries.

Keywords for searching: data store, datastore, data queue, dataqueue.