Data Store overview

The Blendr.io Data Store is a database that can be used to store any type of items, e.g. Contacts, Products from a webshop or Orders. It can be used as permanent data storage or as a temporary data cache (*).

A Data Store is especially useful if you want to process items, and you want to keep track of the status for each item. This is called the exactly once processing pattern.

There are other ways of processing items exactly once in Blendr.io. Most connectors have blocks to retrieve data incrementally, by using an internal pointer. The Data Store however, is more advanced, since it keeps a status of each individual item and allows retrying individual items.

(*) Note on data caching

It is also possible to configure caching on individual blocks in a Blend. The Data Store however gives you more control since you choose which items you add to the Data Store etc.

Creating a Data Store

In Blendr.io, go to the menu in the top right corner and select My Data Stores. Now you can add a Data Store, change the settings of an existing Data Store, and see the items inside a Data Store. alt text

Click the database icon to view items in a Data Store. Click the pencil icon to change the settings of a Data Store.

Once you have created a Data Store, you can add items to it in a Blend, using the Data Store block Add item.

Creating a Data Store in a Blend or Template

If you build a Template that uses a Data Store, you have to create the Data Store inside the Blend/Template. You can do this with the block Create Datastore. Simply put the block at the start of the Blend, under the Start block:

alt text

This block will be executed on each run of the Blend, if the Data Store already exists (based on the name), it will do nothing and return the details of the existing Data Store in the response. This way you can reference the id of the Data Store and use it throughout your Blend: alt text

Notes:

  • Do not put the block Create Datastore in the Setup flow of a Blend, because in that case you will be unable to access the id of the Data Store when your Blend is executed.
  • Make sure the name of your Data Store is unique, especially if multiple instances of a Template are used inside one customer account. Add e.g. a unique identifier in the name (e.g. bundle guid) to make sure that 2 Template instances will each have their own Data Store.

Status of items

Each item in a Data Store has a status:

  • Ready to process: a new item, not yet processed by a Blend
  • Processed: an item that has been processed by a Blend
  • Error: an item that has been processed but the processing failed. It will be retried on the next run of the Blend.
  • Failed: an item that will not be retried for processing

Avoiding duplicates

The Data Store is a great way to avoid duplicates. Each item in the Data Store has a unique key. The unique key can be anything, but typically it is the id of the item, e.g. an order id when you are processing orders from a web shop.

For each Data Store, you can configure how the Data Store handles adding new items with the same unique key. See setting Action to perform on duplicate. You can choose between 3 settings:

  • Update: when you add an item with a unique key that already exists in the Data Store, the existing item will be updated, its status will NOT change. So if the existing item has status Processed, it will NOT be processed again after the update.
  • Replace: when you add an item with a unique key that already exists in the Data Store, the existing item will be replaced with this new item, and the new item will have a new status Ready to process.
  • Do nothing: when you add an item with a unique key that already exists in the Data Store, the new item will be ignored.

Example scenario: process orders exactly once

A typical scenario is that you have a Blend that reads items from a data source, and writes them to the Data Store. For example you may read orders from your webshop every day and store them in a Data Store for further processing.

The Data Store will automatically avoid creating duplicates, so in the Blend you could simply retrieve ALL orders every time and add them to the Data Store over and over. There's no need to read them incrementally. In the Data Store, every order will only exist once.

For this scenario, make sure to set Action to perform on duplicate to "Do nothing".

A second Blend can read orders from the Data Store, process them and set the status of each order to Processed.

Add items to a Data Store

Use the Data Store block Add item to add an item to a Data Store. Here's an example that retrieves attendees of an event from Eventbrite, and adds the attendees one by one to a Data Store called Event attendees. Note that the whole item (object) of each item is sent to the Data Store (using the input field Data), and that the attendee id from Eventbrite is used as unique key to avoid duplicates (input field Uniquekey). alt text

Add items to a Data Store in bulk

Use the block Add items (as opposed to Add item) to add items in Bulk, this will dramatically increase the speed of inserting/updating a high number of records.

Use a block Loop batch to create batches from a list of records. Example: alt text

The batch from the loop is the input of the block Add items: alt text

Make sure that each item (object) in the list has a key with the name as defined under Uniquekey. Note that in the block Add items, the input Uniquekey is a fieldname, not a value. You can use dot-notation for nested properties, e.g. contact.id.

Existing records in the Data Store will be updated based on this field. In the above example, items will be updated in the Data Store based on the id field from each contact from Salesforce.

Viewing items in the Data Store

The above Blend could be scheduled to run every hour, and it will not create duplicates thanks to the unique key. You can view the items in the Data Store by going to My Data Stores, selecting the Data Store Event attendees, and clicking on the Database icon: alt text

You can click on one item to see the item in full: alt text

Processing items from a Data Store

You can create a Blend to process items from a Data Store. Use the block List items to process to get items from a Data Store with status Ready to process and items with status Error.

Note that the block List items to process will not include items in status Processed and Failed, but it will include items in status Error in order to retry processing the item.

In the following example, we are processing items and adding them to a Google Sheet. If the insert succeeded, we set the status of the item to Processed using the block Update item status: alt text

Automatic update if item status to Processed has been deprecated

The behaviour where items are set to Processed automatically has been deprecated (Q4 2019), you have to set an item to processed explicitly using the block Update item status.

On the next run of the Blend, the block List items to Process will output all items in the Data Store with status Ready to process AND status Error. This allows you to retry processing items.

You could implement a counter in the Data Store for each retry, and said the status of the item to Failed after a number of retries.

Keywords for searching: data store, datastore, data queue, dataqueue.