Implement BPM-like long running processes with human interaction

Blendr.io is an iPaaS platform, focussed on building API integrations between cloud applications. In Blendr.io you build Blends (workflows) to implement data flows. These Blends typically run based on a trigger or a schedule and they run in the background until completed. This means that by default, Blends cannot ask for human input while they are running.

Note: Blends can, however, accept human input at the start of the execution, see the Input block for more information.

This article describes how you can implement a Blend that requests human input at any point in a flow, waits for that human input and proceeds with the flow after the human input was received. In other words, we show how BPM-like processes (Business Process Management workflows) can be implemented in Blendr.io that are potentially long-running and where human input is needed at certain points during the execution of the flow. We also refer to this as the man in the loop pattern.

We use the Blendr.io Data Store to create transactions (flows). We use the Data Store to persist these transactions until they are completed.

We create a scheduled Blend that runs e.g. every 5 minutes. The Blend consists of 4 parts that are executed on each run:

  • Create new transactions if needed, based on a certain data source
  • Request human input
  • Wait for human input and process the human input
  • Expire old transactions

Note that this logic can also be implemented in multiple separate Blends, that all read from/write to the same Data Store (the Data Store is used as a simple Message Bus).

In the following paragraph, we will cover the different components of the flow.

Create new transactions and request human input

Our example Blend will check for new companies created in a CRM, on each run. If no new companies are found nothing happens. If a new company is found, a transaction is created in the Data Store. alt text

We use the company id as the unique transaction id: alt text

In order to request human input, we created a form using a Form Builder tool. In this example we used Wufoo.

In the Blend, we send out an email to a user, to ask for human input. We add a link to the form in the email. We pre-populate the form by adding parameters in the query string. We make sure to also add the transaction id in the form: alt text

This is the full URL to the form, used in the above example:

https:// blendrio.wufoo.com /forms/m1qe8v0m160x8ua/def/ field1={$.listUpdatedCompaniesIncrementallyOnlyOnce.item.id} &field2={ $.listUpdatedCompaniesIncrementallyOnlyOnce.item.name }

The user will receive the email, click the link and the form will be opened, with certain fields pre-filled: alt text

When the user submits the form, a new form of entry is created in Wufoo. The next part of the Blend will process new form entries (form submissions).

Process human input

In the next part of the Blend (or in a separate Blend), we check for new form entries. When a new form entry is available, we check if we can find a matching transaction in the Data Store. If found, we will process the human input and set the transaction to processed.

Note that this means we wait for human input because the Blend will run e.g. every 5 minutes and as long as no new form entries are available, nothing will happen. alt text

For each new form entry, we get the transaction id from the form submission to find an existing item in the Data Store: alt text

In the current example, we use human input to update the name of the company in the CRM: alt text

Remember that the transaction id from the form (Field1) is also the company id in the CRM. Field2 in the form is the company name field.

Finally, we set the transaction to processed in the Data Store: alt text

This is only needed to exclude them from the next part of the Blend (setting old transactions to expired).

Expire old transactions

In the last part of the Blend (or in a separate Blend), we set old transactions to expired and we send out an alert by email:

alt text

Note that the block List items to process will not return items that have the status Processed.

For transactions that are not yet processed, we check their timestamp last update. If that is more than 24 hours ago, we set the transaction to processed (or failed) and we send out an alert email: alt text

This article provides a very simple example. It is, of course, possible to add more logic, e.g. send out reminders to users if no form submission is received or escalating to other users. This would be accomplished by updating the transaction in the Data Store, and keeping track of emails already sent out, etc. including the timestamps of these events.