Implement BPM-like long running processes with human interaction

Qlik Application Automation for OEM is an iPaaS platform, focussed on building API integrations between cloud applications. In Qlik Application Automation for OEM you build automations (workflows) to implement data flows. These automations typically run based on a trigger or a schedule and they run in the background until completed. This means that by default, automations cannot ask for human input while they are running.

Note: automations can, however, accept human input at the start of the execution. See the Input block for more information.

This article describes how you can implement an automation that requests human input at any point in a flow, waits for that human input and proceeds with the flow after the human input was received. In other words, we show how BPM-like processes (Business Process Management workflows) can be implemented in Qlik Application Automation for OEM that are potentially long-running and where human input is needed at certain points during the execution of the flow. We also refer to this as the man in the loop pattern.

We use the Qlik Application Automation for OEM Data Store to create transactions (flows). We use the Data Store to persist these transactions until they are completed.

We create a scheduled automation that runs e.g. every 5 minutes. The automation consists of four parts that are executed on each run:

• Create new transactions if needed, based on a certain connector.
• Request human input.
• Wait for human input and process the human input.
• Expire old transactions.

Note that this logic can also be implemented in multiple separate automations that all read from/write to the same Data Store (the Data Store is used as a simple Message Bus).

In the following paragraph, we will cover the different components of the flow.

Create new transactions and request human input

Our example automation will check for new companies created in a CRM on each run. If no new companies are found, nothing happens. If a new company is found, a transaction is created in the Data Store.

We use the company id as the unique transaction id:

In order to request human input, we created a form using a Form Builder tool. In this example we used Wufoo.

In the Blend, we send out an email to a user to ask for human input. We add a link to the form in the email. We pre-populate the form by adding parameters in the query string. We make sure to also add the transaction id in the form:

This is the full URL to the form, used in the above example:

https:// blendrio.wufoo.com /forms/m1qe8v0m160x8ua/def/
      field1={$.listUpdatedCompaniesIncrementallyOnlyOnce.item.id}
   &field2={ $.listUpdatedCompaniesIncrementallyOnlyOnce.item.name }

The user will receive the email, click the link and the form will be opened, with certain fields pre-filled:

When the user submits the form, a new form of entry is created in Wufoo. The next part of the automation will process new form entries (form submissions).

Process human input

In the next part of the automation (or in a separate Blend), we check for new form entries. When a new form entry is available, we check if we can find a matching transaction in the Data Store. If found, we will process the human input and set the transaction to processed.

Note that this means we wait for human input because the automation will run e.g. every 5 minutes and as long as no new form entries are available, nothing will happen.

For each new form entry, we get the transaction id from the form submission to find an existing item in the Data Store:

In the current example, we use human input to update the name of the company in the CRM:

Remember that the transaction id from the form (Field1) is also the company id in the CRM. Field2 in the form is the company name field.

Finally, we set the transaction to processed in the Data Store:

This is only needed to exclude them from the next part of the automation (setting old transactions to expired).

Expire old transactions

In the last part of the automation (or in a separate Blend), we set old transactions to expired and we send out an alert by email:

Note that the block List items to process will not return items that have the status Processed.

For transactions that are not yet processed, we check their timestamp last update. If that is more than 24 hours ago, we set the transaction to processed (or failed) and we send out an alert email:

This article provides a very simple example. It is, of course, possible to add more logic, e.g. send out reminders to users if no form submission is received or escalating to other users. This would be accomplished by updating the transaction in the Data Store, and keeping track of emails already sent out, etc. including the timestamps of these events.