Skip to main content Skip to complementary content

Deduplicate list block

The deduplicate list block removes duplicate items from a list. If the list contains three identical items, the block will leave one instance of that item. Items are compared based on a defined key.

deduplicate list block

A deduplicate list block is commonly used in the following situations:

  • When an automation merges lists resulting in duplicate items.
  • When an automation requires user input resulting in duplicate items in a list.

Configuring a deduplicate list block

Configure the deduplicate list block to compare items within a single list.

Information note

List blocks do not use iterators. This means that lists are loaded in memory before doing the processing of any data. If your automation contains large lists, the automation might fail with an out-of-memory error. Instead, use a loop block to iterate through large lists.

Inputs

The deduplicate list block requires a list and a key. A list is an array of objects. The list can be a field mapped list from a preceding block. The key should be a unique key that appears in the list.

Deduplicate list block input fields

deduplicate list block inputs

Example list

[
      {"email": "sven@acme.com", "name": "Sven Doe"},
      {"email": "svenja@acme.com", "name": "Svenja Doe"}
]

Settings

Select the Compare value case-insensitive checkbox if you want to ignore case when removing duplicates.

Example of removing duplicates

Input list:

[
      {"email": "john@doe.com", "name": "John Doe"},
      {"email": "bill@acme.com", "name": "Bill Davis"},
      {"email": "john@doe.com", "name": "John"},
]

With the key email, the output of the deduplicate list block is the following list:

[
      {"email": "john@doe.com", "name": "John Doe"},
      {"email": "bill@acme.com", "name": "Bill Davis"}
]

The block searches for items in the list that matches the unique key email, then removes all but the first match of that item.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!