跳到主要内容 跳到补充内容

Working with incremental data processing

This section shows you how process data incrementally using incremental blocks. Incremental blocks are a type of block that retrieves and updates records every time the automation is executed.

提示注释

You can think of using incremental blocks like polling a data source. Incremental data processing is useful for processing historical data.

Overview of incremental blocks

Some data source connector blocks are incremental blocks. This means they process data incrementally, and they keep an internal pointer. You can tell if a block is an incremental block because the Pointers section is available on the block configuration pane only for incremental blocks.

Pointers tab on pointer block

Pointers tab on the block configuration pane for a github block

信息注释

You can modify these incremental block pointers with the pointer blocks.

Pointers are unique for for each combination of automation, automation type, block input, and the data source. This means that an incremental block with multiple inputs will have pointers for each input value, and two automations with the same incremental block will each have a unique pointer. If the same block is used twice in the same automation with the same inputs and same data source, then these two blocks will share the same pointer.

Each time the incremental block executes, it creates a pointer. Pointers are listed under the Pointers section.

Pointer information

An automation with a list commit block showing incremental listing

Incrementing the pointer

The pointer is incremented for each execution of the automation if a successful response was received from the API, even of the response was empty. If the API is down or did not respond with a 200 OK response code, the pointer will not be incremented. This means that on the next run, the API call sent with the same pointer so that you do not miss out on any data.

If the incremental endpoint received some data but the automation fails while looping over all the data, the pointer will still be increased.

信息注释

If the API is down when the automation is executed, the automation will retry for up to five minutes. This means that downtime of an API under five minutes, it will not cause any issues in your automation.

本页面有帮助吗?

如果您发现此页面或其内容有任何问题 – 打字错误、遗漏步骤或技术错误 – 请告诉我们如何改进!