Skip to main content Skip to complementary content

Appending to and overwriting partitions in data loads

Upon ingest of data for an entity or multiple entities, in addition to the default create New partition option, users can Append to or Overwrite the data on an existing partition. This functionality is also available upon execution of prepare dataflows. This option is not available for Addressed entities. Note that when a load is appended to or overwritten, the workorder (partition) summary for the original load is deleted.

Creating a new partition

Default. All records are instantiated in appropriate data directories.

Creating a new partition

partition configuration data load modal

Append to an existing partition

Parent load partition will be updated to reflect the incremented (total) record count of the appended loads in the case of entity level Managed (DATAload type) and Registered (STATISTICSload type). This option is not available for entity level Addressed (METADATA load type).

Profile statistics in the parent partition workorder are consolidated across first new load (base) and subsequent appends.

Append loads will not available for data selection in prepare and publish or sample data drop downs, only the parent partition is available.

Warning note

Deleting logs for an Append load will only remove the logs and not affect any data

If data is loaded and then a user changes the data type on any field (ex. integer to string) and then loads the data again as Append, the load will fail and the changes will need to be reverted and data reloaded with the original data types for all fields.

Select a partition to append to an existing partition

Partitions are selectable from data load modal

Overwrite an existing partition.

Overwrite deletes all previous files in an existing partition and replaces them with a new load of records. This functionality is available for entity level Managed (DATA load type) and Registered (STATISTICS load type). This option is not available for entity level Addressed (METADATA load type).

The load type of the work order will be Overwrite and it is not available for data selection in prepare and publish or sample data drop downs. The parent partition workorder can be selected (the overwrite records will then become the entirety of the parent partition record count.)

  • Deleting the logs for an Overwrite load will only remove the logs and not affect any data.
  • Profile data is overwritten reflecting profile information of the new records.

Overwrite existing partitions

Selection of overwrite button deletes and replaces all previous loads

Load logs for appended and overwritten partitions, record counts

DATA: New load default. All records are instantiated in appropriate sample | good | bad | ugly log folders.

APPEND: Record counts are appended to the parent workload/partition incrementally. After the jobs have finished successfully, data in temporary sample | good | bad | ugly log folders will be moved to the existing partition, overwriting previous profile data and statistics.

OVERWRITE: Deletes all previous records and replaces them with new record load. After the jobs have finished successfully, data in temporary sample | good | bad | ugly log folders will replace those in parent partition, overwriting profile data/statistics.

APPEND and OVERWRITE logs:: parent partition reflects aggregate record counts

Overwrite and append load types impact record counts differently

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!