Upon ingest of data for an entity or multiple entities, in addition to the default create New partition option, users can Append to or Overwrite the data on an existing partition. This functionality is also available upon executing prepare dataflows. This option is not available for
Creating a new partition
Default. All records are instantiated in appropriate data directories.
Append to an existing partition
Parent load partition will be updated to reflect the incremented (total) record count of the appended loads in the case of entity level Managed (DATA load type) and Registered (
Profile statistics in the parent work order/partition are consolidated across first new load (base) and subsequent appends.
Append loads are not available for data selection in prepare and publish or sample data drop downs, only the parent partition is available.
Deleting logs for an Append load will only remove the logs and not affect any data
If data is loaded and then a user changes the data type on any field (ex. Integer to String) and then loads the data again as Append, the load will fail and the changes will need to be reverted and data reloaded with the original data types for all fields.
Overwrite an existing partition.
Overwrite deletes all previous files in an existing partition and replaces them with a new load of records. This functionality is available for entity level Managed (DATA load type) and
The load type of the work order will be OVERWRITE and it is not available for data selection in prepare and publish or sample data drop downs. The parent work order/partition is selectable (the overwrite records now constitute the entirety of the parent work order/partition record count.)
Deleting the logs for an Overwrite load will only remove the logs and not affect any data.
Profile data is overwritten reflecting profile information of the new records.
Load logs for appended/overwritten partitions, record counts
DATA: New load default. All records are instantiated in appropriate sample | good | bad | ugly log folders.
APPEND: Record counts are appended to the parent workload/partition incrementally. After the jobs have finished successfully, data in temporary sample | good | bad | ugly log folders will be moved to the existing partition, overwriting previous profile data and statistics.
OVERWRITE: Deletes all previous records/files and replaces them with new record load. After the jobs have finished successfully, data in temporary sample | good | bad | ugly log folders will replace those in parent partition, overwriting profile data/statistics.