Appending to and overwriting partitions in data loads
Upon ingest of data for an entity or multiple entities, in addition to the default create New partition option, users can Append to or Overwrite the data on an existing partition. This functionality is also available upon execution of prepare dataflows. This option is not available for Addressed entities. Note that when a load is appended to or overwritten, the workorder (partition) summary for the original load is deleted.
Creating a new partition
Default. All records are instantiated in appropriate data directories.
Append to an existing partition
Parent load partition will be updated to reflect the incremented (total) record count of the appended loads in the case of entity level Managed (DATAload type) and Registered (STATISTICSload type). This option is not available for entity level Addressed (METADATA load type).
Profile statistics in the parent partition workorder are consolidated across first new load (base) and subsequent appends.
Append loads will not available for data selection in prepare and publish or sample data drop downs, only the parent partition is available.
Deleting logs for an Append load will only remove the logs and not affect any data
If data is loaded and then a user changes the data type on any field (ex. integer to string) and then loads the data again as Append, the load will fail and the changes will need to be reverted and data reloaded with the original data types for all fields.
Overwrite an existing partition.
Overwrite deletes all previous files in an existing partition and replaces them with a new load of records. This functionality is available for entity level Managed (DATA load type) and Registered (STATISTICS load type). This option is not available for entity level Addressed (METADATA load type).
The load type of the work order will be Overwrite and it is not available for data selection in prepare and publish or sample data drop downs. The parent partition workorder can be selected (the overwrite records will then become the entirety of the parent partition record count.)
- Deleting the logs for an Overwrite load will only remove the logs and not affect any data.
- Profile data is overwritten reflecting profile information of the new records.
Load logs for appended and overwritten partitions, record counts
DATA: New load default. All records are instantiated in appropriate sample | good | bad | ugly log folders.
APPEND: Record counts are appended to the parent workload/partition incrementally. After the jobs have finished successfully, data in temporary sample | good | bad | ugly log folders will be moved to the existing partition, overwriting previous profile data and statistics.
OVERWRITE: Deletes all previous records and replaces them with new record load. After the jobs have finished successfully, data in temporary sample | good | bad | ugly log folders will replace those in parent partition, overwriting profile data/statistics.