Skip to main content

Record format matching

ON THIS PAGE

Record format matching

Data type

When a user applies a transform control to a source entity the record format of that entity output port advances to become the record format of the input port of the target entity. For the most part the field data types on either side of the transform will match though some functions require that the data type changes as a function of the operator.

Functions where data types change

Cast Functions

TO_DECIMAL, TO_DOUBLE, TO_INTEGER, TO_LONG, TO_STRING

Boolean CASE Operators

Where if a Boolean condition is met, these values convert to true/false and stage to the next function.

Boolean Functions

Data Type STRING, DOUBLE, LONG convert to BOOLEAN type. Examples are some date functions and Boolean functions that start with 'IS': DateIsBetweenExclusive, DateIsBetweenInclusive, DateIsEqual, DateIsValidFormat, IS_DOUBLE, IS_INTEGER, IS_LONG, IS_NULL, IS_NULL_OR_EMPTY

Transforms where field data type must be the same

JOIN

Shared common fields must be of the same data type for a Join to validate. It is sometimes possible to use a casting function to transform a field data type (ex. DOUBLE to STRING) for this scenario so that the data types match.

UNION

All fields must be the same data type on either side of the control.

Field names

It is a good practice to rename field names to indicate how the value has been transformed. In the case of some transform controllers such as Aggregator where the same field can being queried multiple times, fields must be renamed.

JOIN

Shared common fields can have different names.

UNION

Prepare Unions require identical field names.

Many errors in Prepare around field names can be avoided or resolved by renaming target fields that have duplicate names and then disconnecting and reconnecting the target entity if it has not executed, thereby refreshing the target entity if it has executed (and data had been loaded to the entity).

Mismatched field errors are oftentimes the result of new data being loaded to a target entity with record format in place from previous transform definition. Once reconnected, updated field names in target entity display.

Field names

Field counts will not always be the same on both sides of the transform control. For example, a transform operation may delete fields resulting in the target entity having fewer fields than the source entity. Some errors can be re-mediated by deleting the connector and reconnecting the control to the target entity.

JOIN

Source Entities can have different field counts.

UNION

Field counts must match for both Source Entities.