Skip to main content Skip to complementary content

Record format matching

Data type

When a user applies a transform control to a source entity the record format of that entity output port advances to become the record format of the input port of the target entity. For the most part, the field data types on either side of the transform will match though some functions require that the data type changes as a function of the operator.

Functions where data types change

Cast Functions

TO_DECIMAL, TO_DOUBLE, TO_INTEGER, TO_LONG, TO_STRING

Boolean CASE Operators

Where if a boolean condition is met, these values convert to true/false and stage to the next function.

Boolean Functions

Data Type STRING, DOUBLE, LONG convert to BOOLEAN type. Examples are some date functions and Boolean functions that include the string 'IS': DateIsBetweenExclusive, DateIsBetweenInclusive, DateIsEqual, DateIsValidFormat, IS_DOUBLE, IS_INTEGER, IS_LONG, IS_NULL, IS_NULL_OR_EMPTY

Example of a boolean expression

example of boolean expression 'is salary greater than > 5000'
Transforms where field data type must be the same

Join

Shared common fields must be of the same data type for a Join to validate. It is sometimes possible to use a casting function to transform a field data type (ex. DOUBLE to STRING) for this scenario so that the data types match.

Union

All fields must be the same data type on either side of the control.

Field names

It is a good practice to rename field names to indicate how the value has been transformed. In the case of some transform controllers such as Aggregator where the same field can being queried multiple times, fields must be renamed.

Transforms with field name requirements

Join

Shared common fields can have different names.

Union

Unions require identical field names in the source entities being united.

Transform example: Field names can be renamed within the controller

Field names in source and entity target should differ indicating how the field transformed

Many errors in prepare around field names can be avoided or resolved by renaming target fields that have duplicate names and then disconnecting and reconnecting the target entity if it has not executed; this step refreshes the target entity if it has executed (and data had been loaded to the entity).

Duplicate names in the target entity will error, indicated by red outline around target entity

Mismatched field errors are oftentimes the result of new data being loaded to a target entity with record format in place from previous transform definition. Once reconnected, updated field names in target entity display.

Rename duplicate names in target entity

Field counts will not always be the same on both sides of the transform control. For example, a transform operation may delete fields resulting in the target entity having fewer fields than the source entity. Some errors can be re-mediated by deleting the connector and reconnecting the control to the target entity.

[Transforms with field count requirements]

Join

Source entities can have different field counts.

Union

Field counts must match for both source entities.

field count error example input 1 field count doesn't match input 2 field count

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!