Skip to main content

Custom

For analysts proficient in Pig scripting, the Custom package provides a tool to write and execute scripts in prepare, an ideal environment for Pig data transformations. While Pig scripting is outside the scope of this instruction, the following is an example of a script used in a custom transformation to create a subset of two fields from an entity with ten fields.

output_1_temp = FOREACH input_1 GENERATE (chararray) FIRST_NAME AS FIRST_NAME,(chararray) LAST_NAME AS LAST_NAME;
output_1 = output_1_temp;
Information noteNote there must be a space before and after equals (=) signs in the Pig script to avoid an error.
Example: Custom package

Custom dataflow example with script displayed

Double-click or drag the Custom icon onto the canvas to activate the live control.

Custom controller

When the square icon becomes a live controller on the canvas, double-click to reveal INPUT, SCRIPT, and OUTPUT criteria for the operation.

INPUT, SCRIPT, OUTPUT definition

The controller is composed of three expandable sections: input, script, output

INPUT: Connect the source entity (output port) to the Custom control (input port), the input fields populate within the package.

Custom input fields expanded

SCRIPT: Users generate Pig script in the SCRIPT window. The example below creates a custom subset extracting 3 Fields from the source entity.

Custom dataflow with script panel displayed

OUTPUT: Users format the target entity by entering Field Names and defining each field's Data Type. In the example below, the script has structured the target entity to have three fields. Users name each field and select the appropriate data type.

The script used for this example defines the following three fields to display for each record:

FIRST_NAME

LAST_NAME

SALARY

 

Custom controller output panel is based on output defined by script

Save the dataflow, Validate the package, then Execute the data load into the target entity. For this example, the target entity  'EMPLOYEES_CUSTOM' shows output of the Custom operation.

Sample data of custom dataflow target entity

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!