Converting and filtering records with Talend Data Mapper

This article explains how to convert data types and filter invalid records using a map.

Talend Data Mapper is a powerful mechanism for processing records. A common operation during records processing is to validate data and reject or adjust records with invalid values.

<readings>
	<reading>
		<Timestamp>1/5/2019 2:14 AM</Timestamp>
		<AmbientTempF>44.8</AmbientTempF>
		<ProbeTempF>33</ProbeTempF>
		<Humidity>74</Humidity>
	</reading>
	<reading>
		<Timestamp>1/5/2019 2:04 AM</Timestamp>
		<AmbientTempF>45</AmbientTempF>
		<ProbeTempF>N/C</ProbeTempF>
		<Humidity>73</Humidity>
	</reading>
</readings>

In some of the records, the value in the ProbeTempF element is N/C, which is invalid. This article explains how configure a map to split valid and invalid values into different files and create a third file in which the invalid values are replaced by a valid one to avoid errors.

The map created in this example also converts String data types into more appropriate types, such as Date/Time or Double.

The map is then used in a simple data integration Job.