Testing Spark Jobs with test cases

The test framework in Testing Jobs and Services using test cases also applies to Spark Jobs in Continuous Integration workflows.

Tip: Local mode is selected by default in the Spark configuration tab. Use Local mode when designing and testing Spark Jobs. Depending on the number of input and output flows, context variables are automatically created to specify the input and reference files, and the Use context variable radio button in the Basic settings tab of tFixedFlowInput or tBoundedStreamInput is automatically selected.

Note: Before you create a test case for a Job, make sure all Job components are configured.

Follow the same steps described in Testing Jobs and Services using test cases to create a Spark test case. A dedicated test skeleton is available for Spark Jobs.

By default, a Spark Test Skeleton includes:

One or more tFixedFlowInput components (or tBoundedStreamInput for a Spark Streaming Job), depending on the number of input flows in the Job, to load the input files.
The read-only INPUT and OUTPUT icons that are used to indicate the beginning and the end of the part to be tested.
One or more tCollectAndCheck components, depending on the number of output flows in the Job, to compare the temporary output files with the reference files. The test passes if the files are identical, and fails otherwise.

For more information about Continuous Integration and implementation with Talend, see the Software Development Life Cycle best practices guide.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here