Skip to main content Skip to complementary content
Qlik Resources
Loading SearchUnify's search If you need assistance with your product, please contact Qlik Support.
Qlik Customer Portal
Loading SearchUnify's search If you need assistance with your product, please contact Qlik Support.
Qlik Customer Portal

Generating a Job to Identify duplicate values in an analyzed column

When you use the Profiling perspective of Talend Studio to analyze a column in a database table and provide simple statistics on the number of distinct, unique and duplicate values, you can later generate a ready-to-use Job that removes duplicate values from the specified column.

Procedure

  1. From the Profiling perspective of Talend Studio, create a column analysis and set the simple statistics indicators on the specified column. For detailed steps on how to create a column analysis, see Defining the columns to be analyzed and setting indicators.
  2. Execute the analysis to open the results in the Analysis Results view.
  3. Click the name of the analyzed column in which you want to separate unique and duplicate values, and then click Simple Statistics to expand the simple statistics section.
  4. In the Label list, right-click Distinct Count, Unique Count or Duplicate Count and then select Identify duplicates from the contextual menu.

    The Integration perspective opens in Talend Studio showing the generated Job with the corresponding components.

    The database input component and the tUniqueRow components are already configured according to your connection and the column you are analyzing.

    The two output components are file components in this ready-to-use Job, but you can replace them with database output components to write the duplicate and distinct values directly in the desired database.

  5. Configure the two output components:
    1. If you use a SQLite database, an error occurs during the execution. As SQLite locks the database while writing, replace the tDBOutput component by a tFileOutputDelimited component.
    2. Double-click the first output component with the Uniques connection and in the Local Filename field, set the path to the output file that will hold the distinct values.
    3. Double-click the second output component with the Duplicates connection and in the File Name field, set the path to the output file that will hold the duplicate values.
    4. Save the Job and press F6 to execute it.
      Distinct and duplicate values are written to the specified output files.
    5. If required, right-click tFileOutputdelimited in the generated Job and select Data Viewer.

      A preview of the standardized data is open in Talend Studio.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here