Onboarding data

The first step of creating a data pipeline in a Qlik Cloud Data Integration data project is onboarding the data. This involves transferring the data from the on-premises data source and storing datasets in read-optimized format. You can update data with continuous change handling, or use scheduled reloads.

You create onboarding in a single operation, but it is performed in two steps.

• Landing the data

  This involves transferring the data continuously from the on-premises data source to a landing area, using a Landing data task.

  Landing data from data sources

• Storing datasets

  This involves reading the initial load of landing data or incremental loads, and applying the data in read-optimized format using a Storage data task.

  Storing datasets

When you have onboarded the data, you can use the stored datasets in several ways.

• You can use the datasets in an analytics app.

• You can create transformations.

• You can create a data mart.

Onboard data

You start onboarding data in a data project. Datasets will be stored in the cloud data warehouse defined in the data project. For more information about data projects, see Creating a data pipeline.

1. Click Add new and then Onboard data.

   Tip noteYou can also click on an existing source in the data project, and then click Onboard data.

2. Add Name and Description for the onboarding.

   Click Next.

3. Select the source connection.

   You can select an existing source connection or create a new data connection to the source.

   For more information, see Connecting to data sources.

   Click Next.

4. Select data to load.

   For more information, see Selecting data.

   Click Next.

   Settings is displayed, where you can select update method and history settings.

5. Select which method to use to update data in Update method:

   • Change data capture (CDC)

     If your data also contains tables that do not support CDC, or views, two data pipelines will be created. One pipeline with all tables supporting CDC, and another pipeline with all other tables and views using Reload and compare.

   • Reload and compare

6. Select if you want to replicate history of previous data in addition to current data in History.

   Click Next when you are ready.

7. Preview the data tasks that are created to onboard data, and rename them if you prefer.

   Tip noteThe names are used when naming database schemas in the storage data asset. As a schema can only be associated with one task, consider using names that are unique to avoid conflicts with data assets in other data projects using the same data platform.

8. Select if you want to open any of the data tasks that are created, or return to the data project.

   When you are ready, click Finish.

The onboarding data tasks are now created. To start replicating data you need to:

• Prepare and run the landing data task.

  For more information, see Landing data from data sources.

• Prepare and run the storage data task.

  For more information, see Storing datasets

Selecting data

You can select specific tables or views, or use selection rules to include or exclude groups of tables.

Use % as a wildcard to define a selection criteria for schemas and tables.

• %.% defines all tables in all schemas.

• Public.% defines all tables in the schema Public.

Selection criteria gives you a preview based on your selections.

You can now either:

• Create a rule to include or exclude a group of tables based on the selection criteria.

  Click Add rule from selection criteria to create a rule, and select either Include or Exclude.

  You can see the rule under Selection rules.

• Select one or more datasets, and click Add selected datasets.

  You can see the added datasets under Explicitly selected datasets.

Selection rules only apply to the current set of tables and views, not to tables and views that are added in the future.