Connecting to SaaS applications

You can land or replicate data from a large number of SaaS applications in Qlik Cloud Data Integration.

To see all supported applications, go to Supported SaaS applications.

To rapidly develop connectors for the ever-growing number of cloud application sources, Qlik is creating applications connectors for Qlik Cloud Data Integration using standard APIs and generative AI technology .

Connecting to a SaaS application

Connecting to SaaS applications require that you set up Data Gateway - Data Movement. For more information, see Qlik Data Gateway - Data Movement.

1. The first step of connecting to a SaaS application is to add a connection to the data source. You can do this in several ways.

   • Click on Create connection where source connections are listed.

   • Click on Add new in the top bar, and then click Data connection.

   Select Application in Category to filter the list of connectors to SaaS application connectors.

   You can also select from recently used connectors.

2. Select which application to connect to.

3. Complete the connection details to authenticate the connection. The connection details and authentication process are different for different SaaS applications.

   For more information, go to Connecting to SaaS applications and select your data source.

   Select Open connection metadata, and click Create when you are ready.

4. The metadata manager is displayed, where you can define metadata for the connection.

   Click Generate metadata to create metadata based on sampling the source data.

   For more information about metadata loading, see Scanning the data for metadata generation.

   Tip noteYou can also click Import metadata to import a metadata definition that is already generated. For more information, see Exporting and importing metadata.

5. Select the source datasets you want to use, and then click OK.

6. Select options for scanning the data. You can perform a full scan or a quick scan.

   A full scan will be more accurate, but may take a long time to process. If you select full scan, this data will be used in the initial load when landing data.

   You can also set default metadata settings.

   When you are ready, click Generate metadata.

When metadata generation is finished, you can use the connection in a landing or onboarding task.

Change processing

You can land data from SaaS applications using two different update methods. The update method is set in Settings in the landing data task. It is not possible to change update method once the landing data task is prepared.

• Change data capture (CDC)

  The landing starts with a full load. The landed data is then kept up-to-date using incremental loading based on date fields. CDC may not be supported by all data sources.

  Information noteDeletes are not supported. This means that if a row is deleted in the source, it will not be deleted in the landing data. If delete handling is important, use Reload and compare instead.

  You can set the interval between reading changes from the source in the landing data task, under Settings > Runtime.

• Reload and compare

  The landing performs full loads only from the source. This is useful if your source does not support CDC, but can be used with any supported data source.

  You can schedule the reloads periodically.

Limitations

Some tables returned by the SaaS application are not supported by Change data capture (CDC). In this case you will see a warning message in Validation errors. You can either:

• Delete the table from the data task.

• Change the update method of the data task to Reload and compare.

Managing metadata

When you create the connection to the data source, you also need to define metadata for the datasets that are included. Metadata is used when generating schemas to normalize and define appropriate target tables. The metadata allows you to select which columns to be used in the replication to the target. When nested structures are returned from the source, the output tables are normalized to preserve the granularity of the data.

You can either:

• Generate metadata by sampling data from the source tables

• Import metadata that was exported from a connection with the same type of data source.

You can manage the metadata of a connection by clicking Metadata on the connection that you want to manage in Data connections.

Scanning the data for metadata generation

When you are generating metadata, you can select to perform a full scan of the data, or a quick scan in Metadata loading.

• Full data scan will be more accurate, but may take a long time to process.

• Quick data scan is based on a sample of the data. This will be quicker, but not as accurate. You can select how many data samples to use in the scan with Number of data samples. Increase the number to improve accuracy.

  The number of records contained in a data sample depends on which data source is used.

You can also select how to set String columns size.

• Fixed

  Set a fixed string column size between 1 and 10000.

• Based on data values

  Set the string column size to the longest observed value for the field in the data sample, multiplied by the value in Multiplied by.

  If the field is empty for all rows in the sample, the column string size is set to the value in Default size when there is no value.

Selecting datasets

You can add datasets to the metadata.

• Click Select datasets.

  You can now select datasets to add to the metadata. If the dataset is already included in the metadata, it will be reloaded.

  Click OK when you are finished.

Deleting datasets

• To delete a dataset from the metadata, click ... on the dataset, and click Delete

Editing columns in a dataset

You can edit the columns in a dataset to set it as a key, make it nullable, or change data type.

1. Select the dataset.

2. Select the column to edit.

3. Click Edit.

Deleting columns in a dataset

1. Select the dataset.

2. Select the column to delete.

3. Click Delete.

Exporting and importing metadata

You can export metadata from a connection, and import it to another connection.

• Click Export metadata to export metadata.

• Click Import metadata to add metadata from an exported metadata file. The metadata file must have been exported from the same type of data source. If there are tables with the same name, you can either skip them when importing, or overwrite the existing metadata.

Reloading metadata

When there are structural changes in the source data, you must reload the metadata of the connection.

• To reload metadata for all datasets, click Reload metadata.

• To reload a specific dataset, click ... on the dataset, and click Reload metadata

For more information about metadata loading, see Scanning the data for metadata generation.