Skip to main content

Hybrid data delivery

You can deliver data ready for consumption in analytics apps to Qlik Cloud or to cloud data warehouses, such as Snowflake, Google Cloud BigQuery, and Azure Synapse Analytics. Data sources can be on-premises or in the cloud. The data can be kept up-to-date without manual intervention using CDC (Change Data Capture) or batch technologies, such as scheduled reloads. Hybrid data delivery is available in Qlik Cloud Data Services Home.

You can land data from a large number of data sources.

  • Landing on-premises data via Qlik Replicate

    You can land data from data sources supported by Qlik Replicate.

    For more information, see Supported data sources

  • Landing data from cloud data sources

    You can land data through Qlik Cloud data connections.

    For more information, see Supported Qlik Cloud data sources

Before you start, you should review System requirements and limitations to make sure that you have everything needed to get started.

Information noteYou need to add an Enterprise Manager resource to connect Qlik Enterprise Manager and Qlik Cloud. You only need to perform this configuration once. This requires a tenant admin role in Qlik Cloud and Admin permission on Enterprise Manager level in Qlik Enterprise Manager. For more information, see Connect Qlik Cloud and Qlik Enterprise Manager
Qlik Sense Enterprise SaaS Government noteQlik Sense Enterprise SaaS - Government (US) users cannot use hybrid data delivery.

Delivering on-premises data via Qlik Replicate

Hybrid data delivery - landing data via Qlik Replicate

The diagram shows the workflow of hybrid data delivery when landing data via Qlik Replicate, and how the system components interact.

  1. You need to add an Enterprise Manager resource to connect Qlik Enterprise Manager and Qlik Cloud.

    You only need to perform this configuration once.

    Requires a tenant admin role in Qlik Cloud and Admin permission on Enterprise Manager level in Qlik Enterprise Manager.

    Connect Qlik Cloud and Qlik Enterprise Manager

  2. Create a Replicate task in Qlik Enterprise Manager to transfer data from the data source to a cloud landing area.

    The data source can be on-premises or in the cloud. The cloud landing area can be an Amazon S3 bucket or a cloud data warehouse, such as Snowflake, Google Cloud BigQuery, or Azure Synapse Analytics.

    You can land data in two different modes.

    • Full load & CDC

      The landing starts with a full load. The landed data is then kept up-to-date using CDC.

    • Full load

      You can also land data by performing full loads only from the source. This is useful if your source does not support CDC, but can be used with any supported data source.

    Create a replication task

  3. Create a landing data asset in Qlik Cloud Data Services and associate it with the Replicate task. The landing data asset controls the Replicate task that lands the data. If you do not use CDC, you can schedule the reloads periodically to keep the data up-to-date without manual intervention.

    Landing data from on-premises data sources

  4. Create a storage data asset to generate tables from the landing data.

    • If you landed data to an Amazon S3 bucket, you can generate QVD tables.

    • If you landed data to a cloud data warehouse, you can generate tables and views in the same cloud data warehouse solution.

    Generating and storing tables

  5. Consume the datasets in an analytics app.

    Creating an analytics app using tables generated with Hybrid data delivery

Delivering data from cloud data sources

Hybrid data delivery - landing data from cloud data sources

The diagram shows the workflow of hybrid data delivery when landing data from cloud data sources, and how the system components interact.

  1. Create a landing data asset that lands data from a cloud data source to a landing area

    Landing data from cloud data sources

  2. Create a storage asset that creates ready to consume tables in a cloud data warehouse from the data landed by the landing data asset.

    Generating and storing tables

  3. Consume the datasets in an analytics app.

    Creating an analytics app using tables generated with Hybrid data delivery

Data Services home

You can access Data Services home by selecting Data Services from the launcher menu (). You can create data assets that land data from your data sources to a landing area, and generate datasets ready for analytics.

Click Manage data assets to find all your data assets. You can filter which data assets to display by space, owner and tags. You define the tags when you create a data asset.

Data spaces

Data spaces are governed areas of your Qlik Cloud tenant that are used to create and store data assets, like a fit-for-purpose collection of tables. Inside the space, you can also create new data connections with connectors, and you can add remote Enterprise Manager resources.

For more information, see Data spaces

All users also have a personal space where the data assets they create are stored by default. Personal spaces are private.

Information noteIt is not possible to move data assets between spaces.

Data assets

A data asset is a fit-for-purpose collection of tables. It is the main unit of work in Qlik Cloud Data Services. You can create data assets of the following types:

Tip noteYou create a new data asset by clicking on Add new in the top bar, and then clicking Create data asset. You select which type of app to create in the Create data asset dialog.

The asset card displays status information for the asset.

  • Data is updated to

    Data from all source transactions up-to this time is available for consumption from this data asset. This information is available for a data asset when:

    • A full load is completed for all tables.

    • If CDC is used, the first set of changes is processed.

  • Tables in error

    The number of tables in error.

  • Status

    Status of the data asset.

    When an app is in status Running it is continuously waiting for updated data to copy, it is not necessarily copying data at the time.

Information noteData assets are not managed in the Management Console.

Data resources

Data resources connect to external storage and Qlik Enterprise Manager.

  • Data connections

    You create data connections to let data assets access external storage and cloud data warehouses.

    You create a new data connection by clicking on Add new in the top bar, and then clicking Create data connection. You will need to select which type of data source, and then enter address and authentication information.

    Data connections can only be updated by the owner of the connection.

  • Remote Enterprise Manager resources

    This is the connection to the remote Qlik Enterprise Manager server that controls the Qlik Replicate server performing the landing task.

    Connect Qlik Cloud and Qlik Enterprise Manager

Related learning: