Hybrid data delivery
You can deliver data ready for consumption in analytics apps to Qlik Cloud or to cloud data warehouses, such as Snowflake, Google Cloud BigQuery, and Azure Synapse Analytics. Data sources can be on-premises or in the cloud. The data can be kept up-to-date without manual intervention using CDC (Change Data Capture) or batch technologies, such as scheduled reloads. Hybrid data delivery is available in Qlik Cloud Data Services Home.
You can land data from a large number of data sources.
Landing on-premises data via Qlik Replicate
You can land data from data sources supported by Qlik Replicate.
For more information, see Supported data sources
Landing data from cloud data sources
You can land data through Qlik Cloud data connections.
For more information, see Supported Qlik Cloud data sources
Before you start, you should review System requirements and limitations to make sure that you have everything needed to get started.
Delivering on-premises data via Qlik Replicate
The diagram shows the workflow of hybrid data delivery when landing data via Qlik Replicate, and how the system components interact.
You need to add an Enterprise Manager resource to connect Qlik Enterprise Manager and Qlik Cloud.
You only need to perform this configuration once.
Requires a tenant admin role in Qlik Cloud and Admin permission on Enterprise Manager level in Qlik Enterprise Manager.
Create a Replicate task in Qlik Enterprise Manager to transfer data from the data source to a cloud landing area.
The data source can be on-premises or in the cloud. The cloud landing area can be an Amazon S3 bucket or a cloud data warehouse, such as Snowflake, Google Cloud BigQuery, or Azure Synapse Analytics.
You can land data in two different modes.
Full load & CDC
The landing starts with a full load. The landed data is then kept up-to-date using CDC.
You can also land data by performing full loads only from the source. This is useful if your source does not support CDC, but can be used with any supported data source.
Create a landing data asset in Qlik Cloud Data Services and associate it with the Replicate task. The landing data asset controls the Replicate task that lands the data. If you do not use CDC, you can schedule the reloads periodically to keep the data up-to-date without manual intervention.
Create a storage data asset to generate tables from the landing data.
If you landed data to an Amazon S3 bucket, you can generate QVD tables.
If you landed data to a cloud data warehouse, you can generate tables and views in the same cloud data warehouse solution.
Consume the datasets in an analytics app.
Delivering data from cloud data sources
The diagram shows the workflow of hybrid data delivery when landing data from cloud data sources, and how the system components interact.
Create a landing data asset that lands data from a cloud data source to a landing area
Create a storage asset that creates ready to consume tables in a cloud data warehouse from the data landed by the landing data asset.
Consume the datasets in an analytics app.
Data Services home
You can access Data Services home by selecting Data Services from the launcher menu (). You can create data assets that land data from your data sources to a landing area, and generate datasets ready for analytics.
Click Manage data assets to find all your data assets. You can filter which data assets to display by space, owner and tags. You define the tags when you create a data asset.
Data spaces are governed areas of your Qlik Cloud tenant that are used to create and store data assets, like a fit-for-purpose collection of tables. Inside the space, you can also create new data connections with connectors, and you can add remote Enterprise Manager resources.
For more information, see Data spaces
All users also have a personal space where the data assets they create are stored by default. Personal spaces are private.
A data asset is a fit-for-purpose collection of tables. It is the main unit of work in Qlik Cloud Data Services. You can create data assets of the following types:
Landing via Qlik Replicate
Copy data from an on-premise data source to a landing area. The landing data asset is connected to a Qlik Replicate task which performs the actual landing. The landing area can be an Amazon S3 data bucket or a cloud target.
You can select to use CDC which keeps data up-to-date without manual intervention, or just perform full loads. You can schedule full reloads periodically to keep the data up-to-date without manual intervention.
Copy data from a cloud data source to a landing area using a Qlik Cloud data connection. The landing area can be a cloud target.
You can perform full loads and schedule reloads periodically to keep the data up-to-date without manual intervention.
Create ready to consume QVD tables from data copied by a Landing via Qlik Replicate data asset to an Amazon S3 landing area.
The QVD tables are kept up-to-date with the landing data without manual intervention.
Storage (Cloud data warehouse)
Create ready to consume tables in a cloud data warehouse from the data copied by the landing data asset. The tables are kept up-to-date with the landing data without manual intervention.
To create a Storage data asset from landing data that is continuously updated, see Creating a storage data asset with change handling in a cloud data warehouse
To create a Storage data asset from landing data that is periodically updated via full loads only, see Creating a storage data asset with full load in a cloud data warehouse
You can also consume another Storage (Cloud data warehouse) data asset and add data from SQL-based tables.
The asset card displays status information for the asset.
Data is updated to
Data from all source transactions up-to this time is available for consumption from this data asset. This information is available for a data asset when:
A full load is completed for all tables.
If CDC is used, the first set of changes is processed.
Tables in error
The number of tables in error.
Status of the data asset.
When an app is in status Running it is continuously waiting for updated data to copy, it is not necessarily copying data at the time.
Data resources connect to external storage and Qlik Enterprise Manager.
You create data connections to let data assets access external storage and cloud data warehouses.
You create a new data connection by clicking on Add new in the top bar, and then clicking Create data connection. You will need to select which type of data source, and then enter address and authentication information.
Data connections can only be updated by the owner of the connection.
Remote Enterprise Manager resources
This is the connection to the remote Qlik Enterprise Manager server that controls the Qlik Replicate server performing the landing task.