Data movement

Qlik Data Movement helps customers onboard their data rapidly and securely from their on-premise and cloud-based data sources to cloud data warehouses and data lakes. An end-to-end solution for data movement, this service securely accesses data sources, automatically replicates in real-time to cloud targets, and catalogs data sets without manual scripting.

The data movement process in Qlik Cloud Data Integration is managed from the Qlik Cloud hub. It initializes and monitors the rocess of capturing data from Enterprise and Cloud application data sources.

Data from on-premise systems or running in a customer's cloud does not pass through, nor is it stored in Qlik Cloud, unless Qlik Cloud is the chosen destination for the data. SaaS application source data is captured by Qlik Cloud and stored transiently while data flows from source to target via Qlik Data Gateway - Data Movement.

Qlik Data Gateway - Data Movement

A challenge for many customers when moving to SaaS is providing access to their on-premise and private cloud data sources without compromising security. Qlik's solution to this is Qlik Data Gateway - Data Movement. This allows customers to access data sources in their data center and private cloud, without exposing them to the public internet

Qlik Data Gateway - Data Movement is a component controlled from Qlik Cloud, but physically located near to your data. It initiates connections to your source and target systems, orchestrating both full loads and change data capture (CDC). For simplicity we will refer to this simply as the Data Movement Gateway going forward.

Source data is onboarded directly into and persisted to the target cloud platform by the Qlik Data Movement Gateway, removing the need to expose data sources to the internet.

When started, the Data Movement Gateway makes an outbound connection to Qlik Cloud, which then initiates a reverse tunnel back to the gateway for command and control.

From on-premise to cloud data warehousing

Delivering data from on-premise data sources to cloud data warehouses is achieved with the Data Movement Gateway:

• Source — The data source types available in the Data Movement Gateway will govern what sources of data can be delivered from. For more information, see Data sources in the help.

• Target — The target data warehouse will need to be one of the supported ones in Qlik Cloud Data Integration (see Connecting to cloud data platforms in your data projects in the help). Currently, those targets are:

  • Snowflake^®

  • Azure Synapse Analytics^®

  • Databricks^®

  • Google BigQuery^®

  • Amazon Redshift^®

  • Microsoft SQL Server^®

Both staging and storage will happen in the target system when data is delivered via the pipeline. It is possible to use targets in a private cloud; these connections will be proxied via the Data Movement Gateway.

Delivering your data to Qlik Cloud

You can deliver data from on-premise and cloud data sources directly to Qlik Cloud and store as QVD files (Qlik's proprietary file format, designed for fast loading into memory) with the Data Movement Gateway.

• Source — The data source types available in the Data Movement Gateway will govern what sources data can be delivered from. We are regularly adding new sources. See Data sources in the help for details on the latest available sources.

• Target — There are two options for target storage of these files, Qlik-managed storage and Customer-managed storage:

  • The Qlik-managed storage option requires customers to bring their own Amazon S3 bucket for the staging area. This storage is configured, maintained, and financed by the customer. Qlik will however provide storage for the storage area once the files have passed staging and are stored at rest. This is recommended if your goal is to make the data available for Qlik Cloud Analytics.

  • The Customer-managed storage option means the customer brings their own Amazon S3 bucket for both the staging area and the storage area, which means configuring, maintaining, and financing them. This is recommended if you need to make the data available to sources in addition to Qlik Cloud Analytics.

From cloud sources to cloud data warehouses or Qlik Cloud

Delivering data from cloud sources and storing it directly in cloud data warehouses is also possible with Qlik Cloud Data Integration. This still requires the Data Movement Gateway. This allows us to support public and private cloud data warehouses and data lakes as source and target.