Qlik Data Gateway - Data Movement

Qlik Data Gateway - Data Movement facilitates data movement from your enterprise data sources and SaaS applications to supported targets, over a strictly outbound, encrypted and mutually authenticated connection. By eliminating the need to open inbound firewall ports, Qlik Data Gateway - Data Movement provides a secure and trusted means of moving your enterprise data.

A unique key-pair is established for each Data Movement gateway server connected to the Qlik Cloud tenant. This key pair is used for authentication and for end-to-end data encryption on top of the HTTPS transport level encryption. Moreover, as communication with Qlik Cloud and data transfer is initiated via outbound ports only, you do not need to open any inbound ports to your corporate network.

Common use cases

The following section highlights two common use cases: Moving data to a target that can only be accessed via a private connection, and moving data to targets that are directly accessible from Qlik Cloud.

Qlik Data Gateway - Data Movement plays a central role in the data movement process. The diagrams below illustrate how Data Movement gateway interacts with Qlik Cloud and your data sources to securely move data to supported targets.

Supporting private connections in Qlik Cloud Data Integration

A private connection means that your data source or target is not exposed to the Internet, making it inaccessible from Qlik Cloud.

There are several types of private connection, examples of which include:

• Snowflake AWS PrivateLink where public connectivity (via port 443) has been disabled
• Microsoft SQL Server (or any other RDBMS) running in a VPC that has a private IP address only

The Data Movement gateway deployed to access your target performs ELT operations, which include some or all of the following: data registration, storage, transformations, and data marts.

Note that while it is possible to use the same gateway to access both the data source and the data target, using two gateways - one accessing the source and the other accessing the target - will often provide better performance. For example, if the source database is Oracle on-premises and the target database is Snowflake AWS PrivateLink, one gateway (defined in the Oracle source connection) will be used to land data from Oracle to Snowflake, while the other gateway (defined in the Snowflake target connection) will be used to perform the ELT operations. Using two gateways is especially recommended if the source and target databases are not in close proximity to each other. In such a scenario, the landing gateway should be as close as possible to the source, while the target gateway should be as close as possible to the target. If data is being ingested from multiple data sources, it might also make sense to set up a dedicated gateway for each data source, depending on both the type and the location of the data sources. An example of this type of setup is shown in the image below.

Moving data from enterprise data sources to supported targets

The high-level communication sequence is as follows:

1. Data Movement gateway establishes a mutually authenticated and encrypted connection to Qlik Cloud, using HTTPS.

2. Qlik Cloud sends task execution instructions to the Data Movement gateway, which in turn reports the task status to Qlik Cloud.

3. Data Movement gateway retrieves the data from the data source and pushes it to the designated target.

Moving data from SaaS applications to supported targets

The high-level communication sequence is as follows:

1. Data Movement gateway establishes a mutually authenticated and encrypted connection to Qlik Cloud, using HTTPS.

2. Qlik Cloud sends landing task execution instructions to the Data Movement gateway, which reports the task status back to Qlik Cloud.

3. When the landing task is started, Qlik Cloud pulls the data from the SaaS application and stores it (encrypted with the tenant key).

4. Data Movement gateway retrieves the SaaS application data from Qlik Cloud and pushes it to the designated target.

Moving data from SaaS applications and enterprise data sources to supported targets

The high-level communication sequence is a combination of the "Moving data from enterprise data sources" and "Moving data from SaaS applications" use cases described above.

Moving data from SaaS applications and enterprise data sources in a VPC to targets in the VPC

The high-level communication sequence for moving data from the SaaS application is as follows:

1. Data Movement gateway establishes a mutually authenticated and encrypted connection to Qlik Cloud, using HTTPS.

2. Qlik Cloud sends landing task execution instructions to the Data Movement gateway, which reports the task status back to Qlik Cloud.

3. When the landing task is started, Qlik Cloud pulls the data from the SaaS application and stores it (encrypted with the tenant key).

4. Data Movement gateway retrieves the SaaS application data from Qlik Cloud and pushes it to the designated target.

The high-level communication sequence for moving data from the enterprise data source is as follows:

1. Data Movement gateway establishes a mutually authenticated and encrypted connection to Qlik Cloud, using HTTPS.

2. Qlik Cloud sends landing task execution instructions to the Data Movement gateway, which reports the task status back to Qlik Cloud.

3. Data Movement gateway retrieves the data from the data source and pushes it to the designated target.