Qlik Data Gateway - Data Movement

Qlik Data Gateway - Data Movement (hereafter Data Movement gateway) facilitates data movement from your enterprise data sources and SaaS applications to supported targets, over a strictly outbound, encrypted and mutually authenticated connection. By eliminating the need to open inbound firewall ports, Data Movement gateway provides a secure and trusted means of moving your enterprise data.

When is Data Movement gateway required?

Data Movement gateway is required in the following cases:

• Your data source or target is not directly accessible from Qlik Cloud. Examples of data sources or targets that are not directly accessible from Qlik Cloud include data sources or targets located in a VPC, or behind a firewall closed to inbound communication.

  For more information on private connections, see Supporting private connections in Qlik Talend Data Integration below.

• You need near real-time CDC from database sources.
• You need to access target platforms via a proxy server.

• You have a Premium or Enterprise subscription and you need incremental loads from SaaS application sources less than every 30 minutes. When using Data Movement gateway, you can set the interval to as little as 5 minutes with these subscriptions.

  For information about minimum scheduling intervals according to data source type and subscription tier, see Minimum allowed scheduling intervals.

• Your data source is one of the following:
  • IBM DB2 for z/OS
  • IBM DB2 for iSeries
  • IBM DB2 for LUW
  • SAP Extractor
  • SAP (Application)
  • SAP ODP
  • SAP HANA (Database)
  • Oracle
  • Microsoft SQL Server (Microsoft log based)
• You need the added layer of security provided by Data Movement gateway, which prevents data from passing through Qlik Cloud and allows customers to control the data path. This is only relevant to RDBMS, mainframe, and SAP data sources.

In the following example, Data Movement gateway is not required as both the SaaS application source and the data warehouse target are directly accessible from Qlik Cloud:

Protecting your data

A unique key-pair is established for each Data Movement gateway server connected to the Qlik Cloud tenant. This key pair is used for authentication and for end-to-end data encryption on top of the HTTPS transport level encryption. Moreover, as communication with Qlik Cloud and data transfer is initiated via outbound ports only, you do not need to open any inbound ports to your corporate network.

Common use cases

The following section highlights two common use cases: Moving data to a target that can only be accessed via a private connection, and moving data to targets that are directly accessible from Qlik Cloud.

Qlik Data Gateway - Data Movement plays a central role in the data movement process. The diagrams below illustrate how Data Movement gateway interacts with Qlik Cloud and your data sources to securely move data to supported targets.

Supporting private connections in Qlik Talend Data Integration

A private connection means that your data source or target is not exposed to the Internet, making it inaccessible from Qlik Cloud.

There are several types of private connection, examples of which include:

• Snowflake AWS PrivateLink where public connectivity (via port 443) has been disabled
• Microsoft SQL Server (or any other RDBMS) running in a VPC that has a private IP address only

The Data Movement gateway deployed to access your target performs ELT operations, which include some or all of the following: data registration, storage, transformations, and data marts.

Note that while it is possible to use the same gateway to access both the data source and the data target, using two gateways - one accessing the source and the other accessing the target - will often provide better performance. For example, if the source database is Oracle on-premises and the target database is Snowflake AWS PrivateLink, one gateway (defined in the Oracle source connection) will be used to land data from Oracle to Snowflake, while the other gateway (defined in the Snowflake target connection) will be used to perform the ELT operations. Using two gateways is especially recommended if the source and target databases are not in close proximity to each other. In such a scenario, the landing gateway should be as close as possible to the source, while the target gateway should be as close as possible to the target. If data is being ingested from multiple data sources, it might also make sense to set up a dedicated gateway for each data source, depending on both the type and the location of the data sources. An example of this type of setup is shown in the image below.

Moving data from enterprise data sources to supported targets

The high-level communication sequence is as follows:

1. Data Movement gateway establishes a mutually authenticated and encrypted connection to Qlik Cloud, using HTTPS.

2. Qlik Cloud sends task execution instructions to the Data Movement gateway, which in turn reports the task status to Qlik Cloud.

3. Data Movement gateway retrieves the data from the data source and pushes it to the designated target.

Moving data from SaaS applications to supported targets

The high-level communication sequence is as follows:

1. Data Movement gateway establishes a mutually authenticated and encrypted connection to Qlik Cloud, using HTTPS.

2. Qlik Cloud sends landing task execution instructions to the Data Movement gateway, which reports the task status back to Qlik Cloud.

3. When the landing task is started, Qlik Cloud pulls the data from the SaaS application and stores it (encrypted with the tenant key).

4. Data Movement gateway retrieves the SaaS application data from Qlik Cloud and pushes it to the designated target.

Moving data from SaaS applications and enterprise data sources to supported targets

The high-level communication sequence is a combination of the "Moving data from enterprise data sources" and "Moving data from SaaS applications" use cases described above.

Moving data from SaaS applications and enterprise data sources in a VPC to targets in the VPC

The high-level communication sequence for moving data from the SaaS application is as follows:

1. Data Movement gateway establishes a mutually authenticated and encrypted connection to Qlik Cloud, using HTTPS.

2. Qlik Cloud sends landing task execution instructions to the Data Movement gateway, which reports the task status back to Qlik Cloud.

3. When the landing task is started, Qlik Cloud pulls the data from the SaaS application and stores it (encrypted with the tenant key).

4. Data Movement gateway retrieves the SaaS application data from Qlik Cloud and pushes it to the designated target.

The high-level communication sequence for moving data from the enterprise data source is as follows:

1. Data Movement gateway establishes a mutually authenticated and encrypted connection to Qlik Cloud, using HTTPS.

2. Qlik Cloud sends landing task execution instructions to the Data Movement gateway, which reports the task status back to Qlik Cloud.

3. Data Movement gateway retrieves the data from the data source and pushes it to the designated target.