Upgrading and uninstalling Data Movement gateway

Support for installing Data Movement gateway on Red Hat 9.x

Customers can now install Data Movement gateway on Red Hat 9.x or on any corresponding and compatible Linux distribution.

Setting up Data Movement gateway

Changes to Data Movement gateway service commands

The commands for stopping, starting and checking the status of the Data Movement gateway service have changed.

Data Movement gateway service commands

Changes to Amazon Redshift data type mapping

• The BOOLEAN data type, which was mapped to VARCHAR(1) in Amazon Redshift, will now be mapped to BOOLEAN.
• The BYTES and BLOB data types, which were mapped to VARCHAR(1) in Amazon Redshift, will now be mapped to VARBINARY (length).

Newly supported databases, database versions, and driver versions

This section lists the newly supported databases, database versions, and driver versions.

• Newly supported data source versions and editions

  The following data source versions are now supported:

  • Azure Database for MySQL - Flexible Server (Supported via the MySQL source connector)
  • MariaDB 10.4 - 10.11 (previously 10.4 and 10.5)

• Newly supported target data platforms and editions

  The following data target versions are now supported:

  • Azure Database for MySQL - Flexible Server (Supported via the MySQL target connector)
  • Databricks: Databricks 13.3 LTS and Serverless SQL Warehouse

• Newly supported SAP HANA driver version

  Customers with a SAP HANA source who want to install Data Movement gateway on Red Hat Linux 9.x, must install SAP HANA ODBC 64-bit Driver version 2.0.19 or later.

End-of-support database versions

This section provides information about end-of-support database versions.

• Support for the following data source versions has been discontinued:

  • Oracle 11.x
  • SAP HANA 1.0

Resolved issues

• Installing Data Movement gateway without providing a server password would not allow the tenant and proxy URLs to be configured in one command.
• Moving data from a Salesforce (SaaS application) data source would print a large number of redundant warnings, thereby impacting data loading performance.
• When retrieving changes for a SaaS application data source, if an error occurred retrieving changes for one of the tables, that table would be suspended and removed from the pipeline. Now, when encountering an error, the task will try to retrieve the changes up to three times before suspending the table.

New data lake targets for Replication projects

We are continuing to expand the supported targets for the Replication project in Qlik Cloud Data Integration, in addition to Amazon S3, you can now choose Azure Data Lake Storage (ADLS) and Google Cloud Storage (GCS) for data lake delivery, as Parquet, JSON or CSV file formats.

Resolved issues:

• IBM DB2 for LUW source connector

  The connection to the IBM DB2 for LUW would fail when the size of files needed for the connection (such as the SSL client certificate and the keystore file) exceeded 4 KB.

• Driver for IBM DB2 for LUW and IBM DB2 for z/OS source connectors

  The DB2 driver installation would fail when using the driver installation utility.

Support for Microsoft Fabric as a data pipeline target

Microsoft Fabric joins the ever-expanding list of data warehouses that can be used as targets in data pipeline projects.

Resolved issues:

Updated the Snowflake driver version in the driver installation utility.

Expanded Support for Private Connections

A private connection can be used to ensure your data traffic remains secure and compliant. It simplifies both network management and security of your VPC (Virtual Private Cloud) without the need to open inbound firewalls ports, proxy devices or routing tables. Qlik Cloud Data Integration already supports private connections to Snowflake, Microsoft SQL Server and Amazon S3 data pipeline targets. With this release, customers can additionally utilize private connections when moving data to Databricks, Microsoft Azure Synapse, Google BigQuery, and Amazon Redshift.

PostgreSQL-based data sources

• All PostgreSQL-based data sources - Multirange data type support: The following multirange data types are now supported with all PostgreSQL-based data sources (on-premises and cloud).

  • INT4MULTIRANGE
  • INT8MULTIRANGE
  • NUMMULTIRANGE
  • TSMULTIRANGE

• AWS Aurora Cloud for PostgreSQL data source - Non-superuser role support: The user specified in the PostgreSQL connector no longer needs to have the superuser role to move data from an AWS Aurora Cloud for PostgreSQL data source. This is especially useful for organizations with corporate security policies that prevent them from granting superuser access to non-privileged users.

Microsoft Azure SQL (MS-CDC) data source: Geo-replication support

Customers moving data from a Microsoft Azure SQL (MS-CDC) data source can now use a Geo Replica database.

Oracle data source: HSM (Hardware Security Module) support

Qlik Cloud Data Integration now supports reading data from Oracle encrypted tablespaces and encrypted columns during CDC.

New regions support

Qlik Cloud Data Integration now supports tenants in DE and UK regions.

Newly supported versions and third-party software

This section lists newly supported database and driver versions.

• Newly supported data source versions. The following data source versions are now supported:

  • Microsoft SQL Server 2022
  • Oracle 21c
  • PostgreSQL 15.x
  • DB2 13.1 (when working with IBM DB2 for z/OS)
  • IBM DB2 for iSeries 7.5

• Newly supported target data platform versions. The following data target versions are now supported:

  • Databricks (Cloud Storage): Databricks 12.2 LTS and Databricks SQL Serverless

• Driver versions. The following ODBC driver versions are now supported:

  • IBM Data Server Client 11.5.8 for IBM DB2 for z/OZ and IBM DB2 for LUW
  • Simba ODBC driver 3.0.0.1001 for Google Cloud BigQuery
  • MySQL ODBC Unicode Driver 64-bit 8.0.32

Microsoft SQL Server driver - Required upgrade

Customers moving data to or from Microsoft SQL Server need to upgrade their SQL Server ODBC driver version to 18.x or later. Note that continuing to use SQL Server ODBC Driver 17.x might result in data errors. Upgrading the driver can be done using either the driver installation utility or manually. For instructions, see Driver setup.

End-of-support database versions

This section provides information about end-of-support database versions.

• End-of-support data source versions. Support for the following data source versions has been discontinued:

  • PostgreSQL 10
  • MySQL 5.6
  • MariaDB 10.3

• End-of-support target data platform versions. Support for the following data target versions has been discontinued:

  • Databricks 9.1 LTS

2022.11.74

August 15, 2023

Replication support

We take a cloud-first approach that enables rapid innovation and adoption. However, that does not mean that we are cloud-only. As part of our continued commitment to improving the long-term value for our customers, we are pleased to announce the release of a new use case for Replication when creating data projects. The new Replication use case is in addition to the existing ability to create data pipelines for all your data integration needs, such as Data Warehouse modernization.

The Replication project supports real-time data replication from supported data sources to a supported target.

Starting with this latest release, the first target that supports replicating data in real time, is Microsoft SQL Server. In addition to supporting Microsoft SQL Server on-premises, the following cloud providers are also supported:

• On-premises
• Amazon RDS
• Google Cloud
• Microsoft Azure (Microsoft Azure Managed Instance and Microsoft Azure Database)

Microsoft Azure Synapse Analytics target - Required driver upgrade

Customers moving data to Microsoft Azure Synapse Analytics need to upgrade their SQL Server ODBC driver version to 18.x or later. Note that continuing to use SQL Server ODBC Driver 17.x might result in data errors. Upgrading the driver can be done using either the driver installation utility or manually. For instructions, see Driver setup.

New "Load data from source" option for SaaS application sources

A new Load data from source option was introduced, which allows customers to read their data directly from the source during Full Load, instead of using the cached data.

For more information on this option including use cases, see Landing settings.

Updated CA certificates

Data Movement gateway 2022.11.74 includes updated CA certificates, which are needed for authenticating the Qlik Cloud tenant. The updated CA certificates also provide support for the Ireland and Frankfurt regions. Therefore, customers with Qlik Cloud tenants in Ireland or Frankfurt who want to use Qlik Cloud Data Integration must upgrade to this version.

Resolved issue

Tasks landing data from an Oracle source would sometimes fail when a wide table contained unused or unsupported columns, or LOB columns that were not replicated.

2022.11.70

June 28, 2023

Improved driver installation utility for SAP clients

In previous versions, customers needed to run the "source arep_login.sh" command when installing SAP clients. From this version, it is no longer necessary to run this command.

Updated CA certificates

This version includes updated CA certificates, which are needed for authenticating the Qlik Cloud tenant.

Resolved issues

• When a replication task on Data Movement gateway failed and recovered automatically, the recovered state would not be communicated to the Landing data asset in Qlik Cloud.

• End-to-end encryption for Data Movement gateway would not be enabled by default, and was controlled by runtime flags.

2022.11.63

May 2, 2023

Driver installation utility

This version introduces a driver installation utility that eliminates the need to manually install and configure drivers. The new utility shortens the installation process while significantly reducing the possibility of user error. When the utility is run, the required driver is downloaded automatically, if possible, and installed. If the driver cannot be downloaded (DB2 drivers require login, for example), all you need to do is download the driver, copy it to a dedicated folder on the Data Movement gateway machine, and run the utility.

For an example of using the driver installation utility to install a PostgreSQL driver, see Prerequisites

Snowflake cloud data platform

The Snowflake connector now supports 4-byte emoji characters.

Support for Azure Database for PostgreSQL - Flexible Server as a data source

The PostgreSQL connector can now move data from Azure Database for PostgreSQL - Flexible Server.

Support for Cloud SQL for PostgreSQL as a data source

The PostgreSQL connector can now move data from Cloud SQL for PostgreSQL.

Newly supported data source versions

This version introduces support for the following new data source versions:

• PostgreSQL 14
• DB2 (for IBM DB2 for z/OS) 12.1
• IBM DB2 for z/OS 2.5

Newly supported cloud data platform versions

Databricks 11.3 LTS

Discontinued support for the following data source versions

The following data source versions are no longer supported:

• DB2 (for IBM DB2 for z/OS) 11
• PostgreSQL 9.6
• Microsoft SQL Server 2012
• MariaDB 10.2

Resolved issues

This version resolves the following issues:

• Oracle data source: When stopping and resuming a task, the task would sometimes fail with a “Failed to set stream position on context” error.
• SAP Application source: Changes would not be captured during the landing task.