Upgrading and uninstalling Data Movement gateway

Resolved issues

• Aggregation of statistics of partial Full Load

• Databricks Lakehouse (Delta) / Databricks (Cloud Storage) – ODBC test connection fails with proxy (Username/Password) on all staging environments

• Databricks Lakehouse (Delta) Volume - Deleting directory fails due to "CLOUD_FILE_NOT_FOUND" error

• For application mode, full-load only task should enable DDL

• SAP Application target creating special character in data while loading to Snowflake data format

• Full Load S3 target preparing duplicate records on same json or different json files with Source SAP extractor

• Missing relevant error message when table is suspended

• OData target SAP ODP job completed but Full load not completing

• Support SQL Server 2012 (skip Always-Encrypted check)

• Restrict LOCAL INFILE to load mode only for MySQL/MemSQL targets (LFI fix)

• SAPDB source – Bad data (missed data, duplicates) can happen for a table when another table receives LOAD TABLE command.

• Support change datatype length in Databricks delta

• BigQuery Streaming - Nulls truncated from STRING data type fields when concatenating chars

• PostgreSQL LOB lookup fails if primary key contains TIMESTAMPTZ

• When DBCC call in SQL Server Deferred Constructor returns a recoverable error, we must once reconnect and retry, because of the possible timeout

• HANA Source , DECIMAL data type in target dropping Scale values causing data mismatch

• SAP ODP Source in 2025.11 version if used INCLUDE option for Table selection pattern. Full load taking longer hours to complete.

• SQL Server Always Encrypted: Table enters suspension in any task to any target due to corrupted CSV file

• SAP HANA rollback to savepoint support

• Task failing with Token Expire issue when working with Azure Active Directory Service Principal authentication

• Failed to run asset. Error message should show missing table in the log.

• Redshift - column length is not supported

• Triggers are being dropped incorrectly

• Oracle Update operations on compressed rows might not include a primary key column value into After Image causing NULL value being used in the target

• Landing task - removal and addition of the same table doesn't create the _ct table if dropped manually from the target DB

• Appending UTC indicator for datatime column value in S3 Parquet files

• Missing records on target detach/reattach after LOB lookup error

• DB2 Z/os is Green but no changes moving

• DataBricks Delta replaces binary zero in character strings by space

• The alternateBackupFolderTransportCommand is not preserving case

• SAP HANA Full Record Mode - triggers could fail if QTREP/TRAN_LOG fills up

• Unable to Use Parallel Load on SAP VBRP Table for Databricks Lakehouse (Delta)

• Databricks (Cloud Storage) replaces binary zero in character strings by space

• SAPDB - a new occurrence of tables have NULL values in primary key for some events

• Crash in LOB lookup when order of columns in SAP and in backend tables is different

• Error when converting csv to parquet file

• Using Oracle Standby, skipped the Online redo log sequence causing data loss

• SAPDB - Task crash on LOB lookup when required metadata was not read yet

• Oracle reading from Online redo log with Redo not yet prepared.

• Oracle 26ai + TDE encryption - Wrong REDO log parsing of encrypted CDC changes

• Add Canada region support

• Support https proxy connection

• When using managed gateway, diagnostic package includes only the last log

• SQL Server (MS-CDC) source – Invalid change-query generated for geometry columns when a column rename transformation is configured

• Stitch source – Change tracking table files not closed when full load started or ended

• Teradata source and target – Driver setup documentation updated

• Security – Updated netty-codec-http to 4.1.136.Final

• Security – Updated lz4-java to 1.11.1

• Security – Updated jackson-core to 2.18.8

• Security – Updated jetty-security to 12.1.10

• Security – Updated Log4j in SAP ODP and Salesforce source endpoints

• Support errorMessage for fullLoadPartial

• Snowflake target task showed many "Object '<>."attrep_status"' does not exist or not authorized." errors.

Resolved issues

• Syntax issue in the attrep_history Control Table on Databricks and Snowflake

  After upgrading to Data Movement gateway 2025.11.40, INSERT operations into the attrep_history Control Table resulted in a syntax error.

• Oracle source

  Table selection get_table_list failed with Oracle error code '1406' when retrieving table metadata in Oracle Autonomous Database environments with compression enabled.

• Replicate asset failure

  Improved error messaging to display the missing table name in the log when a landing task fails to run due to a missing table.

• Landing task - Change tracking table

  Removing and re-adding a dataset in a landing task no longer created the change tracking table if it was manually dropped from the target database.

• SQL Server source - Always-Encrypted compatibility

  Added support for SQL Server 2012 (using an internal property) by skipping Always-Encrypted validation check, which are not supported in SQL Server versions prior to 2016.

• SQL Server Always Encrypted

  Fixed an issue where tables would enter suspension in any task to any target due to corrupted CSV file handling when using Always Encrypted columns.

• Security

  Updated libcurl library to address CVE-2025-14819 vulnerability.

• Registered data with Database Name containing hyphen

  Fixed an issue when task preparation of registered data failed when database names contained hyphens.

• Snowflake target - COPY INTO timeout

  Fixed timeout errors that occurred during the "COPY INTO" step when loading large datasets to Snowflake.

Databricks driver rebranding

To align with Databricks, the driver for working with the Databricks connector has been rebranded.

Support for new Snowflake and Google BigQuery driver versions

This version introduces support for the following driver versions:

• Snowflake: Snowflake ODBC driver 3.15.0 (64-bit)

• Google BigQuery: Simba ODBC driver version 3.1.6.1026

Support for the Databricks VARIANT data type

From this version, the following data types will be mapped to the Databricks native VARIANT data type:

• STRING

• WSTRING

• NCLOB

• CLOB

Resolved issues

• IBM DB2 for iSeries source

  After resuming a task, capturing changes from a table that contained binary decimal columns would cause the task to crash.

• PostgreSQL source - CDC issue

  After resuming a task that did not capture any changes for the minute before it was stopped, no changes would be captured.

• SSH connector - Test Connection issue

  Clicking Test Connection in the SSH connector would not return errors related to SSH connectivity when there were issues with SSH.

• Snowflake target - Snowpipe streaming issue

  When using Snowpipe Streaming as the loading method, loading very large tables would consume an excessive amount of memory, which would eventually result in the task crashing and entering a loop of loading and crashing.

• Microsoft SQL Server (log-based) source - Unclear error

  When specifying an Alternate backup folder which did not contain the required logs, the task would stop with an unclear message instead of issuing a warning an continuing.

• Security

  The org.apache.logging.log4j:log4j-core component was upgraded to 2.25.3 due to known vulnerabilities with the installed versions.

• SAP ODP source - filtering issue

  The SAP ODP filter would not be passed to SAP when defined on an extractor running in CDC-only mode.

• SAP ODATA source - filtering issue

  Defining more than one filtering range on a single column would cause full load to fail.

Extended Parallel Load support

The Parallel Load feature now supports Databricks target.

Replicating dataset segments in parallel

DDL History control table: Newly supported targets

The attrep_ddl_history (DDL History) control table contains a history of DDL changes that occurred on the source during replication to the target.

The following target endpoints are now supported:

• Google BigQuery
• Google Cloud Alloy DB for PostgreSQL
• Google Cloud SQL for PostgreSQL
• PostgreSQL (on-premises)

DDL history

Enhancement to the “[header]__change_mask” header column

When using Change Tables with Parquet format, the header column [header]__change_mask now pads zeros to the left (instead of the right), to align with the little endian rule of storing bytes in reverse order.

Affected target endpoints:

• Amazon S3

• Google Cloud Storage

Change Tables

Newly supported data source and target versions

• Kafka target: 4.0.x

• IBM z/OS source: 3.2

Newly supported driver versions

• Microsoft ODBC Driver 18.5
• MySQL 8.4

End-of-life for Microsoft Azure Database for MySQL and Microsoft Azure Database for PostgreSQL

Support for Microsoft Azure Database for MySQL and Microsoft Azure Database for PostgreSQL, which have been officially retired by Microsoft, has been discontinued.

End-of-life data source and target versions

• Databricks target: 10.4 LTS and 11.3 LTS

• MariaDB source and target: 10.5

• PostgreSQL source and target: 13.x

• MySQL source and target: 8.1

End-of-life driver versions

• Microsoft ODBC Driver 18.3

Resolved issues

• SAP (Application) source

  Tasks configured to use a SAP (Application) source with a backend process that generated self-committed events, would sometimes crash after reloading a table.

• Microsoft Azure SQL Database target

  In homogenous replication, when working in “Transactional apply” mode or in “Batch Optimized Apply” mode that switched to one-by-one mode, a “0 rows affected” error would occur when a DATETIME column was part of the primary key on the target.

• Oracle source

  Oracle Task crashes in Full Load when a table contains a long raw column, and the max lob size is greater or equal than 64K.

• SAP Extractor source - Unload loop due to CDC cancellation

  During a table reload, long-running SAP CDC jobs are sometimes canceled. This would result in a recoverable error state where the unload never completed.

• Security

  The following components were upgraded due to known vulnerabilities:

  • JRE upgraded from 17.0.17 to 17.0.18

  • org.eclispe.jetty:jetty-io upgraded from 11.0.25 to 11.0.26

  • org.apache.logging.log4j:log4j-core upgraded from 2.23.1 to 2.25.3

• SAP ODP

  The following error would occur during CDC when a filter was defined for a table:

  Invalid value for parameter "I_DIMNAME"

• SAP extractor

  Delta would stop working if the task was stopped and resumed while one of the table was in a Loading state.

• SaaS application source (Workday Reports)

  • During Full Load, when connecting to the S3 staging folder, the task would sometimes fail with the following error:

    Unable to execute HTTP request: Connect to s3.amazonaws.com:443

    The issue was resolved by implementing a multi-retry mechanism for connecting to the S3 folder when the first connection is unsuccessful.

  • CREATE SCHEMA privilege would be required even when the schema already existed.

• Microsoft SQL Server target

  Data loss would sometimes occur in LOB columns and incorrect data replication would occur during CDC when LOB size was set to unlimited in the task settings.

• PostgreSQL target

  Failure to load the CSV file into the Net Changes table would return a recoverable error (resulting in an endless loop) instead of switching to one-by-one apply.

• MySQL-to-MySQL replication

  When the schema of the source and target were in the same MySQL instance and the replication included a control table, the task would sometimes crash with a "Process crashed with signal 11" error.

• PostgreSQL source

  During CDC, the task would sometimes read the same WAL entries over and over again resulting in no data being captured.

• Microsoft SQL Server source

  • An error would occur when decompressing a transaction log that contained encrypted blocks. The issue was resolved by leaving the log compressed in such cases.

  • Redundant errors would be shown when using an alternate backup folder, due to an unnecessary validation check for existing files.

  • The task would sometimes crash after DBCC Lookup invocation.

Support for Kafka target connector

The new version of Data Movement gateway introduces support for the Kafka target connector. Effortlessly replicate your data from any supported source directly into on-premises Kafka or Amazon MSK, ensuring reliable, fault-tolerant transfers that keep your applications agile and informed. Whether you're maintaining data consistency across systems or fueling analytics, this connector delivers low-latency replication with minimal setup.

For more information, see Kafka target

Resolved issues

• MySQL source to MySQL target replication task failure

  When replicating a dataset from a MySQL schema to a different MySQL schema running on the same MySQL server instance, and including a Control table in the replication, CDC would fail with the following error:

  Process crashed with signal 11

• SAP Extractor connector: Table suspended when resuming a task

  Resuming a task while a table was still in full load caused the extraction job to fail and the table to be suspended.

Resolved issues

• PostgreSQL target - Incorrect error behavior

  Failure to load the CSV file into the Qlik Net Changes table would return a recoverable error (resulting in an endless loop) instead of switching to one-by-one apply.

• SaaS application Lite connectors

  During Full Load, tasks would sometimes fail with the following error due to the connection being actively refused by Amazon:

  Unable to execute HTTP request: Connect to s3.amazonaws.com:443

  The issue was resolved by implementing a retry mechanism with a maximum of three retries before the error occurs.

• Microsoft SQL Server source - Log decompression failure

  An error would occur when decompressing a transaction log that contained encrypted blocks. The issue was resolved by leaving the log compressed in such cases.

• Netsuite Suite Analytics source - CDC resume error

  When resuming a CDC task after making a change to the source, the task would sometimes fail with the following error:

  Cannot change column title from type VARCHAR(8000) to VARCHAR(3) because reducing the byte-length of a varchar is not supported.

• Microsoft SQL Server source - False errors

  Redundant errors would be shown when using an alternate backup folder, due to an unnecessary validation check for existing files.

Databricks primary key support

From this version, the data task will create a primary key on target Databricks Delta tables, using the RELY keyword. Although Databricks does not enforce primary key constraints, primary keys with the RELY keyword are assumed to have no duplicates and can therefore be used by Databricks for query optimization.

Requirements:

• You must select a Catalog in the connector settings

• Databricks 14.2 or later

Data type enhancements

• IBM DB2 for iSeries source: Support for the following data types has been added: BINARY-DECIMAL and ZONED-DECIMAL

• Databricks target: In previous versions, BYTES was mapped to STRING. From this version, it will be mapped to VARCHAR (Length in Bytes)

• Google Cloud BigQuery target: The JSON subtype is now supported by the Google Cloud BigQuery target

PostgreSQL-based source: Partitioned tables now supported by default

Replication (Full Load and CDC) of partitions and sub-partitions (and sub-sub partitions) from PostgreSQLbased data sources is now supported by default.

Google Cloud BigQuery target: Support for data truncation error handling

The Google Cloud BigQuery target now supports data truncation error handling. If a data truncation error occurs, you can now choose whether to log the record to the exceptions control table (default), ignore the record, suspend the table, or stop the task.

Newly supported versions

The following source and target endpoint versions are now supported:

• MariaDB (on-premises) and Amazon RDS for MariaDB: 11.4

• MySQL (on-premises), MySQL Percona, Google Cloud SQL for MySQL, Amazon RDS for MySQL, and Microsoft Azure Database for MySQL - Flexible Server: 8.4

• PostgreSQL (on-premises), Google Cloud SQL for PostgreSQL, Amazon RDS for PostgreSQL, and Microsoft Azure Database for PostgreSQL - Flexible Server: 17.x

Newly supported driver versions

IBM Data Server Client 11.5.9 is now supported for the following DB2 sources:

• IBM DB2 for LUW

• IBM DB2 for z/OS

End-of-life source and driver versions

• End-of-life sources and targets: Support for Microsoft Azure Database for MySQL and Microsoft Azure Database for PostgreSQL, which have been officially retired by Microsoft, has been discontinued.

• End-of-life source versions: Support for IBM DB2 for iSeries 7.2 which is EOL, has been discontinued.

• End-of-life driver versions: Support for IBM Data Server Client 11.5.6 has been discontinued.

Resolved issues

• HSTS support

  HSTS was not enabled on the Qlik Data Gateway - Data Movement machine.

• Data Movement gateway Installation failure

  Installation of Data Movement gateway on Redhat Version 9.7 would fail with an "OPENSSL_3.4.0 not found" error.

• PostgreSQL 15.7 Source - No data captured during CDC

  During CDC, the task would keep reading the same WAL entries and processing the same stream position.

• SAP ODP - CDC failure

  After a CDC tasked stopped during changes (due to a connectivity issue for example), resuming the task would sometimes result in row length errors.

• Missing source connection logs

  Source connection logs would be unavailable after running a task that used a Preview or Lite connector.

• Monitoring - Missing information

  The Monitor tab would not show an error message when a table was suspended during the CDC stage of a Full Load and CDC task.

Support for creating knowledge marts on Databricks

Previously knowledge marts were supported on Snowflake only. Now, you can create and run knowledge marts and file-based knowledge marts on Databricks as well.

Resolved issues:

Tasks would fail intermittently during preparation with the following error:

The request was canceled due to the configured HttpClient.Timeout of 300 seconds elapsing.

Resolved issues:

When the metadataschema Internal property was defined with special characters, the task would fail during CDC.

Resolved issues:

• Production alerts in the EU-Central-1 region

  Customers in the EU-Central-1 region would experience frequent production alerts when using Data Movement gateway.

• Jira data source: Stuck table

  When replicating data from the Jira, the USERS table would get stuck in Queued status on the first run, while the other tables completed loading successfully.

• Monitoring issue

  CDC monitoring was missing from the task monitor.

• Oracle-to-Oracle replication failure

  Oracle-to-Oracle replication tasks would sometimes fail due to an incorrect calculation of the string length.

• Security

  The Java server component (sqlite-jdbc) was updated to 3.47.0 due to a known vulnerability with earlier versions.

• Error when starting a task

  A "No license for database" error would sometimes occur when a task was started in resume mode for the first time, without previously being started.

• Salesforce connector issue with new datasets

  Adding new datasets to a landing task with a Salesforce data source would cause the task to fail with a "'Table doesn't exist'" error.

Resolved issues:

• Storage tasks connection error

  Storage tasks would fail with a connection error that required the gateway to be restarted.

• DB2 driver installation failure

  The DB2 driver installation would fail when access to the /tmp folder was restricted.

• Excessive connection time to Qlik Cloud

  After upgrading to Data Movement gateway version 2024.11.65, it would take an excessive amount of time to connect to Qlik Cloud. The issue was resolved by adding a cancellation token with a two-minute timeout.

• Service crashed after upgrade

  After upgrading to Data Movement gateway 2024.11.67, the external-data-provider service (used by Lite connectors) crashed due to access restrictions on the /tmp folder.

• Security fixes

  The following components were replaced with later versions due to known vulnerabilities:

  • SQLite 3.49.1

  • Java Runtime Engine 17.0.15

  • org.bouncycastle:bcpkix-fips 2.0.7 (bcpkix-fips-2.0.7.jar)

  • org.bouncycastle:bc-fips 2.0.0 (bc-fips-2.0.0.jar)

• Knowledge marts - File Share (SMB)

  Connecting to SMB would fail with the following error:

  SmbConnection' object has no attribute 'docs_path'

• Prepare failure

  When preparing a task for the second time and a DDL occurred after the first prepare, the prepare operation would sometimes fail with the following error (excerpt):

  Could not find match between landing metadata with gateway task metadata for the following tables: <name>

• Preview data issue

  The preview data option would not work when using a Microsoft SQL Server target.

Enhancements:

• Added support for Qlik Cloud Government DoD

  This version adds support for using Data Movement gateway with Qlik Cloud Government DoD.

Enhancements:

• Controlling when the table error handling policy is triggered

  A new Number of retries before returning a table error setting has been added to the table error handling settings for replication and landing tasks. Previously, when a table error occurred, the table would be suspended or the task would be stopped (according to the selected error handling action). This sometimes resulted in tables being suspended during the SaaS application maintenance window. With this new option, it is now possible to determine how many times the table should be retried before the table error handling policy is triggered. This option is especially useful if your SaaS application undergoes periodic maintenance, as you can override the default number of retries (3) so that the maintenance can be completed without triggering the table error handling policy.

  Table error handling

  Information noteThis option will only be shown for tasks configured with a Lite or Standard SaaS application connector.

• Preview data now includes column data types

  When previewing an SQL transformation, the results will now include the column data types.

  Previewing data

Enhancements:

• Rollover and purging of replication log files

  A rollover and purge policy has been implemented for the replication log files on the gateway machine. The rollover policy will prevent the log files from becoming too large (which impacts performance and analysis), while the purge policy will prevent them from accumulating and consuming excessive disk space.

Resolved issues:

• Incorrect table status after reload

  When reloading a suspended table, the table status shown in the monitor would sometimes be incorrect.

• Prepare operation timing out

  When preparing a landing task after adding a dataset, a timeout would sometimes occur at the cataloging assets stage. This would cause the Prepare operation to fail.

This patch resolves the following issues:

• Prepare failure when adding a new table

  When adding a new table to a replication task and clicking Prepare, the following error would sometimes occur:

  Prepare data task failed: Object reference not set to an instance of an object.

• Resolved vulnerabilities

  The following Data Movement gateway setup files were updated to versions without known vulnerabilities:
  

  • java

  • jetty-io-<version>.jar

  • jetty-http-<version>.jar

  • spring-expression-<version>.jar

  • spring-core-<version>.jar

  • spring-context-<version>.jar

This patch resolves the following issues:

• IBM DB2 for LUW connectivity failure

  After upgrading to Data Movement gateway 2024.11.54, connecting to the database would fail.

• Prepare failure

  Preparing the data task would fail when the task was configured with a SaaS application source and a Google BigQuery target.

• Tasks with Java-based sources fail after upgrade

  Tasks with Java-based data sources would fail after upgrading to Data Movement gateway 2024.11.54.

• Snowflake target: Tasks would sometimes switch to "one-by-one" mode during CDC

  During CDC, when a STRING column in a source table exceeded 65536 bytes, the data would be truncated in the target STRING column. The task would then switch to "one-by-one" apply mode.

• Security

  The java_file_factory.jar and arep-google-cloud.jar files were upgraded to versions without known vulnerabilities.

This patch resolves the following issues:

• Full load ignoring task settings

  During full load, the task would drop and recreate the target tables, even though the task was configured with If target table already exists: Ignore

• Incorrect record count

  After stopping and preparing an existing replication task with the replication mode set to full load and apply changes, the CDC processed records count would increase, resulting in an incorrect count.

• JIRA source connector

  When a new dataset was added to an existing task, the task would fail with the following error:

  Could not find a match between landing metadata and gateway task metadata for the following tables: <name>

• Test connection failure

  Test connection to SAP and SQL-based sources would fail.

• Upgrade

  After upgrading the gateway, tasks would sometimes not resume as expected.

Support for Snowpipe Streaming

Previously, Bulk loading was the only method available for loading data into Snowflake. Now, customers can use either Bulk loading or Snowpipe Streaming to onboard or replicate their data to Snowflake.

The benefits of Snowpipe Streaming over Bulk loading include: 

• Less costly: As Snowpipe Streaming does not use the Snowflake warehouse, operating costs should be significantly cheaper, although this will depend on your specific use case.
• Reduced latency: As the data is streamed directly to the target tables (as opposed to via staging), replication from the data source to the target should be faster.

Resolved issues

This patch resolves the following issues:

• Microsoft Fabric: Connectivity issue

  When browsing for databases using the Microsoft Fabric connector, the following error would occur (excerpt):

  Create connection failed'' with error 'java.lang.NoSuchMethodError

  The issue was caused by an incompatibility with the 12.9.0-preview.jar JDBC driver, which was installed with the driver setup utility. The utility has now been updated to install a compatible driver.

• Java server restart failure

  After upgrading Data Movement gateway or restarting the Data Movement gateway service, the Java server (required fro Java-based connectors such as SAP ODP) would fail to start.

• Security hardening

  • End-to-end encryption would not be mutually enforced for logging command payloads that returned customer-hosted logs from the Qlik Cloud Administration activity center.

  • The SQLite version used by Data Movement gateway was upgraded due to a vulnerability with version 3.42.0 (CVE-2023-7104).

  • The com.h2database file used by Data Movement gateway was upgraded due to a vulnerability with version h2-2.1.212 (CVE-2022-45868).

This patch resolves the following issue when moving data from an IBM DB2 for iSeries data source:

When using iSeries Access ODBC Driver 07.01.026 or later, the task would fail during CDC with an "Error parsing" error.

Standard

OAuth support for Databricks target

Customers can now connect to Databricks target using OAuth authentication, which is the method that Databricks recommends for account authentication. Personal Access Token authentication will remain available for backward compatibility for now, but it is considered less secure than OAuth.

Resolved issues

• Parallel load issue

  When a dataset was defined with parallel loading and full load was subsequently disabled, the task would still perform a full load of that dataset, even when it was resumed from timestamp.

2024.11.24

Patch

This patch resolves the following issues:

• SAP ODP source: Metadata generation failure

  Metadata generation would fail to complete for tables with a dot in their name (e.g. Deprec.Area).

• Task recovery

  Tasks would only recover once if they failed during full load due to an environmental issue. Now they will recover the number of times configured in Settings > Error handling > Environmental errors > Maximum retry count.

Enhancements

PostgreSQL failover certification

Working with a secondary database after failover has been certified with the following data sources (which can all be accessed using the PostgreSQL source connector):

• Google Cloud SQL for PostgreSQL
• Amazon RDS for PostgreSQL
• PostgreSQL (on-premises)

See also: Setting up failover

MySQL performance improvement

In previous versions, when using a MySQL-based source endpoint in a task that was configured with limited LOB size, Data Movement gateway would use source lookup to read the LOB columns. Now, Data Movement gateway will read the LOB columns directly from binlog, thereby improving performance.

Data type mapping changes

Mappings to LOB columns have been changed for Amazon Redshift and Snowflake target.

Amazon Redshift target

• BLOB is now mapped to VARBYTE(16777216)
• NCLOB is now mapped to NVARCHAR(65535)
• CLOB is now mapped to NVARCHAR(65535)

Snowflake target

• BLOB is now mapped to BINARY (8388608)
• NCLOB is now mapped to NVARCHAR(16777216)
• CLOB is now mapped to VARCHAR(16777216)

Newly supported data types

• IBM DB2 for LUW source

  Now supports the BOOLEAN data type (from DB2 for LUW 11.5).

• Google BigQuery target

  Now supports the BIGNUMERIC data type.

Expanded target support for the DDL History control table

In previous Data Movement gateway versions, the DDL History control table was supported with Databricks and Microsoft Fabric targets only. This version introduces support for the following additional targets connectors:

• Amazon Redshift
• Amazon S3
• Google Cloud Storage
• Snowflake

See also: DDL history

Support for using a non-superuser with Google Cloud SQL for PostgreSQL

From this version, it is now possible to specify a non-superuser account when replicating from Google Cloud SQL for PostgreSQL

See also: Using an account without the superuser role

Newly certified platforms and driver versions

• Databricks target

  15.4 LTS

• Oracle source and target

  23ai

  Information note

  • Oracle 23ai source and target are certified with Oracle Standard Edition only.
  • Oracle 23ai source is supported with TDE encryption only.

• Databricks driver update

  When moving data to Databricks, Simba Spark ODBC Driver 2.8.2 or later is now required. Customers should either run the driver installation utility or install the driver manually.

  See also: Driver setup

End of support

Support for the following database versions has been discontinued:

• PostgreSQL 12.x
• IBM DB2 for z/OS: z/OS 2.4
• Databricks 9.1

Resolved issues

• SAP HANA (Database) connector: Failure to create __CT table

  When the replication mode was set to Store Changes, and the table contained an .INCLUDE column, the task would fail to create the __CT table on the target.

• Lake landing task: Misleading error

  The task would fail and a "No license for database" error would be shown when the Azure Data Lake Storage secret key had expired.

• Amazon S3 target connector: Old files would not be deleted

  Older files would not be deleted when running a new task. This could result in the task failing due to duplicate records or because of non-matching records.

   

• SAP HANA (Database) source: Connection failure due to a user-defined table

  A "feature not supported: cannot select user-defined type" error would occur when selecting a user-defined table for replication. The issue was fixed by excluding user-defined tables (which are not supported) from the table selector.

• Microsoft Fabric target connector: Recovery error

  A "Conversion failed" error would sometimes occur after recovering from a connection error that occurred during CDC.

• Databricks target connector: CDC failure when working without LOB support

  When disabling LOB support, CDC would fail with a "not enough data columns" when trying to insert the change into the __ct table. This error would occur when working in "Store Changes" mode, which is the only mode available for landing tasks.

• MySQL source to Snowflake target: CDC failure

  During CDC, when replicating a "NOT NULL" column with mediumtext data type, the task would sometimes fail with the following error:

  A NOT NULL constraint violation occurred on LOB column from MySQL source

• Microsoft Fabric target connector: Truncation error

  When inserting a VARCHAR column, the following truncation error would occur:

  String or binary data would be truncated while reading column of type 'VARCHAR'

  The issue was resolved by multiplying the VARCHAR column in Microsoft Fabric.

New features

Support for specifying additional ODBC and JDBC connection properties has been added to the Snowflake target connector.

For more information, see Additional connection properties.

Resolved issues

• Metadata not being updated

  When a primary key was altered on the source, the target metadata would not be updated after a reload.

• Amazon S3 target

  When creating and running a new task, files left over from previous tasks would not be deleted from the tables location.

• SAP HANA source

  Trying to connect to a SAP HANA source using log-based CDC would result in the following error:

  Cannot select user-defined type

2024.5.41

Patch

This patch resolves the following issues:

• Amazon S3, Google Cloud Storage, and Qlik Cloud targets: Quoting empty string values

  Empty strings will now be quoted in the CSV file to differentiate them from NULL values.

• Databricks target: Data mismatch

  A mismatch between the source and target data would occur in certain scenarios.

• Databricks target: CDC failure caused by disabling LOB support

  Disabling LOB support in the landing task would cause CDC to fail with the following error (excerpt):

  Cannot write to '<name>__ct', not enough data columns

• Microsoft Fabric target: Connection error resulting in CDC failure

  Applying changes to the target after recovering from a connection error would sometimes fail with a "Conversion failed" error.

2024.5.35

Patch

This patch resolves the following issues:

• Microsoft Fabric target: Data type conversion issue

  During a storage task, data type NVARCHAR(3500) would be incorrectly converted to VARCHAR(10500) instead of VARCHAR(8000) (the maximum allowed), resulting in an error.

• IBM DB2 for LUW source: Support for GRAPHIC data type with code page 941 (Japanese)

  When the source table contained a GRAPHIC data type with code page 941 (Japanese), the following error would occur (excerpt):

  Unexpected CCSID '941' in column 'PNTNM'

2024.5.32

Patch

After updating the username and password in the Workday Reports source connector and then reloading the task, an "Invalid username or password" error would occur even though the credentials were correct. This issue is now resolved.

Patch

This patch resolves the following issues:

Security

Oracle Java Standard Edition 17.0.8, that ships with Data Movement gateway has several known vulnerabilities. This patch replaces 17.0.8 with 17.0.12, thereby resolving the issues.

Salesforce source

During CDC, for each change, multiple INSERT operations would be added to the associated _ct table (instead of only one). This would significantly increase the number of records in the _ct table.

Standard

Patch

This patch resolves the following issues:

Snowflake JDBC Driver vulnerability

On August 14th 2024, Snowflake issued a security advisory about Snowflake JDBC drivers >= 3.2.6 & <= 3.19.1, some of which were used with previous versions of Data Movement gateway. The vulnerability was patched in Snowflake JDBC Driver 3.20, which is the version that is now installed with the driver installation utility. Existing customers should upgrade their driver at the earliest opportunity, by following the Updating the driver procedure described in Driver setup.

Oracle source

• When using redo event 11.22, missing INSERTs would occur when processing multiple INSERTs on a compressed page that was not compressed prior to the INSERTs.
• In rare scenarios, incorrect parsing of DELETE evens in the redo log record would generate a "The Redo Log DELETE event contains an unknown structure" warning followed by various issues.

SAP Extractor source

The instructions in the YAML file were updated to reflect the correct version of the SAP Java Connector.

Snowflake target

When using Data Movement gateway to connect to Snowflake target via a proxy, the connection would fail with the following error:

500 Failed to connect to Data Movement Gateway

Patch

Patch

Resolved issues

• Oracle Source

  • Missing INSERTs would sometimes occur in multiple INSERT operations when using redo event 11.22.

  • After upgrading Oracle 19g to the July 2024 patch, UPDATE operations would sometimes not be captured, and the following warning would be shown:

    A compressed row cannot be parsed

• Snowflake Target

  When the task settings were configured to create the control table schema, the task would fail with the following error:

  Failed to delete directory

• Scheduled CDC

  When a task was scheduled to run periodically, it would sometimes fail with following error:

  The task stopped abnormally

• Stuck transformation and storage tasks

  Transformation and storage tasks would sometimes remain in a Queued state for an excessively long time.

• Metadata manager

  Tasks would fail when using the use_manipulation_pk_for_apply feature flag with Store Changes replication.

• Prolonged timeout resulting in issues with SaaS application sources

  Extended the S3 timeout to 6 hours to prevent issues resulting from prolonged timeouts such as losing the token to download the files.

Standard

Enhancements

• Added Key Pair authentication to the Snowflake target connector

  Key/pair provides a more robust authentication method than user/password for connecting to Snowflake with your service accounts. This approach is recommended for workloads such as data loading (replication or landing tasks) and transformations.

• Support for retrieving metadata from multiple tables in parallel

  In previous versions, refreshing the metadata on an existing dataset or a newly added dataset would sometimes fail with an error. This enhancement ensures that metadata can be retrieved from multiple tables in parallel without any issue.

Resolved issues

• DECIMAL data type wrongly mapped to STRING in Google BigQuery

  When a source table contained a column with a DECIMAL data type - for example, DECIMAL (38, 20) - preparing the storage task on Google BigQuery would fail with the following error (excerpt):

  Column <n> in <table name> has incompatible types: STRING, BIGNUMERIC at [93:1]

  The issue was resolved by mapping the source DECIMAL data type to DECIMAL in Google BigQuery.

• Schema rule error

  After making changes to an existing schema rule in a data task, the following error would occur:

  QRI SQL error not implemented

• Missing /QTQVC/QRI table from the SAP transport

  When preparing a landing task which connected to a SAP Application source, the task would complete successfully, but the following error would be reported in the repsrv.log log file:

  Invalid object name 'hk1./QTQVC/QRI'

Standard

Enhancements

• Support for metadata changes in replication and lake landing tasks without needing to drop and recreate all tables

  Previously, when a metadata change occurred, all tables would be dropped and recreated even if the metadata change did not affect all tables. Now, only the changed tables will be dropped and recreated, thereby improving performance.

• Support for automatic loading of new tables in replication tasks

  Tables created in the source database during the replication data task that match the include pattern, will now be automatically captured during CDC (Change data capture).

• Microsoft SQL Server (MS-CDC) source enhancements

  • Timestamp and transaction ID preservation

    Previously, when reading multiple transactions from Microsoft SQL Server (MS-CDC), Qlik Talend Data Integration would preserve the transaction ID and timestamp of the first transaction only. On the target, this gave the appearance of the records being part of a single transaction. Now, Qlik Talend Data Integration will preserve the original transaction ID and timestamp for each individual record. This benefits customers who wish to leverage the Transaction ID and Timestamp header columns in Change tables.

  • Transaction ID bytes order

    Previously, the bytes in the transaction ID were encoded in reverse order. From this version, the bytes will be encoded in the correct order. Information Customers who would rather preserve the existing behavior can do so using internal parameters. For details, please contact Qlik Support.

• Support for caching SHA-2 pluggable authentication with MySQL

  Qlik Talend Data Integration now supports caching SHA-2 pluggable authentication (caching_sha2_password) when working with MySQL sources or targets, either on-premises or in the cloud. In previous versions, only Native Pluggable Authentication (mysql_native_password) was supported.

• Amazon Redshift data type mapping changes

  BYTES and BLOB data types will now be mapped to VARBYTE on Amazon Redshift instead of VARCHAR.

Newly certified platforms, endpoints and versions

• Data sources

  • PostgreSQL 16.x

  • MySQL 8.1
  • IBM DB2 for z/OS 3.1

• Target platforms

  • PostgreSQL 16.x
  • MySQL 8.1
  • Databricks 14.3 LTS

• Drivers

  • SQL Server ODBC Driver 18.3

End of support

Support for the following database versions has been discontinued:

• All Oracle versions and drivers earlier than Oracle 19.x
• Microsoft SQL Server 2014
• MySQL 5.7
• PostgreSQL 11
• IBM DB2 for LUW 10.5
• IBM DB2 for z/OS: z/OS 2.3

Resolved issues

• SAP (Application) source

  When a captured cluster document change deleted all rows of all its captured tables, missing DELETE operation and unnecessary assertion messages would be encountered.

• Security vulnerabilities

  Updated Microsoft Authentication Library for Java (MSAL4J) and Bouncy Castle to versions without known vulebrabilities.

• Snowflake target

  • The task would sometimes fail when using Snowflake internal storage.
  • The task would fail when the target schema name was Japanese Katakana.

• Oracle source

  When resuming a task with an Oracle source, the task would continue to wait for a deleted Archived Redo Log instead of failing with an appropriate error.

Standard

Enhancements

• Snowflake metadata schema

  From this version, the Snowflake metadata schema (for staged files) will be created if it does not exist.

Resolved issues

• After a change was made to a Rename dataset rule (that concatenated the table name with the schema name) and a View was added to the landing task, the Prepare operation would fail with the following error:

  QRI SQL error not implemented.

• Updated the java_file_factory component to a version without any known vulnerabilities.
• Updated org.postgresql:postgresql to a version without any known vulnerabilities.
• When moving data to SQL Server, the Prepare operation would fail when index names exceeded 128 characters. The issue was resolved by altering the logic to create shorter index names.

• When moving data from SQL Server, a missing column name in the table definitions would cause an infinite notification loop in the repsrv.log file, with the following message:

  mssql_resolve_sqlserver_table_column_attributes(...) failed to find column

Standard

Enhancements

• Parallel metadata retrieval

  Added support for concurrently retrieving the metadata for multiple tables.

• Improved monitoring performance

  The monitoring information for landing and replication tasks will now be updated every 10 seconds (instead of every 60 seconds), providing a more precise indication of their current status.

• Newly supported region

  Japan is now supported as a Qlik Cloud tenant region.

Resolved issues

• The retry interval between the data gateway and Qlik Cloud would continually increase, but never be reset (unless the service was restarted).
• When moving data from a SaaS application source, tables would sometimes enter an error state during reload.

• Tasks with a MySQL source would sometimes fail with the following error during CDC:

  Read next binary log event failed; mariadb_rpl_fetch error 0 Error reading binary log.

• Previously, CDC audit events would only be logged for landing tasks. Now, they will be logged for replication tasks ask well.
• When moving data from SQL Server (MS-CDC), tasks with numerous tables would sometimes take several hours to start.

• When the source table contained CLOB columns and the "Limit LOB size" value exceeded 10240, replication to Snowflake would fail with the following error:

  Invalid character length: 0

Standard

Support for installing Data Movement gateway on Red Hat 9.x

Customers can now install Data Movement gateway on Red Hat 9.x or on any corresponding and compatible Linux distribution.

Setting up Data Movement gateway

Changes to Data Movement gateway service commands

The commands for stopping, starting and checking the status of the Data Movement gateway service have changed.

Data Movement gateway service commands

Changes to Amazon Redshift data type mapping

• The BOOLEAN data type, which was mapped to VARCHAR(1) in Amazon Redshift, will now be mapped to BOOLEAN.
• The BYTES and BLOB data types, which were mapped to VARCHAR(1) in Amazon Redshift, will now be mapped to VARBINARY (length).

Newly supported databases, database versions, and driver versions

This section lists the newly supported databases, database versions, and driver versions.

• Newly supported data source versions and editions

  The following data source versions are now supported:

  • Azure Database for MySQL - Flexible Server (Supported via the MySQL source connector)
  • MariaDB 10.4 - 10.11 (previously 10.4 and 10.5)

• Newly supported target data platforms and editions

  The following data target versions are now supported:

  • Azure Database for MySQL - Flexible Server (Supported via the MySQL target connector)
  • Databricks: Databricks 13.3 LTS and Serverless SQL Warehouse

• Newly supported SAP HANA driver version

  Customers with a SAP HANA source who want to install Data Movement gateway on Red Hat Linux 9.x, must install SAP HANA ODBC 64-bit Driver version 2.0.19 or later.

End-of-support database versions

This section provides information about end-of-support database versions.

• Support for the following data source versions has been discontinued:

  • Oracle 11.x
  • SAP HANA 1.0

Resolved issues

• Installing Data Movement gateway without providing a server password would not allow the tenant and proxy URLs to be configured in one command.
• Moving data from a Salesforce (SaaS application) data source would print a large number of redundant warnings, thereby impacting data loading performance.
• When retrieving changes for a SaaS application data source, if an error occurred retrieving changes for one of the tables, that table would be suspended and removed from the pipeline. Now, when encountering an error, the task will try to retrieve the changes up to three times before suspending the table.

Standard

New data lake targets for Replication projects

We are continuing to expand the supported targets for the Replication project in Qlik Cloud Data Integration, in addition to Amazon S3, you can now choose Azure Data Lake Storage (ADLS) and Google Cloud Storage (GCS) for data lake delivery, as Parquet, JSON or CSV file formats.

Resolved issues:

• IBM DB2 for LUW source connector

  The connection to the IBM DB2 for LUW would fail when the size of files needed for the connection (such as the SSL client certificate and the keystore file) exceeded 4 KB.

• Driver for IBM DB2 for LUW and IBM DB2 for z/OS source connectors

  The DB2 driver installation would fail when using the driver installation utility.

Standard

Support for Microsoft Fabric as a data pipeline target

Microsoft Fabric joins the ever-expanding list of data warehouses that can be used as targets in data pipeline projects.

Resolved issues:

Updated the Snowflake driver version in the driver installation utility.

Standard

Expanded Support for Private Connections

A private connection can be used to ensure your data traffic remains secure and compliant. It simplifies both network management and security of your VPC (Virtual Private Cloud) without the need to open inbound firewalls ports, proxy devices or routing tables. Qlik Cloud Data Integration already supports private connections to Snowflake, Microsoft SQL Server and Amazon S3 data pipeline targets. With this release, customers can additionally utilize private connections when moving data to Databricks, Microsoft Azure Synapse, Google BigQuery, and Amazon Redshift.

PostgreSQL-based data sources

• All PostgreSQL-based data sources - Multirange data type support: The following multirange data types are now supported with all PostgreSQL-based data sources (on-premises and cloud).

  • INT4MULTIRANGE
  • INT8MULTIRANGE
  • NUMMULTIRANGE
  • TSMULTIRANGE

• AWS Aurora Cloud for PostgreSQL data source - Non-superuser role support: The user specified in the PostgreSQL connector no longer needs to have the superuser role to move data from an AWS Aurora Cloud for PostgreSQL data source. This is especially useful for organizations with corporate security policies that prevent them from granting superuser access to non-privileged users.

Microsoft Azure SQL (MS-CDC) data source: Geo-replication support

Customers moving data from a Microsoft Azure SQL (MS-CDC) data source can now use a Geo Replica database.

Oracle data source: HSM (Hardware Security Module) support

Qlik Cloud Data Integration now supports reading data from Oracle encrypted tablespaces and encrypted columns during CDC.

New regions support

Qlik Cloud Data Integration now supports tenants in DE and UK regions.

Newly supported versions and third-party software

This section lists newly supported database and driver versions.

• Newly supported data source versions. The following data source versions are now supported:

  • Microsoft SQL Server 2022
  • Oracle 21c
  • PostgreSQL 15.x
  • DB2 13.1 (when working with IBM DB2 for z/OS)
  • IBM DB2 for iSeries 7.5

• Newly supported target data platform versions. The following data target versions are now supported:

  • Databricks (Cloud Storage): Databricks 12.2 LTS and Databricks SQL Serverless

• Driver versions. The following ODBC driver versions are now supported:

  • IBM Data Server Client 11.5.8 for IBM DB2 for z/OZ and IBM DB2 for LUW
  • Simba ODBC driver 3.0.0.1001 for Google Cloud BigQuery
  • MySQL ODBC Unicode Driver 64-bit 8.0.32

Microsoft SQL Server driver - Required upgrade

Customers moving data to or from Microsoft SQL Server need to upgrade their SQL Server ODBC driver version to 18.x or later. Note that continuing to use SQL Server ODBC Driver 17.x might result in data errors. Upgrading the driver can be done using either the driver installation utility or manually. For instructions, see Driver setup.

End-of-support database versions

This section provides information about end-of-support database versions.

• End-of-support data source versions. Support for the following data source versions has been discontinued:

  • PostgreSQL 10
  • MySQL 5.6
  • MariaDB 10.3

• End-of-support target data platform versions. Support for the following data target versions has been discontinued:

  • Databricks 9.1 LTS

2022.11.74

Standard

August 15, 2023

Replication support

We take a cloud-first approach that enables rapid innovation and adoption. However, that does not mean that we are cloud-only. As part of our continued commitment to improving the long-term value for our customers, we are pleased to announce the release of a new use case for Replication when creating data projects. The new Replication use case is in addition to the existing ability to create data pipelines for all your data integration needs, such as Data Warehouse modernization.

The Replication project supports real-time data replication from supported data sources to a supported target.

Starting with this latest release, the first target that supports replicating data in real time, is Microsoft SQL Server. In addition to supporting Microsoft SQL Server on-premises, the following cloud providers are also supported:

• On-premises
• Amazon RDS
• Google Cloud
• Microsoft Azure (Microsoft Azure Managed Instance and Microsoft Azure Database)

Microsoft Azure Synapse Analytics target - Required driver upgrade

Customers moving data to Microsoft Azure Synapse Analytics need to upgrade their SQL Server ODBC driver version to 18.x or later. Note that continuing to use SQL Server ODBC Driver 17.x might result in data errors. Upgrading the driver can be done using either the driver installation utility or manually. For instructions, see Driver setup.

New "Load data from source" option for SaaS application sources

A new Load data from source option was introduced, which allows customers to read their data directly from the source during Full Load, instead of using the cached data.

For more information on this option including use cases, see Landing settings.

Updated CA certificates

Data Movement gateway 2022.11.74 includes updated CA certificates, which are needed for authenticating the Qlik Cloud tenant. The updated CA certificates also provide support for the Ireland and Frankfurt regions. Therefore, customers with Qlik Cloud tenants in Ireland or Frankfurt who want to use Qlik Cloud Data Integration must upgrade to this version.

Resolved issue

Tasks landing data from an Oracle source would sometimes fail when a wide table contained unused or unsupported columns, or LOB columns that were not replicated.

2022.11.70

Standard

June 28, 2023

Improved driver installation utility for SAP clients

In previous versions, customers needed to run the "source arep_login.sh" command when installing SAP clients. From this version, it is no longer necessary to run this command.

Updated CA certificates

This version includes updated CA certificates, which are needed for authenticating the Qlik Cloud tenant.

Resolved issues

• When a replication task on Data Movement gateway failed and recovered automatically, the recovered state would not be communicated to the Landing data asset in Qlik Cloud.

• End-to-end encryption for Data Movement gateway would not be enabled by default, and was controlled by runtime flags.

2022.11.63

Standard

May 2, 2023

Driver installation utility

This version introduces a driver installation utility that eliminates the need to manually install and configure drivers. The new utility shortens the installation process while significantly reducing the possibility of user error. When the utility is run, the required driver is downloaded automatically, if possible, and installed. If the driver cannot be downloaded (DB2 drivers require login, for example), all you need to do is download the driver, copy it to a dedicated folder on the Data Movement gateway machine, and run the utility.

For an example of using the driver installation utility to install a PostgreSQL driver, see Prerequisites

Snowflake cloud data platform

The Snowflake connector now supports 4-byte emoji characters.

Support for Azure Database for PostgreSQL - Flexible Server as a data source

The PostgreSQL connector can now move data from Azure Database for PostgreSQL - Flexible Server.

Support for Cloud SQL for PostgreSQL as a data source

The PostgreSQL connector can now move data from Cloud SQL for PostgreSQL.

Newly supported data source versions

This version introduces support for the following new data source versions:

• PostgreSQL 14
• DB2 (for IBM DB2 for z/OS) 12.1
• IBM DB2 for z/OS 2.5

Newly supported cloud data platform versions

Databricks 11.3 LTS

Discontinued support for the following data source versions

The following data source versions are no longer supported:

• DB2 (for IBM DB2 for z/OS) 11
• PostgreSQL 9.6
• Microsoft SQL Server 2012
• MariaDB 10.2

Resolved issues

This version resolves the following issues:

• Oracle data source: When stopping and resuming a task, the task would sometimes fail with a “Failed to set stream position on context” error.
• SAP Application source: Changes would not be captured during the landing task.