What's new?

This section describes the new and enhanced features from Replicate May 2022 Service Release 2 until and including Replicate November 2022 SR1.

What's new in endpoints?

This section describes the new and enhanced endpoint features.

New features and enhancements in Replicate November 2022 SR1

The following section describes new and enhanced features introduced in Qlik Replicate November 2022 SR1.

Support for 4-byte emoji characters in Snowflake-based endpoints

Support for replication of data with 4-byte emoji characters was added to the following target endpoints:

• Snowflake on AWS
• Snowflake on Azure
• Snowflake on Google

Features and enhancements first introduced in Replicate November 2022 IR (Initial Release)

The following section describes features and enhancements first introduced in Qlik Replicate November 2022 IR.

New Databricks (Cloud Storage) target endpoint

This version consolidates three Databricks target endpoints - AWS, Google Cloud, and Microsoft Azure - into one unified endpoint.

The new Databricks (Cloud Storage) endpoint offers support for:

• SQL Warehouse and All-purpose cluster types
• Four different storage types: Amazon S3, Google Cloud Storage, Microsoft Azure Data Lake Storage (ADLS) Gen2, and Microsoft Azure Blob Storage.
• Databricks implementations on Microsoft Azure, AWS, and Google Cloud Platform.
• Parquet format when using an SQL Warehouse cluster

Customers using the existing Databricks endpoints do not need to take any action as the new endpoint will automatically replace their existing endpoint, after upgrading to Replicate November 2022.

New SAP ODP (Operational Data Provisioning) source endpoint

This version introduces support for SAP ODP (Operational Data Provisioning) as a source endpoint, enabling replication of CDS Views and various SAP BW objects.

New Amazon MSK target endpoint

This version introduces support for Amazon MSK as a target endpoint.

Databricks Lakehouse (Delta) enhancements

For customers upgrading from Replicate May 2022 Service Release 2, the following enhancements have been added.

Option to create tables as managed or unmanaged

A new option that allows you to choose whether the target tables will be created as Managed or Unmanaged has been added to the Databricks Lakehouse (Delta) target endpoint.

For more information on managed versus unmanaged tables, see https://docs.databricks.com/lakehouse/data-objects.html

Apply changes using SQL MERGE

Databricks Lakehouse (Delta) now supports the Apply changes using SQL MERGE change processing option. For more information on this option, see https://help.qlik.com/en-US/replicate/Content/Global_Common/Content/SharedEMReplicate/Customize%20Tasks/tasks_applychangtunestab.htm#anchor-4.

MongoDB source endpoint enhancements

MongoDB Atlas

The MongoDB source endpoint now supports replication from MongoDB Atlas, in addition to the standard MongoDB support.

LDAP authentication

Added support for LDAP authentication when using MongoDB Standard.

CDC support when replicating from Google Cloud SQL for PostgreSQL

Support for CDC has been added to the Google Cloud SQL for PostgreSQL source endpoint.

Microsoft Azure Synapse Analytics target endpoint enhancements

Support for the Synapse COPY statement

Customers can now choose whether to load data into Microsoft Azure Synapse Analytics tables using PolyBase or the COPY statement. In certain environments, the COPY statement might offer better performance.

Performance tuning

The new Use compression and Number of files to apply in a batch (for Full Load) options can improve performance, depending on the size and number of tables being replicated.

Google Cloud BigQuery enhancements

The following options were added to the Advanced tab of the Google Cloud BigQuery target endpoint:

• Proxy support: Customers can now connect to Google Cloud BigQuery target via a proxy server.
• Use compression: When selected, the endpoint will compress (using gzip) the CSV files before uploading them to Google Cloud BigQuery
• Other: Customers can now choose Other from the Location drop-down list and set the location code in the Region code field. Region codes are listed in https://cloud.google.com/bigquery/docs/locations.

Support for Azure Database for PostgreSQL - Flexible Server as a source

Customers can now replicate from Azure Database for PostgreSQL - Flexible Server using the standard PostgreSQL source endpoint.

Support for IBM Netezza Performance Server (Cloud) as a target

Customers can now replicate to IBM Netezza Performance Server using the standard IBM Netezza target endpoint.

Salesforce (Incremental Load) source endpoint changes and enhancements

Option to select the API automatically

For both Full Load and CDC loading, a Select API automatically option has been added. When selected, If the number of records per table matches or exceeds the value specified in the API switchover threshold (records) field, Replicate will retrieve the data using the Bulk API (if the object supports Bulk API); otherwise, it will retrieve the data using the SOAP API.

Relocation of the "Check for changes" option

The Check for changes option has been moved to the Advanced tab.

Support for Azure Active Directory Service Principal and Azure Active Directory authentication

Support for Azure Active Directory Service Principal authentication was added to the following endpoints:

• Microsoft Azure SQL Managed Instance source
• Microsoft Azure SQL (MS-CDC) source
• Microsoft Azure SQL Database target

Support for Azure Active Directory authentication was added to the following endpoints:

• Microsoft Azure Managed SQL Instance source
• Microsoft Azure SQL (MS-CDC) source

Support for triggered events when using the Microsoft Azure ADLS target endpoint

When the Storage type is set to Azure Data Lake Storage (ADLS) Gen2, the Microsoft Azure ADLS target endpoint now supports Azure Storage Events. To learn more about Azure Storage Events, see https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-event-overview

Features and enhancements first introduced in Replicate May 2022 SR2

The following section describes features and enhancements first introduced in Replicate May 2022 SR2.

New Google Cloud SQL for SQL Server target endpoint

Customers can now replicate data from any supported source to Google Cloud SQL for SQL Server.

New Microsoft SQL Server (MS-CDC) source endpoint

Customers can now replicate data from Microsoft SQL Server using the MS-CDC mechanism. When replicating data from Microsoft SQL Server using MS-CDC, changes are captured from Microsoft Change Tables as opposed to the SQL Server transaction log. Using MS-CDC is simpler from a setup perspective and it eliminates the need to use a high-privileged account to access the transaction logs. On the other hand, because the MS-CDC mechanism is based on Change Tables, the replicated transaction patterns are different from the original DML transaction that occurred in the database. It should be noted that while this is still transactionally consistent, it does not represent the original transactions.

New Databricks Lakehouse (Delta) target endpoint

This version introduces the new Databricks Lakehouse (Delta) target endpoint, offering support for:

• Three different storage options: Amazon, Google, and Microsoft Azure Data Lake Storage Gen2.
• Databricks implementations on Azure, AWS, and Google Cloud Platform.
• Supports both SQL Warehouse and All-purpose cluster types
• All replication options - Full Load, Store Changes and Apply Changes - loading directly into Databricks Delta tables

The new Databricks Lakehouse (Delta) endpoint effectively replaces the existing Microsoft Azure Databricks Delta endpoint, which will cease to be available from Replicate November 2022. Customers using the existing Microsoft Azure Databricks Delta do not need to take any actions as the new endpoint will automatically replace the existing endpoint, after upgrading to Replicate November 2022.

Amazon S3 target endpoint - AWS PrivateLink support

Customers using the Amazon S3 target endpoint can use AWS PrivateLink to connect to EC2 instances in a virtual private cloud (VPC). To enable this functionality, a Use AWS PrivateLink option and VPC Endpoint URL field have been added to the General tab of the Amazon S3 target endpoint settings.

SAP HANA source endpoint - SSL support

An option to connect using SSL was added to the General tab of the SAP HANA source endpoint settings.

SAP HANA target endpoint - Multi tenant support

When replicating to SAP HANA target endpoint, customers can now choose Single tenant or Multi tenant according to their database architecture.

SAP Application source endpoint - SNC support

This version introduces support for connecting to the SAP Server using Secure Network Communication (SNC).

Parquet format support

Support for creating files in Parquet format has been added to the following endpoints:

• Google Cloud Storage

• Microsoft Azure ADLS

To learn about the benefits of Parquet format, see https://databricks.com/glossary/what-is-parquet

FIPS- compliant endpoints

FIPS compliance in Replicate endpoints depends on whether the database or ODBC vendor supports the FIPS cryptographic standard, which is not always the case. To date, the endpoints listed in the table below have been certified as FIPS-compliant. To work with these endpoints in FIPS mode, in addition to the endpoint-specific prerequisites described below, you also need to fulfill the Replicate FIPS prerequisites described in FIPS Compliance.

Snowflake-based endpoints

To better reflect the expected value, the Snowflake URL field in the endpoint settings has been changed to Snowflake Account/Host.

What's new on the server side?

This section describes the new and enhanced server features.

Features and enhancements first introduced in Replicate November 2022 IR (Initial Release)

Replicate November 2022 SR1 does not include any new or enhanced engine features. This section describes the new and enhanced engine features from Replicate May 2022 Service Release 2 until and including Replicate November 2022 (Initial Release).

Checking for NOT NULL violation constraints

A new NOT NULL constraint violation handling option has been added to the Error Handling|Data Errors tab:

For source tables, Replicate can explicitly check for NULL values in each data column of each record, and whether the same column in the corresponding target table is marked as NOT NULL. In such cases, applying the record to the target will trigger a NOT NULL constraint violation that will be difficult to recover from if the task is running in Batch Optimized Apply mode. Therefore, the record is not applied to the target. Instead, the record is handled as an error (see below) without affecting other records in the batch being applied. While this way of handling NOT NULL constraint violation is very effective, it can impact performance, and if the target database does not actually enforce NOT NULL constraint violations, Replicate can be configured to skip that check in order to save CPU time and improve performance. Because Replicate generally knows what target databases enforce NOT NULL constraint violations, it is safe to always let Replicate activate this protection based on the target endpoint type. Still, Replicate gives you the option to override this default and either force a check (with its slight overhead) or disable the check and risk NOT NULL constraint violations.

DELETE and INSERT when updating a primary key column

To ease downstream processing, A new DELETE and INSERT when updating a primary key column has been added to the task settings' Change Processing Tuning tab. For tasks configured with streaming target endpoints (for example, Kafka), the DELETE + INSERT statements will be executed on the actual target tables.

For all other target endpoints, turning on DELETE and INSERT when updating a primary key column only impacts the Change Tables, which are created when the Store Changes replication option is enabled. Note that turning on DELETE and INSERT when updating a primary key column will not impact how Replicate applies Primary Key updates when the Apply Changes replication option is enabled.

Features and enhancements first introduced in Replicate May 2022 SR2

The following section describes features and enhancements first introduced in Replicate May 2022 SR2.

FIPS Compliance

FIPS (Federal Information Processing Standards) is a set of standards developed by the United States Federal Government for use in computer systems. FIPS 140-2 is the subset of standards which defines approved encryption algorithms used for handling sensitive information. Replicate will work in FIPS mode if the following prerequisites have been met:

• A FIPS-compliant Replicate version must be installed. To obtain the FIPS-compliant version, contact Qlik Support.

  Warning noteUpgrading from a non FIPS-compliant Replicate is not supported. You need to uninstall your current version before installing the FIPS-compliant version. Once you have installed the FIPS-compliant version, the next Replicate version that you upgrade to does not need to be FIPS-compliant (as you will already have all the FIPS-compliant software on your machine).
• The operating system on which Replicate is installed is running in FIPS mode.

See also: FIPS- compliant endpoints