What's new?

This section describes the new and enhanced features introduced in Qlik Replicate May 2022.

What's new in endpoints?

New Salesforce (Incremental Load) source endpoint

This version introduces support for the Salesforce (Incremental Load) source endpoint. Though similar to the existing Salesforce (Streaming CDC) source endpoint (previously called Salesforce), there are significant differences in their implementation.

• Full Load: Salesforce (Streaming CDC) uses Bulk API v1 for Full Load while Salesforce (Incremental Load) uses Bulk API v2.

• CDC: Salesforce (Streaming CDC) uses the Salesforce Streaming API which is subject to several limitations, although it does provide better performance. Salesforce (Incremental Load) is based on change polling which, although slower than Salesforce (Streaming CDC), has the advantage of supporting almost all Salesforce objects.

The decision which endpoint to use (Incremental Load or Streaming CDC) will depend on your specific replication requirements, such as the objects you need to replicate and latency considerations. To get a better idea of which endpoint is more suitable for your business needs, review the limitations and considerations for each endpoint in the online help.

The main differentiators are:

• Incremental Load supports a wider set of Salesforce objects and Bulk API v2, with polling based CDC every X minutes.
• Streaming CDC supports low latency capture of data changes from Salesforce from a limited set of objects.

See also:

• Salesforce (Streaming CDC) limitations and considerations
• Salesforce (Incremental Load) limitations and considerations
• What’s the Difference Between Bulk API 2.0 and Bulk API?

New Google Cloud SQL for SQL Server target endpoint

Customers can now replicate data from any supported source to Google Cloud SQL for SQL Server.

New Microsoft SQL Server (MS-CDC) source endpoint

Customers can now replicate data from Microsoft SQL Server using the MS-CDC mechanism. When replicating data from Microsoft SQL Server using MS-CDC, changes are captured from Microsoft Change Tables as opposed to the SQL Server transaction log. Using MS-CDC is simpler from a setup perspective and it eliminates the need to use a high-privileged account to access the transaction logs. On the other hand, because the MS-CDC mechanism is based on Change Tables, the replicated transaction patterns are different from the original DML transaction that occurred in the database. It should be noted that while this is still transactionally consistent, it does not represent the original transactions.

New Databricks Lakehouse (Delta) target endpoint

This version introduces the new Databricks Lakehouse (Delta) target endpoint, offering support for:

• Three different storage options: Amazon, Google, and Microsoft Azure Data Lake Storage Gen2.
• Databricks implementations on Azure, AWS, and Google Cloud Platform.
• All replication options - Full Load, Store Changes and Apply Changes - loading directly into Databricks Delta tables

The new Databricks Lakehouse (Delta) endpoint effectively replaces the existing Microsoft Azure Databricks Delta endpoint, which will cease to be available from Replicate November 2022. Customers using the existing Microsoft Azure Databricks Delta do not need to take any actions as the new endpoint will automatically replace the existing endpoint, after upgrading to Replicate November 2022.

Amazon S3 target endpoint - AWS PrivateLink support

Customers using the Amazon S3 target endpoint can use AWS PrivateLink to connect to EC2 instances in a virtual private cloud (VPC). To enable this functionality, a Use AWS PrivateLink option and VPC Endpoint URL field have been added to the General tab of the Amazon S3 target endpoint settings.

SAP HANA target endpoint - Multi tenant support

When replicating to SAP HANA target endpoint, customers can now choose Single tenant or Multi tenant according to their database architecture.

All-region support for target endpoints with AWS storage

This version introduces the Auto-Detect (the default) and Other options for bucket regions ,thereby providing support for all bucket regions. Although Auto-Detect usually eliminates the need to select a specific region, some regions (for example, those such as AWS GovCloud with rigorous security requirements) still might need to be selected explicitly. If the region you require cannot be auto-detected and does not appear in the regions list, select Other and then specify the region code using either an internal parameter or the dedicated Region code field (depending on the endpoint).

The change affects the following target endpoints:

• Amazon EMR
• Amazon Kinesis Data Streams
• Amazon S3
• Hortonworks Data Platform (HDP)
• Databricks on AWS
• Microsoft Azure Databricks Delta

• Databricks Lakehouse (Delta)

  Information noteSupported from Replicate May 2022 Service Release 02 only.
• Amazon Redshift
• Snowflake on AWS

Support for Government Cloud and China Cloud

This version introduces support for connecting to Azure resource on Government Cloud or China Cloud when using the following target endpoints:

• Microsoft Azure ADLS
• Microsoft Azure Event Hubs
• Microsoft Azure HDInsight
• Microsoft Azure Synapse Analytics
• Microsoft Azure Databricks
• Microsoft Azure Databricks Delta

• Databricks Lakehouse (Delta)

  Information noteSupported from Replicate May 2022 Service Release 02 only.
• Snowflake on Azure

SAP endpoints - SNC support

This version introduces support for connecting to the SAP Server using Secure Network Communication (SNC).

Supported with the following source endpoints:

• SAP Extractor

• SAP Application

  Information noteSNC for the SAP Application source endpoint is supported from Replicate May 2022 Service Release 02 only.

Parquet format support

Support for creating files in Parquet format has been added to the following endpoints:

• Amazon S3 target

• Google Cloud Storage

  Information noteSupported from Replicate May 2022 Service Release 02 only.

• Microsoft Azure ADLS

  Information noteSupported from Replicate May 2022 Service Release 02 only.

To learn about the benefits of Parquet format, see https://databricks.com/glossary/what-is-parquet

Google Cloud BigQuery permission changes

Customers replicating to Google Cloud BigQuery no longer require the Project > Viewer permission.

The updated permissions are as follows:

• BigQuery > BigQuery Data Editor
• BigQuery > BigQuery Job User

Salesforce (Streaming CDC) source endpoint - PICKLIST data type enhancement

As a result of the Salesforce SOAP API and Bulk API being upgraded from version 44.0 to 53.0, the PICKLIST data type length now supports 255 characters (previously it supported only 40 characters).

Microsoft Azure SQL (MS-CDC) enhancements

This version introduces support for:

• Using Microsoft Azure SQL Database as a source
• Replicating a table or schema with an apostrophe (') in its name

FIPS- compliant endpoints

FIPS compliance in Replicate endpoints depends on whether the database or ODBC vendor supports the FIPS cryptographic standard, which is not always the case. To date, the endpoints listed in the table below have been certified as FIPS-compliant. To work with these endpoints in FIPS mode, in addition to the endpoint-specific prerequisites described below, you also need to fulfill the Replicate FIPS prerequisites described in FIPS Compliance.

What's new in engine?

New task notification for CDC interruptions

The new No changes were captured in the last <n> minutes notification will alert you if no changes were captured from the source database within the specified time limit. With most databases, receiving this notification will usually indicate a connectivity problem. However, with the following databases, it could simply mean that no changes occurred during the specified period.

• Microsoft SQL Server
• IBM DB2 for LUW
• IBM DB2 for iSeries

When the task resumes capturing changes, an additional notification that change capture has resumed will be sent, and the notification timer will be reset.

New default action for data truncation errors when using parallel apply

Previously when using the Apply batched changes to multiple tables concurrently option, the default error-handling action for data truncation errors would be to suspend the relevant table. Now, if a data truncation error is encountered, the default will be to ignore the record and continue the replication of the table.

FIPS Compliance

FIPS (Federal Information Processing Standards) is a set of standards developed by the United States Federal Government for use in computer systems. FIPS 140-2 is the subset of standards which defines approved encryption algorithms used for handling sensitive information. Replicate will work in FIPS mode if the following prerequisites have been met:

• A FIPS-compliant Replicate version must be installed. To obtain the FIPS-compliant version, contact Qlik Support.

  Warning noteUpgrading from a non FIPS-compliant Replicate is not supported. You need to uninstall your current version before installing the FIPS-compliant version. Once you have installed the FIPS-compliant version, the next Replicate version that you upgrade to does not need to be FIPS-compliant (as you will already have all the FIPS-compliant software on your machine).
• The operating system on which Replicate is installed is running in FIPS mode.

INSERT optimization

A new Optimize inserts option has been added to the Change Processing Tuning tab in the task settings. In order to use the new option, Apply changes using SQL MERGE must be turned on. When both of these options are tuned on and the changes consist of INSERTs only, Replicate will perform INSERTs instead of using SQL MERGE. Note that while this will usually improve performance and thereby reduce costs, it might also result in duplicate records in the target database.

Support for replacing endpoint values using the Enterprise Manager API

This version adds the ability to edit specific endpoint values using the Enterprise Manager API. When not editing the value directly in the user interface, this was previously only possible using export, edit, and import operations. The functionality is available through the PatchEndpoint method, which is available in all of the APIs (REST, .NET, and Python).

"Use SSL" renaming

The Use SSL option in the notification email settings has been renamed to the more accurate Use secure email (SMTPS).