Skip to main content Skip to complementary content

Data Errors

Click the Data Errors sub-tab and then click Change to Task Policy.

Information noteData error handling is not relevant for Full Load replication. It is applicable to Apply Changes (CDC) replication only.

Data truncation errors

  • For a data truncation error: Click the triangle to open the list and select what happens when an truncation occurs in one or more specific records. You can select one of the following from the list:

    • Ignore record: The task continues and the error is ignored.
    • Log record to the exceptions table (default): The task continues and the error is written to the exceptions table.
    • Suspend table: The task continues, but data from the table with the error record is moved into an error state and its data is not replicated
    • Stop task: The task is stopped and manual intervention is required.
Information note

Data truncation error handling is only supported in the following with the following target endpoints: MySQL, PostgreSQL, Oracle, Microsoft SQL Server, SAP Sybase ASE, File, Amazon Redshift, Microsoft Azure Database for PostgreSQL, Google Cloud SQL for PostgreSQL, Microsoft Azure Database for MySQL, Google Cloud SQL for MySQL, and Microsoft Azure SQL Database

NOT NULL constraint violations

For source tables, Replicate can explicitly check for NULL values in each data column of each record, and whether the same column in the corresponding target table is marked as NOT NULL. In such cases, applying the record to the target will trigger a NOT NULL constraint violation that will be difficult to recover from if the task is running in Batch Optimized Apply mode. Therefore, the record is not applied to the target. Instead, the record is handled as an error (see below) without affecting other records in the batch being applied. While this way of handling NOT NULL constraint violation is very effective, it can impact performance, and if the target database does not actually enforce NOT NULL constraint violations, Replicate can be configured to skip that check in order to save CPU time and improve performance. Because Replicate generally knows what target databases enforce NOT NULL constraint violations, it is safe to always let Replicate activate this protection based on the target endpoint type. Still, Replicate gives you the option to override this default and either force a check (with its slight overhead) or disable the check and risk NOT NULL constraint violations.

Information noteWhen a batch update fails due to a NOT NULL constraint violation, Replicate has no way of knowing which of the records in the batch caused that violation. Consequently, Replicate switches to a "one-by-one" strategy of applying changes. While this strategy is acceptable for OLTP databases (although even with those databases, it is much slower than Batch optimized apply), it is not practical for analytical data warehouses, specifically those that use columnar storage in files or micro-partitions (for example, Snowflake, Azure Synapse Analytics, Google BigQuery, and Databricks). As such databases are not designed for one-by-one record updates, performing such updates might take an unreasonable amount of time and consume excessive resources. Moreover, If the target database is a cloud data warehouse that charges based on activity, the one-by-one error handling strategy could result in extremely high costs.

    For NOT NULL constraint violations, select one of the following:  

    • Endpoint-determined: This means that the endpoint type (which can be an OLTP database or an analytical data warehouse) determines whether Replicate checks for NOT NULL constraint violations.

      Information noteSome target endpoints do not support checking for NOT NULL constraint violations. In such cases, the tooltip next to the Endpoint-determined option will indicate that the current target endpoint does not allow checking for NOT NULL constraints.

      For OLTP-based target endpoints, the default is not to check for NOT NULL constraint violations as it is less critical (from a cost perspective) if a NOT NULL constraint violation occurs during the task. However, if a NOT NULL constraint violation occurs on any of the tables during the task, Replicate will switch to one-by-one mode, which will impact performance to a certain degree (depending on the number and size of the tables being replicated).

       

      You can override the endpoint-determined handling by selecting either the Check or Don't check options described below.

      Information noteThe string (Check) or (Don't check) will appear after Endpoint-determined according to the selected target endpoint type.
    • Check: Select this if you always want Replicate to check for NOT NULL constraint violations. This might be useful if you know beforehand that some of the source table columns contain NULL values and the corresponding target table columns are configured with NOT NULL constraints. In this case, Replicate will not switch to one-by-one mode, but will continue change processing in Batch Optimized Apply mode. If a source table that will violate a NOT NULL constraint is detected during the check, Replicate will take the action described below. The default action is to suspend the table.

    • Don't check: Select this if you never want Replicate to check for NOT NULL constraint violations. This is recommended if you are confident that no such violations will occur during the task or if you do not mind Replicate switching to one-by-one mode if a NOT NULL constraint violation occurs.

    For the Endpoint-determined and Check options, you can select how Replicate will handle the NOT NULL constraint violation:

    • Suspend table: This is the default. The task continues, but tables with NULL values are suspended and their data is not replicated.
    • Ignore record: The task continues and the error is ignored.
    • Log record to the exceptions table: The task continues, but records with NULL values are written to the exceptions table.
    • Stop task: The task is stopped and manual intervention is required.

Other data errors

  • For other data errors: Click the triangle to open the list and select what happens when an error occurs in one or more specific records. You can select one of the following from the list:

    • Ignore record: The task continues and the error is ignored.
    • Log record to the exceptions table (default): The task continues and the error is written to the exceptions table.
    • Suspend table: The task continues, but data from the table with the error record is moved into an error state and its data is not replicated
    • Stop task: The task is stopped and manual intervention is required.

Escalating error handling

  • Escalate error handling when other data errors reach (per table): Select this check box to escalate error handling when the number of non-truncation data errors (per table) reaches the specified amount. Valid values are 1-10,000.

    Information note

    The escalation options are not available when the Apply changes using SQL MERGE task setting is enabled.

    • Escalation action: Choose what action Replicate should perform when error handling is escalated. Note that the available actions are dependent on the action selected from the For other data errors drop-down list described above.

      • Log record to the exceptions table: The task continues, but the record with the error is written to the exceptions table.
      • Suspend table (default): The task continues, but data from the table with the error record is moved into an error state and its data is not replicated.

        Information note

        The behavior differs according to the Change Processing Mode:

        • In Transactional apply mode, the last changes will not be replicated

        • In Batch optimized apply mode, a situation is possible where there will be no replication of data or data replication will occur in part

      • Stop task: The task is stopped and manual intervention is required.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!