Skip to main content

Modifying ETL settings

For each ETL set, you can modify the settings according to your needs.

  1. In the Manage ETL Sets window, select an ETL set in the left pane and then click Settings. The Setting - <ETL Set Name> dialog box opens.

  2. In the General tab, edit any of the following settings:

    • Sequential Processing: Select this option if you want all the ETL data warehouse processes to run sequentially, even if they can be run in parallel. This may be useful for debugging or profiling, but it may also affect performance.
    • Default History Resolution: Specifies at what resolution level the history timestamp is saved. If no mapping exists for the From Date (FD) column to the source, Qlik Compose for Data Warehouses uses Minutes. At this level, the timestamp includes the hours and minutes of when the change occurred. If you select a value of Days, the FD timestamp includes only the date, not the hours and minutes. If a mapping exists, then the FD level comes from the source.
  3. In the Advanced tab, edit any of the following settings:

    • Log level: Select the log level granularity, which can be any of the following:
      • INFO (default) - Logs informational messages that highlight the progress of the ETL process at a coarse-grained level.
      • VERBOSE - Logs fine-grained informational events that are most useful to debug the ETL process.
      • TRACE - Logs finer-grained informational events than the VERBOSE level.

      The log levels VERBOSE and TRACE impact performance. Therefore, you should only select them for troubleshooting if advised by Qlik Support.

    • JVM memory settings: Edit the memory for the java virtual machine (JVM) if you experience performance issues. Xms is the minimum memory; Xmx is the maximum memory. The JVM starts running with the Xms value and can use up to the Xmx value.
    • Position in default workflow: Select where you want the data warehouse tasks to appear in the default workflow. For more information on workflows, see Workflows.
    • Optimize for initial load: Optimizes initial load in certain cases. Only select this option if the source tables do not reference missing records, use lookups, map different source records to the same record, do not contain Type 1 self-references, or contain historical records. Note also that when this option is selected, data quality rules, derived attributes and the “Handle duplicates” option are not supported. In the event that the ETL set is used for incremental Full Load (using query-based change processing), clear the check box after the initial load task completes and regenerate the ETL set.
    • Write ETL command duration to the TLOG_PROCLOG table in the data warehouse: This option is useful for troubleshooting performance issues with ETL processes as it records the duration of each ETL command in a special table (named TLOG_PROCLOG) in the data warehouse. You can then use this information to locate ETL commands with abnormal duration times and modify them accordingly.

    • Do not truncate staging tables: Select this option if you want the ETL process to preserve the staging tables. Only use for debugging.
    • Stop processing after populating the staging tables: Select if you do not want to proceed to populating the warehouse. Only use for debugging.
    • Do not drop temporary tables: Select this option if you want to keep the temporary tables created during the ETL process. Only use for debugging.

    When updating a non-null data warehouse column with a null value:

    • Do not change the target value: Select this to keep values unchanged between two mappings for the same record. For instance, if the same record exists in two different source tables (A and B), but the record in Table A has a null value for data that is present in Table B (e.g. ZIP Code). In this case, if the record in Table A arrives after the record in Table B, the target value will be set to null. Selecting this option will prevent such an occurrence.
    • Note:

      When creating a new project, the default behavior is to write NULL instead of keeping the values unchanged between the two mappings.

    • Set the target value to null: Select this if you want the source and target values to correspond. This can be useful, for example, when a person moves address and one of the column values (e.g. "State") changes to null.

      Note:

      When ingesting changes from an Oracle source, this option requires full supplemental logging for all source table columns that exist on the target and any source columns referenced in filters, data quality rules, lookups, and expressions.

  4. To save your changes, click OK.