Skip to main content

System Settings: Level Control

ON THIS PAGE

System Settings: Level Control

Level Control allows users to control the level at which specified data is managed in the metadata catalog. All levels (highest to lowest: Managed (default), Registered, and Addressed) capture audit logs of level changes. Consider a typical use case for Level Control functionality: Data in a quarterly report file may no longer be required though the metadata is still relevant—the entity containing the records can be demoted from Managed to Addressed where the metadata can be refreshed (via load) as needed or promoted as necessary back to Managed. Promotion and demotion are user-driven actions, the user changes level for a source or entity through grid drop-down level change (screenshot below). Note that Addressed or Registered entities can be published and those entities are temporarily on-demand published in order to extract metadata, statistics, profile and sample data and automatically returned to original level.

Note: Level control is inherited by entities from sources but can be overwritten at the lower level.
Levels and load type definitions

Level

Load type definition

Managed

Data. Objects are fully ingested with full data profiled and validated, generating an extensive catalog of information around the objects and data.

Registered

Statistics. Objects exist or have been added to the lake independent of Qlik Catalog or have been demoted from Managed status within Qlik Catalog. Analysis of Registered objects produces summary statistics (profiling and validation information) and metadata about the data is retained. Sample data can be produced (user selected option).

Addressed

Metadata. Users are able to see database schema metadata on accessible platforms within the enterprise network without having to move it into the lake.

Users can:

  • Promote data from Addressed to Registered|Managed or from Registered to Managed.
  • Demote data from Managed to Registered|Addressed or from Registered to Addressed.

Default level is controlled in three ways:

  • Upon Ingest: Inherited from source 
  • Source Grid: Users can control object level by checking one or more sources or entities and selecting Change Level
  • Inclusion in a process (dataload, dataflow, publish) where level is temporarily promoted and then reset to original level

Change Level

Administrative rights are required to access and modify Level Control (governed by access mask) accessed in the administration panel.

If Lock level to only Managed is checked, Default level for Sources is disabled.

Level Control

Checkbox to lock level to managed only

If Lock level to only Managed is unchecked, Default level for Sources can be locked to only allow the selected level to be used. This lock and setting can be toggled. 

Lock level to only Managed: If this setting has been unlocked to allow Registered and Addressed levels for entities, it cannot be relocked. The default setting for objects in Qlik Catalog is Managed (full data ingest and management). Therefore if administrators do not change anything on this screen, application behavior will not change and all objects will continue to be Managed. When Lock is checked, level is set for all sources and entities to Managed. This setting disables the ability to change level for Source Connections.

Default level for new sources (Top Level Managed Control is Un-checked):

  • Managed: If selected (default), Qlik Catalog will keep all entity data (on distribution tables and file system) and profiling and validation metadata gathered upon ingest. If unlocked, users with permissions can change level for individual objects in grid.
  • Registered: If selected, Qlik Catalog will keep only sample data along with the profile data and schema metadata but will delete all the other data from distribution tables and files system that may have imported as part of the data ingestion process. If unlocked, users with permissions can change level for individual objects in grid.
  • Addressed: If selected, Qlik Catalog will only gather schema metadata upon ingest or if the object is demoted from Managed or Registered, Qlik Catalog will remove all data from distribution tables and file system gathered on the data ingestion process, including sample data and profiling statistics. If unlocked, users with permissions can change level for individual objects in grid.
Overview: Level Control

Level

Options

Lock level to only Managed

Once unlocked, this action is not reversible (items cannot be relocked).

When lock is on, all sources and entities are managed, and level cannot be modified:

  • All sources and entities are owned and managed by Qlik Catalog
  • Complete validation
  • Full profiling
  • No promotion required
Default Level Control for New Sources

Level

Description

Managed

Full ingest, tightly coupled to all features/capabilities within Qlik Catalog.

Data load log type: Data

Qlik Catalog makes an internal copy of the external source data

Sample data can be produced (user specified inheritable core_env.properties: default.record.sampling.probability)

Good, Bad, Ugly partitioned record tables maintained in a table view (such as Hive)

Registered

Data has been added into data lake independent of Qlik Catalog OR demoted from Managed.

Data load log type: Statistics

Qlik Catalog will not retain a new copy of the data. When the object is ingested initially Qlik Catalog creates sample data, generates profile statistics and schema metadata

Sample data can be produced if sampling probability meets threshold (user selected option in core_env.properties: default.record.sampling.probability)

Good, Bad, Ugly tables NOT populated/maintained in a table view

Can be used to execute prepare and publish jobs if sample data is produced

On demotion from Managed, system removes the data and table partitions

On promotion from Addressed, system accesses the source, executes dataload to create sample data, generates profile statistics and schema metadata

Addressed

Source has been defined and analyzed outside of Qlik Catalog OR a Managed/Registered object has been demoted.

Data load log type: Metadata

Schema metadata is acquired and maintained

Fields metadata is updated when fields are added or deleted, or changes are made to data type, business name/description, index and properties they are updated in external and internal entities

Can be used for creating prepare dataflows and publish jobs where they are temporarily promoted before being returned to their original level

Note:

If a field name is changed, previous custom properties will not migrate

On demotion to Addressed, only schema metadata is maintained: data in file system and table views do not exist or are no longer maintained in Qlik Catalog. Sample data and profiling statistics are also not retained upon demotion to Addressed.