System Settings: Level Control
Level Control allows users to control the level at which specified data is managed while retaining a metadata catalog. All levels (highest to lowest: 'Managed' (default), 'Registered', and 'Addressed') capture audit logs of level changes. Consider a typical use case for Level Control functionality: Data in a quarterly report file may no longer be required though the metadata is still relevant—the entity containing the records can be demoted from Managed to Addressed where the metadata can be refreshed (via load) as needed or promoted as necessary back to Managed. Promotion/Demotion is a user-driven action, the user changes level for a source or entity through grid dropdown level change (see screenshot below). Note that Addressed or Registered entities can be published and those entities are temporarily on-demand published in order to extract metadata, statistics, profile and sample data and then automatically returned to its original level.
|
Level |
Load type definition |
|---|---|
|
Managed |
Data. Objects are fully ingested with full data profiled and validated, generating an extensive catalog of information around the objects and data. |
|
Registered |
Statistics. Objects exist or have been added to the lake independent of QDC or have been demoted from Managed status within QDC. Analysis of Registered objects produces summary statistics (profiling and validation information) and metadata about the data is retained. Sample data can be produced (user selected option). |
|
Addressed |
Metadata. Users are able to see database schema metadata on accessible platforms within the enterprise network without having to move it into the lake. |
Users can:
- Promote data from Addressed to Registered|Managed or from Registered to Managed.
- Demote data from Managed to Registered|Addressed or from Registered to Addressed.
Default level is controlled in three ways:
- Upon Ingest: Inherited from Source
- Source Grid: Users can control object level by checking one or more sources or entities and selecting 'Change Level'
- Inclusion in a process (Dataload, Dataflow, Publish) where level is temporarily promoted and then reset to original level
Change Level
Administrative rights are required to access and modify Level Control (governed by access mask) accessed in the Administration Panel.
If Lock level to only Managed is checked, Default level for Sources is disabled.
Level Control
If Lock level to only Managed is unchecked, Default level for Sources can be locked to only allow the selected level to be used. This lock and setting can be toggled.
Lock level to only Managed: If this setting has been unlocked to allow Registered and Addressed levels for entities, it cannot be relocked. The default setting for objects in QDC is Managed (full data ingest and management). Therefore if administrators do not change anything on this screen, application behavior will not change and all objects will continue to be managed. When Lock is checked, level is set for all sources/entities to Managed. This setting disables the ability to change level for Source Connections.
Default level for new sources (Top Level Managed Control is Un-checked):
- Managed: If selected (default), QDC will keep all entity data (on Hive and HDFS) and profiling and validation metadata gathered upon ingest. If unlocked, users with permissions can change level for individual objects in grid.
- Registered: If selected, QDC will keep only sample data along with the profile data and schema metadata but will delete all the other data from Hive and HDFS that may have imported as part of the data ingestion process. If unlocked, users with permissions can change level for individual objects in grid.
- Addressed: If selected, QDC will only gather schema metadata upon ingest or if the object is demoted from Managed or Registered, QDC will remove all the data from Hive and HDFS gathered on the data ingestion process, including sample data and profiling statistics. If unlocked, users with permissions can change level for individual objects in grid.
|
Level |
Options |
|---|---|
|
Lock level to only Managed |
Once unlocked, this action is not reversible (items cannot be relocked). When lock is on, all sources/entities are managed, and level cannot be modified:
|
|
Level |
Description |
|
Managed Full ingest, tightly coupled to all features/capabilities within Qlik Data Catalyst. Data load log type: Data |
Qlik Data Catalyst makes an internal copy of the external source data Sample data can be produced (user specified inheritable core_env.properties: default.record.sampling.probability) Good/Bad/Ugly partitioned record tables maintained in a table view (such as Hive) |
|
Registered Data has been added into data lake independent of Qlik Data Catalyst OR demoted from Managed. Data load log type: Statistics |
Qlik Data Catalyst will not retain a new copy of the data. When the object is ingested initially QDC creates sample data, generates profile statistics and schema metadata Sample data can be produced if sampling probability meets threshold (user selected option in core_env.properties: default.record.sampling.probability) Good/Bad/Ugly tables NOT populated/maintained in a Table view Can be used to execute Prepare and Publish jobs if sample data is produced On demotion from Managed, system removes the data and Table partitions On promotion from Addressed, system accesses the source, executes dataload to create sample data, generates profile statistics and schema metadata |
|
Addressed Source has been defined and analyzed outside of Qlik Data Catalyst OR a Managed/Registered object has been demoted. Data load log type: Metadata |
Schema metadata is acquired and maintained Fields metadata is updated when fields are added or deleted, or changes are made to data type, business name/description, index and properties they are updated in external and internal entities Can be used for creating Prepare dataflows and Publish jobs where they are temporarily promoted Information note
If a field name is changed, previous custom properties will not migrate On demotion to Addressed, only schema metadata is maintained: data in File System and Table views do not exist or are no longer maintained in Qlik Data Catalyst. Sample data and profiling statistics are also not retained upon demotion to Addressed |