Navigating source module

Source module provides external source properties and ingest specifications for building metadata environments and onboard of data.

Available object grids in descending hierarchical order: source hierarchy | sources | entities | fields provide information and configurable properties appropriate to that level.

Select the desired object grid and navigate to or hover over object rows of interest.

To access the level below the current grid (for example if in source drill-down to entity level) select the view (eye) icon. For details about the various object levels and search and filter options refer to discover: navigation.

To access Details relevant for the object by object-level select the edit (pencil) icon. Note that disabled greyed-out fields cannot be edited.

For Source Information details: Source Hierarchy can be edited via dropdown. Other editable fields include Name, Business Name, Business Description, Tags.

View Source Connection details by expanding the dropdown to the left of the source connection name.

Selection of Add More Entities will transition users to second step of the wizard where source specifications are made.

To view actions available for the current object, select the More dropdown to access information and configurable attributes and properties: Load Logs, Delete (source), Discover (internal source), View/Edit General Info (same modal as Details), and View/Edit Properties.

Source module: Source hierarchy

Source hierarchy is the highest parent object level. Sources, entities, and fields are nested within source hierarchies.

Source hierarchies can also be nested within other source hierarchies by editing (pencil icon) and selecting a parent hierarchy.

To create a new Source Hierarchy:

1. Select Source Hierarchy; source hierarchies display
2. Select + Create Hierarchy
3. Enter Name for new source hierarchy
4. Save

Source module: Source

External Source: view/edit properties

Add, edit or delete properties. Remember to Save before exiting modal.

Source module: Entities

From the entity grid, select the view (eye) icon to drill-in to fields.

Select Edit (pencil) icon) to display and modify external entity properties. Note that while some properties are shared between source and discover modules, the source module contains dataload-specific attributes and structural properties that are not included in discover. For example, the General Information tab includes the following attributes: Entity Base Type and Source Connection. The default source-level connection can be overwritten for each entity in this screen.

To view actions available for an entity in the source module, select the More dropdown to access information and configurable attributes and properties:

Load (data), Load Logs (details every data load), Delete (entity), Discover (switch to internal source view), Source Connection (to change connection), View/Edit General Info (same modal as details accessed by selecting edit [pencil]), and View/Edit Properties.

Source (External) Entity: View/Edit Properties Select the plus button (add property) to open a drop-down with optional entity properties.

Add, edit, or delete properties. Remember to Save before exiting modal.

Source module: Fields

Fields grid in the source module provides a hotlink to the field object in the discover module where users can view metadata and sample data for that field.

View details (Pencil icon) to open field information with five tabs: General Information, Properties, Lineage, Tags, Blog.

Select Edit (pencil icon) to display/modify external field properties. Note that while some properties are shared between source and discover modules, the source module contains dataload-specific attributes and structural properties that are not included in discover.

Field: General information tab

Source module field attributes display on this tab. This information (metadata) describes the external source of data which is parsed into the application data model (in Postgres).

Enable data type change before ingest or upon reload (JDBC and flat files only)

Change the data type in the source module on field general information screen either before the Hive object has been created or after deleting the Hive object before entity is reloaded. This external field data type change will be propagated upon load or reload of the data to internal field data type.

• Schema of the Hive tables will not change by default if data type change is made after the first ingest:

• If data type change is being made after first ingest, add (from external entity property dropdown) and apply new property:

  'entity.hive.alterTableToCorrectType=true'

• Update or remove the following internal field properties:

  • 'field.hive.ddl.data.type=<datatype>'
  • 'numeric.precision.scale.rounding.mode=[x,y]'
• To ensure a matching data type is found, it is best to use standard names like STRING, DOUBLE, DECIMAL, BOOLEAN, INTEGER [a few non-standard names will work such as INT4, FLOAT, TEXT] Regardless of whether a mapping internal data type is found, a warning will display in the UI when the data type is changed
• Prepare dataflows with pre-existing field data types will fail validation and need to be fixed manually
• Predefined publish jobs will not be impacted by these changes if publishing to a Hadoop target
• Hive will not read sample data if the datatype format is not supported by Hive. Sample data displays with HDFS method for that data that has been processed with supported data types.

External field: Properties

Select the plus button (add property) to open a drop-down with optional entity properties.

Add, edit or delete properties. Remember to Save before exiting modal.

External field: Lineage

Parent lineage shows root source of field data. (In discover, the internal objects list the external parent source)

In source, this tab will display the child (internal) objects created from this source (external).

External field: Tags

Tag the objects with metatags to assist in locating and organizing data.

External field: Comments

Description of Field content, formulas, derivation, analyst notes. Blogs are subject to collaborative review and can be saved as Draft or Approved.

External and internal field metadata are different, comments created from the source grid do not display in discover and vice versa.

External fields: More actions

To view actions available for a field in the source module, select the More dropdown to access the same information available in the field information as tabs: View/Edit General Information, View/Edit Properties, View Lineage, View/Edit Tags, View/Edit Comments.