tHiveConfiguration properties for Apache Spark Structured Streaming

Property Type

Select the way the connection details will be set.

• Built-In: The connection details will be set locally for this component. You need to specify the values for all related connection properties manually.

• Repository: The connection details stored centrally in Repository > Metadata will be reused by this component.

  You need to click the [...] button next to it and in the pop-up Repository Content dialog box, select the connection details to be reused, and all related connection properties will be automatically filled in.

Enter the location of the metastore of the Hive system to be used by specifying the name of its Host and the number of its listening Port. If HA metastore has been defined for this Hive system, select the Enable high availability check box and in the field that is displayed, enter the URIs of the multiple remote metastore services, each being separated with a comma(,).

If you are accessing a Hive metastore running with Kerberos security, select this checkbox.

Then you need to enter the Hive principal that should have been defined in the hive-site.xml file of the cluster to be used.

Hive principal uses the value of hive.metastore.kerberos.principal. This is the service principal of the Hive metastore.

Spark catalog

This component is used standalone. Add it to the same Job as the Hive-related subJob so that the configuration is available to the whole Job at runtime.

This component, along with the Spark Structured Streaming component Palette it belongs to, appears only when you are creating a Spark Structured Streaming Job.

In the Structured Streaming Configuration tab of the Run view, define the connection to the Spark cluster for the whole Job. Specify the directory to which the Job's dependent JAR files are transferred so Spark can access them.

This connection is effective on a per-Job basis.