Either Built-In or Repository.
Built-In: No property data stored centrally.
Repository: Select the repository file where the properties are stored.
A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.
This component offers the advantage of the dynamic schema feature. This allows you to retrieve unknown columns from source files or to copy batches of columns from a source without mapping each column individually. For further information about dynamic schemas, see Dynamic schema.
This dynamic schema feature is designed for the purpose of retrieving unknown columns of a table and is recommended to be used for this purpose only; it is not recommended for the use of creating tables.
Built-In: You create and store the schema locally for this component only.
Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.
Insert: Add new entries to the table.
Update: Make changes to existing entries.
Insert or update: Insert a new record. If the record with the given reference already exists, an update would be made.
Update or insert: Update the record with the given reference. If the record does not exist in the index pool, a new record would be inserted.
Delete: Remove entries corresponding to the input flow.
This option allows you to call SQL functions to perform actions on columns, provided that these are not insert, update or delete actions, or actions that require pre-processing. This option is not available if you have just created the database table (even if you delete it beforehand). Click the [+] button under the table to add column(s), and set the following parameters for each column.
Name: Type in the name of the schema column to be altered or inserted.
SQL expression: Type in the SQL statement to be executed in order to alter or insert the data in the corresponding column.
Position: Select Before, Replace or After, depending on the action to be performed on the reference column.
Reference column: Type in a reference column that the current component can use to locate or replace the new column, or the column to be modified.
Key in update: Select the check box for the corresponding column based on which the data is updated.
Key in delete: Select the check box for the corresponding column based on which the data is deleted.
Updatable: Select the check box if the data in the corresponding column can be updated.
Insertable: Select the check box if the data in the corresponding column can be inserted.
Max total number of connections: enter the maximum number of connections (idle or active) that are allowed to stay open simultaneously.
The default number is 8. If you enter -1, you allow unlimited number of open connections at the same time.
Max waiting time (ms): enter the maximum amount of time at the end of which the response to a demand for using a connection should be returned by the connection pool. By default, it is -1, that is to say, infinite.
Min number of idle connections: enter the minimum number of idle connections (connections not used) maintained in the connection pool.
Max number of idle connections: enter the maximum number of idle connections (connections not used) maintained in the connection pool.
Time between two eviction runs: enter the time interval (in milliseconds) at the end of which the component checks the status of the connections and destroys the idle ones.
Min idle time for a connection to be eligible to eviction: enter the time interval (in milliseconds) at the end of which the idle connections are destroyed.
Soft min idle time for a connection to be eligible to eviction: this parameter works the same way as Min idle time for a connection to be eligible to eviction but it keeps the minimum number of idle connections, the number you define in the Min number of idle connections field.
When using Google Dataproc, specify a bucket in the Google Storage staging bucket field in the Spark configuration tab.
When using HDInsight, specify the blob to be used for Job deployment in the Windows Azure Storage configuration area in the Spark configuration tab.
When using on-premises distributions, use the configuration component corresponding to the file system your cluster is using. Typically, this system is HDFS and so use tHDFSConfiguration.
Standalone mode: use the configuration component corresponding to the file system your cluster is using, such as tHDFSConfiguration Apache Spark Batch or tS3Configuration Apache Spark Batch.
If you are using Databricks without any configuration component present in your Job, your business data is written directly in DBFS (Databricks Filesystem).