A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.
Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:
View schema: choose this option to view the schema only.
Change to built-in property: choose this option to change the schema to Built-in for local changes.
Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion.
If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.
Built-In: You create and store the schema locally for this component only.
Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.
Use Inline Content: enter the data that you want to generate.
Use context variable: enter the name of variable to be used to provide data. This variable must have been defined in the Context view of the current Job.
The syntax to call a variable is context.VariableName.
When using Google Dataproc, specify a bucket in the Google Storage staging bucket field in the Spark configuration tab.
When using HDInsight, specify the blob to be used for Job deployment in the Windows Azure Storage configuration area in the Spark configuration tab.
When using on-premises distributions, use the configuration component corresponding to the file system your cluster is using. Typically, this system is HDFS and so use tHDFSConfiguration.
Standalone mode: use the configuration component corresponding to the file system your cluster is using, such as tHDFSConfiguration Apache Spark Batch or tS3Configuration Apache Spark Batch.
If you are using Databricks without any configuration component present in your Job, your business data is written directly in DBFS (Databricks Filesystem).