File target overview
The File target endpoint generates the data files either in delimited text file format (e.g. CSV) or in JSON format (according to the format selected in the endpoint settings).
Delimited text files are used to store data in tabular format. Examples of delimited text file formats include the CSV (Comma Separated Values) and TSV (Tab Separated Values) formats. Some organizations may implement procedures that export data from a database to a delimited text file while others may simply prefer this format as a convenient way of storing tabular data.
In a delimited text file, each record in the table occupies a separate row. Delimiters are used to mark the beginning of a new row or the beginning of a new column. Virtually any character can be used as a delimiter, although a newline (\n) is often used to separate rows, and commas are commonly used to separate columns.
In JSON files, each record is represented by a single line.
So, for example, the following table:
| book_id | title | price | is_hardcover |
|---|---|---|---|
|
123 |
Angels |
6.99 |
false |
|
456 |
The Fallen |
6.49 |
true |
|
789 |
Rise Up |
7.23 |
true |
Will be represented as:
{ "book_id": 123, "title": "Angels", "price": 6.99, "is_hardcover": false }
{ "book_id": 456, "title": "Fallen", "price": 6.49, "is_hardcover": true }
{ "book_id": 789, "title": "Rise Up", "price": 7.23, "is_hardcover": true }
When using a File as a target in a Replicate task, both the Full Load and the CDC data are written to CSV or JSON files (depending on the endpoint settings). While the explanations in this topic relate to CSV files, the same is true for JSON files.
Full Load files are named using incremental counters e.g. LOAD00001.csv, LOAD 00002.csv, etc. whereas Apply Changes files are named using timestamps e.g. 20141029-1134010000.csv.
When Parallel Load is used, the naming convention for Full Load files is slightly different:
LOAD_$(SegmenteID)_$(IncreasingCounter)
Example:
LOAD_1_00000001 | LOAD_1_00000002 | LOAD_1_00000003 | LOAD_2_00000001 | LOAD_2_00000002
- The Apply Changes CSV files appear with a .tmp extension while they are in idle state. For more information on idle state, see Change Processing.
- When the Create metadata files in the target folder option is enabled, a corresponding metadata file is created using the same naming format, but with a .dfm extension.
For each source table, a folder is created under the specified target folder. All files - i.e. Full Load, Apply Changes, and Metadata (if enabled) - are written to the relevant folder, according to the settings defined in the File target’s General tab.
After a task completes, you can define another task with a File source endpoint that uses the generated CSV files.