Skip to main content

Message Format

The Message Format tab will only be shown if the task is defined with a streaming target endpoint that supports this feature.

When a task is defined with such an endpoint, you can specify a custom message format that will override the default Replicate message format. This may be useful if the consumer application needs to process the message in a particular format.

The custom message format can be defined at task level and/or at table level. When it is defined at both task and table level, the message format defined for the table will take precedence over the message format defined for the task.

For information on defining a custom message format at table level, see Message format.

Note:
  • Supported by the Kafka target endpoint only

  • The custom message format feature cannot be used in conjunction the Attunity Avro Message Decoder SDK

  • Hierarchical structured messages are not supported.

General rules and usage guidelines

When defining a custom message, it is import to consider the rules and usage guidelines listed below.

The following naming rules apply:

  • Section names must start with the characters a-z, A-Z, or _ (an underscore) and can then be followed by any of the following characters: a-z, A-Z, 0-9, _
  • With the exception of the Record name and Key name sections (that do not end with a slash), removing the slash from section names will flatten the hierarchy of the associated section (see Slashes below).
  • All section names except Record name and Key name can be deleted (see Deletion: below)
  • The Data name and Include record Before-data section names cannot both be deleted

  • The Data name and Include record Before-data section names cannot be the same

Some of the section names in the UI end with a slash (e.g. beforeData/). The purpose of the slash is to maintain a hierarchy of the different sections within the message. If the slash is removed, the following will occur:

  • The hierarchical structure of that section will be flattened, resulting in the section name being removed from the message
  • The section name will be prefixed to the actual metadata, either directly or using a separator character (e.g. an underscore) that you appended to the name

Example of a data message when headers/ is specified with a slash:

		"message":{
		        "data":{
			      "COL1": "159",
			      "COL2": "159"
                       },
                       "beforeData": null,
		        "headers": {
			"operation": "INSERT",
			"changeSequence": "2018100811484900000000233",

Example of a data message when headers_ is specified with an underscore instead of a slash:

		"message":{
		        "data":{
			      "COL1": "159",
			      "COL2": "159"
                       },
                       "beforeData": null,                     
                       "headers_operation": "INSERT",
			"headers_changeSequence": "2018100811484900000000233",

Deleting a section name from the message will flatten the hierarchical structure of that section. This will result in all of that section's data appearing immediately below the content of the preceding section.

Example of a data message with the headers section name:

	        "message":{
		        "data":{
			      "COL1": "159",
			      "COL2": "159"
                       },
		        "headers": {
			"operation": "INSERT",
			"changeSequence": "2018100811484900000000233",

Example of a data message without the headers section name:

	        "message":{
		        "data":{
			      "COL1": "159",
			      "COL2": "159"
                       },
                       "beforeData": null,                   
                       "operation": "INSERT",
			"changeSequence": "2018100811484900000000233",

You can add variables to names by clicking the button at the end of the row. The following variables are available:

  • SERVER_NAME - The host name of the Replicate Server
  • TARGET_TABLE_NAME - The name of the table
  • TARGET_TABLE_OWNER - The table owner
  • TASK_NAME - The name of the Replicate task
Note:

The TARGET_TABLE_OWNER variable is not available for the Record name and Key name options (described in the table below).

Defining a custom message format

To define a custom message format, click the Custom Message Format button and configure the options as described in the table below.

To revert to the default message format, click the Default Message Format button.

Message format options
Option Description

Include Namespace

A unique identifier for the message. This should be a string, separated by periods. Note that the Namespace will be included in both the message and the message key.

Example:

mycompany.queue.msg

Default name:

com.attunity.queue.msg.{{TASK_NAME}}.{{TARGET_TABLE_OWNER}}.{{TARGET_TABLE_NAME}}

Default name for Control Tables:

com.attunity.queue.msg.{{TARGET_TABLE_NAME}}

Note:

The default Control Table Namespace cannot be changed via the UI. You can, however, change the default Control Table Namespace as follows:

  1. In the Task Settings Message Format tab, click the Custom Message Format button and then click OK.
  2. Save the task and then export it using the Export Task toolbar button.
  3. Open the exported JSON file and add the control_table_namespace parameter as follows (replacing MyNameSpace with your own value):

    Example JSON file export with "control_table_namespace" field highlighted

  4. Save the JSON file and then import it to Replicate using the Import Task toolbar button.

For instructions on exporting/importing tasks, see Migrating tasks.

Record name

The name of the record (i.e. message).

DataRecord

Data name

All of the data columns included in the record.

data/

Include Headers

Header columns provide additional information about the source operations.

For more information, see Include the following headers below.

Include headers namespace

A unique identifier for the header columns section. This should be a string, separated by periods.

Example:

headers.queue.msg

com.attunity.queue.msg

Headers name

The name of the section containing the Replicate column headers.

headers/

Include the following headers

For a description of the available header columns, see Data Messages in Metadata and data messages.

All columns are included by default, except the External Schema ID column.

Include record Before-data

When this check box is selected (the default), both pre and post UPDATE data will be included in UPDATE messages.

To include only the post UPDATE data in messages, clear the check box.

beforeData/

Key name

The name of the section containing the message key.

keyRecord

This option is only applicable when:

  • The Encode the message key in Avro format option is enabled
  • The Metadata Publishing Method is set to Publish data schemas to Confluent Registry Schema or Publish data schemas to Hortonworks Schema Registry
  • The Message key is not set to None