Robust event processing

Deprecated

The primary focus of the Robust event processing is to make sure none of the events are lost or skipped in case the REST or the JMS brokers are down, or the JMS collector is not able to deliver events to the Event Logging server. As shown in the image below, these are the four critical paths where robust event processing is currently implemented.

To start the robust event processing feature, use the following command in the container:

tesb:start-el-dlq

This command will start a camel route which handles the robust re-delivery of messages to the Event Logging server. In case the REST receiver or the JMS broker are down, the REST and JMS senders will send the events to a DLQ (Dead letter Queue) configured in the org.talend.eventlogging.dlq.cfg file. The Robust processing camel route will monitor this DLQ.

Once the events arrive in DLQ, it will ping the JMS broker to check for its availability.

Once the broker is available, it will send the events from the DLQ to the JMS broker.

The configuration for this is as follows:

# JMS connection url
dlq.buffer.jms.url=vm://eventloggingbroker?create=true&broker.use
Jmx=false&broker.persistent=true
# Name of JMS queue
dlq.buffer.jms.queue=event.logging.dlq
# JMS Username
dlq.buffer.jms.username=tadmin
# JMS Password
dlq.buffer.jms.password=tadmin

dlq.timeout=10000

dlq.destination.jms.url=tcp://localhost:61616
dlq.destination.jms.queue=event.logging.server
dlq.destination.jms.username=tadmin
dlq.destination.jms.password=tadmin

As shown in the above image at Step 3, if the events arrive in the JMS broker but the JMS Collector is down, or for some reason not able to pick the events, the events will be stored in the JMS broker until the JMS collector picks them up.

In Step 4, if the JMS Collector is not able to send events to the Event Logging server, it will send the events to a DLQ configured in the org.talend.eventlogging.dlq.cfg file.

The events will be stored in this DLQ permenantly and will not be processed by the JMS Collector or any other component of the Event Logging, or will not be send to the Event Logging server later. It will be the responsibility of the Administrator to manually handle them.

The detailed configuration for the Robust event processing feature is as follows:

`# JMS connection url dlq.buffer.jms.url=vm://eventloggingbroker?create=tr ue&broker.useJmx=false&broker.persistent=true`	The address of the JMS broker which is used to store the events in the DLQ. The default configuration uses the embedded broker which is also used for internal event jms buffering at the agent side.
`# Name of JMS queue dlq.buffer.jms.queue=event.logging.dlq`	DLQ queue name
`# JMS Username dlq.buffer.jms.username=tadmin # JMS Password dlq.buffer.jms.password=tadmin`	DLQ queue Active MQ broker credentials
`dlq.timeout=60000`	Timeout in milliseconds in which the robust event processing feature will re-deliver events from DLQ queue to destination.
`dlq.destination.jms.url=tcp://localhost:61616`	Destination (Event Logging server) queue broker address
`dlq.destination.jms.queue=event.logging.server`	Destination (Event Logging server) queue name
`dlq.destination.jms.username=tadmin dlq.destination.jms.password=tadmin`	Destination (Event Logging server) broker credentials

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here