Skip to main content Skip to complementary content

Changing the cleanup cycle of the task logs

Define how often the logs of Job task runs are cleaned up from the Talend Remote Engine machine.

These logs are stored in this directory: <RemoteEngineInstallationDirectory>/TalendJobServersFiles/jobexecutions/logs/

Note that only the data integration logs are stored here. Logs for Routes and data service tasks are only available in the Run overview in Talend Management Console.
Information noteTip: Starting with Talend Remote Engine v2.13.7, released in R2024-09, logs for task runs are no longer duplicated in the engine logs stored at <RemoteEngineInstallationDirectory>/data/log. To restore the duplication, see Restore the duplication of task run logs in the engine logs.

Procedure

  1. Stop the Talend Remote Engine if it is running.
  2. Open the <RemoteEngineInstallationDirectory>/etc/org.talend.ipaas.rt.jobserver.client.cfg file.
  3. Set these two parameters and adapt the values according to your needs of the cleanup frequency of these task run logs: 
    tasks.logs.clean.interval.days=1
tasks.logs.clean.older.than.days=30
  4. Save the file.
  5. Restart your Talend Remote Engine.

Results

These task logs are deleted regularly based on this configuration.
If you need to disable this cleanup feature, set the value of the tasks.logs.clean.interval.days parameter to be 0, that is to say,
tasks.logs.clean.interval.days=0

Restore the duplication of task run logs in the engine logs

To analyze task execution logs in the engine logs, configure the engine to duplicate the logs in the engine logs stored at <RemoteEngineInstallationDirectory>/data/log.

However, this approach can clutter the engine logs and cause premature archiving due to the log rotation policy.

Procedure

  1. Stop the engine if it is running.
  2. Open the <RemoteEngineInstallationDirectory>/opt/org.ops4j.pax.logging.cfg file.
  3. Find the log4j2.appender.rolling.filter.skipExecutionSpecificLogs.markers property in the # Rolling file appender section.
    This property should read like this:
    log4j2.appender.rolling.filter.skipExecutionSpecificLogs.markers = JOB_LOG,ESB_LOG,MANAGEMENT_LOG,JOB_LOG_FATAL,ESB_LOG_FATAL
    This property use markers to exclude execution logs from being written to the engine logs.
  4. Identify the markers for the logs you want to duplicate to engine logs and remove them from this property.
    For example, removing JOB_LOG and JOB_LOG_FATAL results in the duplication of data integration Job logs to the engine logs, while removing ESB_LOG and ESB_LOG_FATAL does the same for the Routes and data service Job logs.
  5. Save the changes.
  6. Restart your engine.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here