Skip to main content Skip to complementary content

Understanding the Talend Remote Engine clean-up cycle

The Talend Remote Engine cleans up Job artifacts on a schedule, based on data cleaning parameters. You can customize the Talend Remote Engine clean-up schedule by changing the parameters in the org.talend.remote.jobserver.server.cfg file, listed under Temporary data cleaning parameters. Configuring these values is optional.

Information noteNote: Make sure each Remote Engine instance has its separate TalendJobServersFiles folder, as explained in Deploying multiple Remote Engines.

General clean-up frequency

The general clean-up schedule is defined by the FREQUENCY_CLEAN_ACTION parameter. You can disable the general clean-up by setting this parameter to 0.

Job repository and archive clean-up

Job artifacts are cleaned up in the next clean-up cycle when the following conditions are met:
  • The Job is not running.
  • Either the MAX_DURATION_BEFORE_CLEANING_OLD_JOBS or MAX_OLD_JOBS parameter is met.

Job execution log clean-up

Job logs are cleaned up in the next cycle when the following conditions are met:

  • Either the MAX_DURATION_BEFORE_CLEANING_OLD_EXECUTIONS_LOGS or MAX_OLD_EXECUTIONS_LOGS parameter is met.
  • The Job execution is released. A Job is released after 50 Job executions. You can change the Job release frequency by changing the MIN_NUMBER_JOB_EXECUTIONS_BEFORE_RELEASE parameter.

A log without errors is released if the elapsed time since the start of the log is greater than the time defined in the MAX_DURATION_BEFORE_JOB_EXECUTION_RELEASE_NORMAL_CASE and MAX_DURATION_BEFORE_JOB_EXECUTION_RELEASE_ABNORMAL_CASE parameters.

Information noteNote: For each MAX_OLD_* and MAX_DURATION_* parameter pair, whichever is reached first will trigger a cleaning action.

Talend Remote Engine clean-up parameters

The following tables lists the default values for each of the clean-up parameters.
Information noteNote: All parameters in this file are prefixed with org.talend.remote.jobserver.commons.config.JobServerConfiguration.
Parameter Default Description
FREQUENCY_CLEAN_ACTION 10 minutes Defines the time between each Talend Remote Engine cleaning action. Set this value to 0 to disable automatic cleaning.
MAX_OLD_JOBS 200 Defines the maximum number of artifacts and deployed Jobs to keep. Once this value is passed, artifacts are deleted beginning with the oldest. Set this value to 0 to disable this parameter.
MAX_DURATION_BEFORE_CLEANING_OLD_JOBS 3 months Defines the maximum before cleaning archives and deployed Jobs. Set this value to 0 to disable this parameter.
MAX_OLD_EXECUTIONS_LOGS 1000 Defines the maximum number of execution logs to keep. Once this value is passed, logs are deleted beginning with the oldest. Set this value to 0 to disable this parameter.
MAX_DURATION_BEFORE_CLEANING_OLD_EXECUTIONS_LOGS 3 months Defines the maximum time before cleaning execution logs. Set this value to 0 to disable this parameter.
MIN_NUMBER_JOB_EXECUTIONS_BEFORE_RELEASE 50 Defines the minimum number of Job executions before a Job is released. Job logs cannot be cleaned until the Job is released.
MAX_DURATION_BEFORE_JOB_EXECUTION_RELEASE_NORMAL_CASE 5 minutes Defines the maximum time for a normal execution before a Job is released.
MAX_DURATION_BEFORE_JOB_EXECUTION_RELEASE_ABNORMAL_CASE 24 hours Defines the maximum time before an abnormal execution before a Job is released. The abnormal duration applies to Job execution log files that contain errors.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here