Skip to main content Skip to complementary content

Setting up monitoring and auditing systems for Talend JobServer

Configure an audit logging system to record key operations performed on Talend JobServer.

Additionally, when using your JobServer in a development environment, you can configure a transfer system for stats and trace messages.

Configuring audit logs for Talend JobServer

Starting from Talend JobServer version 8.0.2.20250110_1323_patch (zip name: Patch_20250114_TPS-6002_v4-8.0.1), you can activate audit logging to track operations triggered by client applications such as Talend Studio or Talend Administration Center.

This allows for monitoring and auditing of critical activities. Audit logs record the following operations:
  • Receiving a Job
  • Deploying a Job
  • Running a Job
  • Killing a running Job execution
  • Verifying if a user is authorized for an operation
  • Getting the useful CRC (Cyclic Redundancy Check) in case of errors before deployment

About this task

A logger for audit logs has been pre-configured in the log4j2.xml file:
<Logger name="org.talend.audit" level="info" additivity="false">
   <AppenderRef ref="Audit"/>
</Logger>
This procedure shows:
  • How to activate or deactivate audit logs.
  • How to update the audit appender used by this logger to specify the storage of audit logs.

Procedure

  1. Activate or deactivate the audit log using the org.talend.remote.jobserver.commons.config.JobServerConfiguration.AUDIT_LOG_LEVEL property in the <jobserver_path>/conf/TalendJobServer.properties file.
    The values to be used are:
    • 0 to deactivate audit logs
    • 1 to activate audit logs
  2. Configure whether audit logs are written to a dedicated file or included with other logs.
    • Recommended: to write audit logs in a separate file with specific file pattern, in the log4j2.xml file, update the fileName and the filePattern attributes in the RollingFile appender called Audit. For example: 
      <RollingFile name="Audit" fileName="jobserver_audit.log"
   filePattern="jobserver_audit-%d{MM-dd-yyyy}.log.gz" ignoreExceptions="false">
      <!-- Default log layout skipped for brevity -->
</RollingFile>
      With this configuration, Talend JobServer writes audit logs to the specified file jobserver_audit.log.
    • To include audit logs with other logs in jobserver.log, keep the default configuration of the Audit appender. The default configuration reads as follows:
      <RollingFile name="A1" fileName="jobserver.log" filePattern="jobserver-%d{MM-dd-yyyy}.log.gz"
  ignoreExceptions="false">
    <!-- skipped for brevity -->
</RollingFile>

<RollingFile name="Audit" fileName="jobserver.log" filePattern="jobserver-%d{MM-dd-yyyy}.log.gz"
 ignoreExceptions="false">
   <PatternLayout pattern="%d %-5p %c{1} 
   - %m%n %X{userId,jobId,utcTime,jobServerId,clientIP,projectId,failed}%n"/>
   <ThresholdFilter level="INFO" onMatch="ACCEPT" onMismatch="DENY"/>
   <Policies>
      <SizeBasedTriggeringPolicy size="1000KB"/>
   </Policies>
   <DefaultRolloverStrategy max="100"/>
</RollingFile>

Results

By default, audit logs use the MDC format: %X{userId,jobId,utcTime,jobServerId,clientIP,projectId,failed}.

You can retrieve the MDC fields to create a custom log layout such as JSON with named fields.

Configuring stats and trace message transfer for development environment

You can specify a port through which Talend Studio fetches the latest stats and trace messages from Talend JobServer for Jobs being executed remotely.

About this task

The stats and trace messages feature is for development environments.

Turn off the transfer of stats and trace messages in production environments. This prevents excessive memory usage and helps protect against external connections if the network security is inadequate.

Using real-time statistics may result in excessive memory usage on JobServer side. If you encounter ZeroMQ related memory leaks, turn off stats and trace messages on JobServer, although this will make real-time statistics unavailable in Talend Administration Center.

Procedure

  1. Go to the directory <root>/conf/, where <root> is the Talend JobServer path, and open the TalendJobServer.properties file to edit it.
  2. To specify the port for message transfer, set: 
    org.talend.remote.jobserver.server.TalendJobServer.PROCESS_MESSAGE_PORT=<port_number>

    The default port is 8555, but you can use any available port.

  3. To turn on stats and trace message transfer, set the following parameter to true. 
    org.talend.remote.jobserver.server.TalendJobServer.ENABLED_PROCESS_MESSAGE=true
  4. Save your changes and restart the Talend JobServer so that the configuration takes effect.

Results

Turn on the statistics feature in Talend Studio and Talend Administration Center when working with the transfer system: As mentioned above, turn on statistics only in development environments for performance and security reasons. In production environments, turn off the statistics feature in both Talend Studio and Talend Administration Center.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here