Masking sensitive connection information in the task run logs in Talend Management Console
With Remote Engine v2.12.3 onwards, if displayed in the logs in Talend Management Console, the connection information is automatically masked. This information could be any of the connection parameters you have locked in Talend Management Console, such as the host, the port number or the username.
About this task
Procedure
Results
Masking sensitive connection information may increase CPU and memory consumption, when a lot of information is being masked while a large number of logs are held in memory.
To mitigate this spike, fine-tune the following parameters:
- Browse to the <RemoteEngineInstallationDirectory>/etc folder and open the org.talend.ipaas.rt.jobserver.client.cfg file.
- Adjust the values as follows:
The following table presents the parameters used to adjust the size of log rows and the number of stored log events Parameter Description job.log.msg.size.max Its default value is 10000 characters, meaning the maximum size allowed for each log row. By decreasing this value to limit the size of each log row to be handled, you truncate each log row to reduce the resource consumption.
If truncating logs is an issue for you, consider adding additional resources to the engine.
job.log.msg.buffer.size.max This parameter sets the maximum number of log events that can be temporarily stored in memory, with a default value of 10000. During high peaks of concurrent log events, this maximum size may be reached, resulting in the loss of log data. To prevent this loss, increase this buffer size and accordingly, allocate additional memory resources to your engine.
Information noteNote: To alleviate the load of your memory and CPU, you can also add a thread.timeout parameter to specify the maximum time that log processing can run before timing out. The default value is 5 minutes, at the end of which log processing is stopped and context information cleaned up. By decreasing this value, you release resources more frequently, but be aware that the logs that have not been processed are lost. If this is a concern, consider allocating more resources to your engine.When this timeout event occurs, it is logged as a WARN event. A log entry of this event could read like:Force cleanup context for empty streamSet, executionId {} - Each character requires 2 bytes of memory. This way, when calculating additional
resources required for log processing, the calculation can be done as follows:
- For 2000 characters on each log row and 2000 log events, 80MB of memory is required.
- For 20000 characters on each log row and 5000 log events, 200MB of memory is required.
- For 20000 characters on each log row and 20000 log events, 800MB of memory is required.
After increasing this allocation in the host machine, if your engine has been deployed as a system service, customize the JVM configuration in the <RemoteEngineInstallationDirectory>/etc/talend-remote-engine-wrapper.conf file accordingly.