Skip to main content Skip to complementary content

Enabling and configuring log rotation on the engine

Logs related to the pipelines that are executed on the Remote Engine Gen2 are written inside the Livy container /tmp/livy/jobs where the log rotation is disabled by default.

The following table presents the log rotation parameters
Parameters Values Description
LIVY_JOB_LOG_PURGE_ENABLE true or false false (default value): log rotation is disabled.

true : enables log rotation
LIVY_JOB_LOG_RETAIN_DAYS Integer, the default value is 30 When log rotation is enabled, logs older than XX (in days) are removed.
LIVY_JOB_LOG_RETAIN_DISK Integer, the default value is 1048576 When log rotation is enabled and the total disk space used by the logs folder is greater than XX (in Bytes) all the existing logs are removed.

Procedure

  1. Go to the Remote Engine Gen2 installation directory.
  2. Edit the following file used to start the engine:
    default/docker-compose.yml if you are using the engine in the AWS USA, AWS Europe, AWS Asia-Pacific or Azure regions.

    eap/docker-compose.yml if you are using the engine as part of the Early Adopter Program.

  3. Find the service named livy and add the appropriate parameters and values.

    Example

    To enable log rotation and remove logs older than 3 months with a clean trigger of 10 GB:
    livy:
	...
	environment:
		...
		LIVY_JOB_LOG_PURGE_ENABLE: "true"
		LIVY_JOB_LOG_RETAIN_DAYS: 90
		LIVY_JOB_LOG_RETAIN_DISK: 10485760
  4. Restart Livy by executing this command: 
    docker-compose up -d livy

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here