Skip to main content

Recommended configuration

Description

Location

Configuration property

Default/Minimal value

Recommended value

Maximum number of database connections in the Quartz connection pool

Talend Administration Center configuration file

"WEB-INF/classes/quartz.properties" :

org.quartz.dataSource.QRTZ_DS.maxConnections

30

org.quartz.threadPool.threadCount + 3

Maximum number of concurrent Jobs handled by the Scheduler

Talend Administration Center configuration file

"WEB-INF/classes/quartz.properties" :

org.quartz.threadPool.threadCount

30

MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS

Maximum database connections for Talend Administration Center (apart from Quartz)

Talend Administration Center configuration file

"WEB-INF/classes/configuration.properties" :

hibernate.c3p0.max_size

32

MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS + MAX_CONCURRENT_LOGGED_USERS

Defines the period between each remote Job check

Talend Administration Center database table configuration

scheduler.conf.taskStatusRefreshTime

1

  • if MAX_CONCURRENT_TASK_EXECUTIONS < 20 : scheduler.conf.taskStatusRefreshTime = 1
  • if MAX_CONCURRENT_TASK_EXECUTIONS > 20 : scheduler.conf.taskStatusRefreshTime = MAX_CONCURRENT_TASK_EXECUTIONS /20

Defines the size of thread pool which checks the latest executions at startup

Talend Administration Center database table configuration

dashboard.conf.taskExecutionsHistory.threadPoolSize

10

( MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS ) / 25

Defines the size of thread pool which checks all the tasks at startup

Talend Administration Center database table configuration

scheduler.conf.simultaneousThreadsForStatusRefresh

5

MAX_CONCURRENT_TASK_EXECUTIONS / 50

Defines the number of maximum opened files for database process

Host of database server

Maximum opened files:

For example, under Linux set the Mysql configuration property "open_files_limit" and ensure that the system file limit is >= to the formula on the right

(depends on operating system)

( MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS + MAX_CONCURRENT_LOGGED_USERS ) x 3

Defines the number of maximum connections allowed to the database

Database server

Max connections:

For example, set the Mysql configuration property "max_connections = 10000"

(depends on database vendor)

(org.quartz.dataSource.QRTZ_DS.maxConnections + hibernate.c3p0.max_size) x 1.2

Defines the maximum number of concurrent connections accepted by the JobServer

JobServer configuration file

"conf/TalendJobServer.properties" :

org.talend.remote.server.MultiSocketServer.

MAX_CONCURRENT_CONNECTIONS

1000

MAX_CONCURRENT_JOBS_EXECUTIONS x 2

Definition of variables
Variable Description

MAX_CONCURRENT_JOBS_EXECUTIONS

Maximum expected number of concurrent executed Jobs on JobServer side

MAX_CONCURRENT_LOGGED_USERS

Maximum expected number of concurrent logged users (Talend Administration Center + Studio) on Talend Administration Center side

MAX_CONCURRENT_PLAN_EXECUTIONS

Maximum expected number of concurrent plan executions on Talend Administration Center side

MAX_CONCURRENT_TASK_EXECUTIONS

Maximum expected number of concurrent task executions on Talend Administration Center side

You might encounter performance issues if the properties are not properly configured.

The following properties have biggest influence on performance and memory consumption. By default they are:

  • hibernate.c3p0.max_size=32
  • org.quartz.threadPool.threadCount = 30
  • org.quartz.dataSource.QRTZ_DS.maxConnections = 30

Default values works if you have less than 10 tasks running at the same time.

If you have more tasks you can find optimal org.quartz.threadPool.threadCount using formula:

org.quartz.threadPool.threadCount = 3 x peak quantity of Jobs running at the same time

You do not need too many redundant threads because each thread consumes memory.

The number of connections depends on how fast Talend Administration Center is interacting with database server and also how many Jobs are running at the same time.

The total number of database connections = org.quartz.threadPool.threadCount + hibernate.c3p0.max_size

The total number of database connections cannot be more than what is configured on the server, for example in MySQL it is 150. The optimal number can also be calculated using formula:

org.quartz.dataSource.QRTZ_DS.maxConnections = 3 x peak quantity of Jobs running at the same time

hibernate.c3p0.max_size=3 x peak quantity of Jobs running at the same time

In some cases if the connection to database has no delay, you can even use the following formula:

org.quartz.dataSource.QRTZ_DS.maxConnections = 2 x peak quantity of Jobs running at the same time

hibernate.c3p0.max_size=2 x peak quantity of Jobs running at the same time

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!