Recommended configuration
|
Description |
Location |
Configuration property |
Default/Minimal value |
Recommended value |
|---|---|---|---|---|
|
Maximum number of database connections in the Quartz connection pool |
Talend Administration Center configuration file |
"WEB-INF/classes/quartz.properties" : org.quartz.dataSource.QRTZ_DS.maxConnections |
30 |
org.quartz.threadPool.threadCount + 3 |
|
Maximum number of concurrent Jobs handled by the Scheduler |
Talend Administration Center configuration file |
"WEB-INF/classes/quartz.properties" : org.quartz.threadPool.threadCount |
30 |
MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS |
|
Maximum database connections for Talend Administration Center (apart from Quartz) |
Talend Administration Center configuration file |
"WEB-INF/classes/configuration.properties" : hibernate.c3p0.max_size |
32 |
MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS + MAX_CONCURRENT_LOGGED_USERS |
|
Defines the period between each remote Job check |
Talend Administration Center database table configuration |
scheduler.conf.taskStatusRefreshTime |
1 |
|
|
Defines the size of thread pool which checks the latest executions at startup |
Talend Administration Center database table configuration |
dashboard.conf.taskExecutionsHistory.threadPoolSize |
10 |
( MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS ) / 25 |
|
Defines the size of thread pool which checks all the tasks at startup |
Talend Administration Center database table configuration |
scheduler.conf.simultaneousThreadsForStatusRefresh |
5 |
MAX_CONCURRENT_TASK_EXECUTIONS / 50 |
|
Defines the number of maximum opened files for database process |
Host of database server |
Maximum opened files: For example, under Linux set the Mysql configuration property "open_files_limit" and ensure that the system file limit is >= to the formula on the right |
(depends on operating system) |
( MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS + MAX_CONCURRENT_LOGGED_USERS ) x 3 |
|
Defines the number of maximum connections allowed to the database |
Database server |
Max connections: For example, set the Mysql configuration property "max_connections = 10000" |
(depends on database vendor) |
(org.quartz.dataSource.QRTZ_DS.maxConnections + hibernate.c3p0.max_size) x 1.2 |
|
Defines the maximum number of concurrent connections accepted by the JobServer |
JobServer configuration file |
"conf/TalendJobServer.properties" : org.talend.remote.server.MultiSocketServer. MAX_CONCURRENT_CONNECTIONS |
1000 |
MAX_CONCURRENT_JOBS_EXECUTIONS x 2 |
| Variable | Description |
|---|---|
|
MAX_CONCURRENT_JOBS_EXECUTIONS |
Maximum expected number of concurrent executed Jobs on JobServer side |
|
MAX_CONCURRENT_LOGGED_USERS |
Maximum expected number of concurrent logged users (Talend Administration Center + Studio) on Talend Administration Center side |
|
MAX_CONCURRENT_PLAN_EXECUTIONS |
Maximum expected number of concurrent plan executions on Talend Administration Center side |
|
MAX_CONCURRENT_TASK_EXECUTIONS |
Maximum expected number of concurrent task executions on Talend Administration Center side |
You might encounter performance issues if the properties are not properly configured.
The following properties have biggest influence on performance and memory consumption. By default they are:
- hibernate.c3p0.max_size=32
- org.quartz.threadPool.threadCount = 30
- org.quartz.dataSource.QRTZ_DS.maxConnections = 30
Default values works if you have less than 10 tasks running at the same time.
If you have more tasks you can find optimal org.quartz.threadPool.threadCount using formula:
org.quartz.threadPool.threadCount = 3 x peak quantity of Jobs running at the same time
You do not need too many redundant threads because each thread consumes memory.
The number of connections depends on how fast Talend Administration Center is interacting with database server and also how many Jobs are running at the same time.
The total number of database connections = org.quartz.threadPool.threadCount + hibernate.c3p0.max_size
The total number of database connections cannot be more than what is configured on the server, for example in MySQL it is 150. The optimal number can also be calculated using formula:
org.quartz.dataSource.QRTZ_DS.maxConnections = 3 x peak quantity of Jobs running at the same time
hibernate.c3p0.max_size=3 x peak quantity of Jobs running at the same time
In some cases if the connection to database has no delay, you can even use the following formula:
org.quartz.dataSource.QRTZ_DS.maxConnections = 2 x peak quantity of Jobs running at the same time
hibernate.c3p0.max_size=2 x peak quantity of Jobs running at the same time