# R2022-11-RT (monthly release cumulative patch) | Info | Value | | --- | --- | | Patch Name | Patch\_20221219\_R2022-11\_v1-RT-7.3.1.R2022-09-RT | | Release Date | 2022-12-19 | | Target Version | 20221005\_0949-7.3.1.R2022-09-RT | | Product affected | Talend ESB Runtime | ## Introduction This patch is cumulative. It includes the previous generally available patches from Talend ESB Runtime 7.3.1.R2022-09-RT. **NOTE**: To download this patch, liaise with your Support contact at Talend. ## Fixed issues This patch contains the following fixes: ### TESB * TPRUN-4706: Integrate jobserver 7.3.1.20221206\_1150\_patch * TPRUN-4971: [7.3.1] CVE-2022-30126,org.apache.tika:tika-core:1.27 - update to tika 1.28.4 * TPRUN-4561: CVE-2022-42889, org.apache.commons:commons-text:[1.4-1.9] * TPRUN-4972: [7.3.1] Prevent runtime patches > R2022-07 from installing on default install * TPRUN-4976: [7.3.1] Update release notes with gen1/runtime common update reco * TPRUN-4290: CVE-2022-34169: Xalan 2.7.2 is removed * TPRUN-4514: CVE-2022-42003,CVE-2022-42004, jackson-databind-2.13.2.2.jar * TPRUN-4414: CVE-2022-40149: jettison upgrade to 1.5.1 * TPRUN-4695: Make access port configurable in tesb-derby-starter * TPRUN-4871: [CVE-2022-31692] Spring-security update to 2.6.9. * TPRUN-4497: Fail to execute "feature:install camel-spring-redis" on Runtime ### Job Server * TPRUN-3405: The FileListener does not jail the path to the jobserver deploy directory. * TPRUN-1296: Backport 'Prevent path manipulation attack in the FileServer' to 7.3. * TPRUN-3450: JobServer should not weaken TLS in the TACClient (backport to 7.3) * TPRUN-3451: CommandServer Denial of Service vulnerability (backport to 7.3) * TPRUN-3508: AuthorizationKey is logged * TPRUN-3697: JobServer should close stream of temporary context. * TPRUN-3604: Unzipper Incorrect size limit check and created files not deleted in case of error * TPRUN-3777: Non thread safe ClasspathJar writing * TPRUN-3679: Modularize function required for user impersonation. * TPS-5285: [7.3.1] Code cleanup & deprecation of 'launchFromShellScript' (TPRUN-3775) * TPRUN-3605: Unzipper add limits for nesting and path length. * TPRUN-3784: Update JobServer configuration/docs related to TLS version * TPRUN-3948: Align versions of JAVA source/target, dependencies and plugins on pom(s).xml * TPS-5359: [7.3.1] JobServer File server has no authentication. (TPRUN-3518) * TPRUN-4022: Update patch creation process * TPRUN-3916: Use RockyLinux as base image for JobServer docker in tests * TPRUN-4131: Check Zip Slip and Zip Symlink vulnerabilities * TPRUN-4126: Upgrade to OSHI 6.2.2 * TPRUN-3836: Improve error message in case Job archive checks fail * TPRUN-3523: Add ability to disable the monitoring service * TPRUN-1740: Simplify approach to let users install patches and (windows) services * TPRUN-4023: Reduce merging pain between active branches due to different logging framework * TPRUN-4267: Folder name length check not working for ZIP without folder entries * TPRUN-4238: Attempt to publish a large job (while FileServer authentication is available?) causes a command server timeout * TPRUN-4400: JobServer client checkServer returns wrong compatibility info * TPRUN-4255: Do not log warnings when properties are not set but default value exists * TPRUN-4355: Ensure Copyright is up-to-date for JAVA classes with UnitTesting * TPRUN-4269: After Unzipper Exception partially unzipped file remain * TPRUN-3519: Add constraints on jobs to prevent DoS attacks * TPS-5372: [7.3.1] Adding File path traversal guard (TPRUN-4050) * TPRUN-4515: Delete deployedJobPath directory before re-deploying * TPRUN-4486: JobServer - Cleanings * TPRUN-4447: JobServer start\_jconsole.bat script has wrong classpath * TPRUN-4761: Issue with FileEventsPacket * TPRUN-4048: Review Merge compulsory requirements * TPRUN-4005: Reading issue due to improper locking of job resuming log * TPRUN-3520: Check job archive signature * TPRUN-4753: Job archives that do not have a signature can be executed * TPS-5388: [7.3.1] Reading issue due to improper locking of job resuming log ( TPRUN-4005 ) * TPRUN-4523: Update osgi.cmpn to 5.0.0+ and org.osgi.core to 6.0.0+ * TPRUN-4892: parallel send protection error with tac and virtual servers * TPRUN-4898: JobServer checks cause problems for TAC deployments ### TDM * TDM-9289 Remove ExecutionProperties from the ExecutionStatus * TDM-9278 [OldRuntime]Execution status is accumulated when there are multiple executions for a tHMap * TDM-9226 Null item in JSON array is omitted on output * TDM-9178 CVE: org.hibernate:hibernate-core:[5.0.9-5.3.20.Final] * TDM-9033 Add representation options to reduce size of JSON output * TDM-9029 NullPointerException on Show Document for JSON not matching data * TDM-9018 tuj can't stopped (job tdm*DIColumns*SingleColumn\_ParallelizedJob can't be stopped) * TDM-8946 Add capability to put and get values in a hashmap saved in the Runtime ExecutionProperties * TDM-8927 One xml structure show as csv get error * TDM-8903 Expression with combination of 0-scale Decimal and Trim input option fails * TDM-8951 Restarting ESB Runtime produces 'Resource is not open' error in log * TDM-8851 Option to wrap the output to the array even if there is a single object * TDM-8683 Update XStream version used by TDM * TDM-8856 Remove conflicting bundle mvn:org.talend.transform/org.apache.xml.resolver * TDM-8843 EDI ISA16 should be used for component repetition, but Talend Studio is using the default of \ instead and not picking up the mapped ':' * TDM-8810 cMAP - Output is lost if cMap is terminal * TDM-8761 Eclipse runtime:route of main project use map refer reference project's customer bean throw warning * TDM-8694 Message with single quote messes the XQuery * TDM-8681 Security: Upgrade Commons Collections * TDM-8682 Security: Hibernate dependency * TDM-8660 EDI Reader not reporting wrong element on certain errors * TDM-8659 tHMapRecord job run fail use spark 2.3 on 741 which created and works on 721 * TDM-8648 [tHMap]HL7V2 Warnings are not shown in the Run Log when an HL7v2 transformation is used * TDM-8635 Remove dependency on DQ lib 6.0.1 * TDM-8603 Issue with upgrade to Studio 7.3.1 * TDM-8599 Replace avro-based configuration with regular JSON * TDM-8580 Job with multiple tRunJob fail with NoClassDefFoundError * TDM-8574 The specified value cannot be converted to the specified type * TDM-8571 Can't connect to mysql db with JDK11 * TDM-8524 [internal] Prepare runtime for native compilation and GraalVM * TDM-8516 Hikari DataSource and associated pool are not closed when route is stopped * TDM-8484 Json with Map Group,structure can't show as csv * TDM-8482 JSON Writer produces wrong XML Attributes * TDM-8446 Facing memory issues with a job using TDM after migrating to 7.1 * TDM-8415 Support Map Group as root when writing Avro datum * TDM-8409 tHMap with payload output of HL7V2 representation has an NPE execution error * TDM-8391 JSON: problem to write array of map * TDM-8364 TDM IO WriteURL broken * TDM-8363 Map isn't working after "R2020-09" patch installation (Error: "Input to cast cannot be atomized") * TDM-8359 Warning about overflow is incorrect for negative Cobol numbers * TDM-8327 NumberFormatException when running an imported project with a Map rep on the output map element * TDM-8326 Cobol Reader stops on 0xFF values with Variable Blocked format * TDM-8323 show document for json/xml structure with UTF-8 BOM encoding will return error * TDM-8318 Cobol Reader should silently truncate records with VB option * TDM-8308 Implicit Decimal Not In Output * TDM-8307 High memory usage by TDMEndpoint class in Runtime * TDM-8293 highlight is not right when show document for json with null element or invisible group * TDM-8225 cMap throws classcastException and not able to map a property from java bean * TDM-8217 Warning should not be issued for BTS and FTS segments * TDM-8210 Unable to MAP HL7 with CSV * TDM-8198 Export more packages in org.talend.transform.saxonpe.osgi * TDM-8163 Add new Function FormatDateTime * TDM-8125 DatabaseLookup creating new DataSources for each message on the ESB * TDM-8106 Remove dependency on org.codehaus.jackson in JSON io module * TDM-8094 Databaselookup fails on new runtime unless it is a top-level expression * TDM-8092 XML Reader should honor encoding set in the XML Representation * TDM-8089 Problem with camel headers when cJMS and cMap are used * TDM-8084 [7.3.1] Using thmap is getting an error when using a map with X12*5050*HIPPA structure * TDM-8074 Field alignment in positional flat file structures * TDM-7969 TDM adds unencrypted passwords to error message * TDM-7908 ReadNested within CSV or HashMap Representation fails * TDM-7789 CSV reader should use the optimization done for the CSV writer * TDM-7781 Result is incorrect when map attributes from xml to flat * TDM-7780 Result is incorrect when map attributes from xml to json * TDM-7427 data type optional segment is in test run result * TDM-6896 Upgrade Saxon library to 9.9 * TDM-6619 Mapper bundles in state 'Failure' after deployment * TPS-4793 [7.3.1] cMAP - Output is lost if cMap is terminal (TDM-8810) ## Prerequisites Consider the following requirements for your system: * Talend ESB Runtime 7.3.1.R2022-09-RT must be installed. * Depending on the product, `{container}` is `Talend-ESB-V7.3.1.R2022-09-RT/container/` or `Talend-Runtime-V7.3.1.R2022-09-RT/` * Before applying the patch, and **if old TDM patches have been installed** (ie: `org.talend.transform.runtime.distrib-X.Y.Z_yyyyMMdd_HHmm.zip`), please check the repository **files** are actually available on system, using this command: ``` karaf@trun()> feature:version-list talend-data-mapper | grep file Version | Repository | Repository URL --------------------+------------+--------------------------------------------------------------------------------------------------------- 7.3.1.R2022-09-RT.20200413_0622 | | file:/opt/TALEND/org.talend.transform.runtime.distrib-7.3.1.R2022-09-RT_20200413_0651/features.talend-esb.xml 7.3.1.R2022-09-RT.20200528_1359 | | file:/opt/TALEND/org.talend.transform.runtime.distrib-7.3.1.R2022-09-RT_20200528_1415/features.talend-esb.xml ``` Here for instance, check these files are available: ``` /opt/TALEND/org.talend.transform.runtime.distrib-7.3.1.R2022-09-RT_20200413_0651/features.talend-esb.xml /opt/TALEND/org.talend.transform.runtime.distrib-7.3.1.R2022-09-RT_20200528_1415/features.talend-esb.xml ``` If not, make sure to re-extract the old TDM patches to make these files available at the above locations **After successful** execution of the current patch, these files can be removed * Before applying the patch, and **if TAC is used**, latest TAC patch should be installed * Before applying the patch, please change the following properties in file `{container}/etc/org.apache.karaf.jaas.cfg` ``` encryption.enabled = true encryption.name = basic (or jasypt) ``` For all **inserted** properties: * if property already present (commented or uncommented), won't insert * if property not already present, will backup related file in dir `{container}/patches/Patch_20221219_R2022-11_v1-RT-7.3.1.R2022-09-RT/backup/` and insert property For all **updated** properties: * if property commented or not already present, won't update * if property already present, will backup related file in dir `{container}/patches/Patch_20221219_R2022-11_v1-RT-7.3.1.R2022-09-RT/backup/` and update property If any change required, update value after patch execution. * Patch will **insert** these properties in `{container}/etc/org.talend.remote.jobserver.server.cfg`: ``` # Set password of server side ssl key (command and file server) - optional org.talend.remote.server.ssl.keyPassword= # Set password of server side ssl key (monitoring server) - optional org.talend.jmxmp.ssl.keyPassword= # Set this to true to disable hostname verification for the TACClient - optional #org.talend.remote.jobserver.commons.config.JobServerConfiguration.TLS_DISABLE_CN_CHECK=true # Max size in bytes that an unzipped archive is allowed to be. The default is 1G. org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_UNZIPPED_SIZE=1073741824 # Max number of entries allowed in a zipped archive org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_ZIPPED_ENTRIES=2048 # Maximum length of zip file names: org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_ZIP_NAME_LENGTH=240 # Restrict the length of any folder name in paths inside the zip file: org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_UNZIPPED_FOLDER_NAME_LENGTH=240 # Restrict the length of any file name inside the zip file: org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_UNZIPPED_FILE_NAME_LENGTH=240 # Restrict the nesting levels of folders inside the zip file: org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_ZIP_DEPTH=64 # Enable the Monitoring port or not. true by default org.talend.remote.jobserver.server.TalendJobServer.ENABLE_MONITORING_PORT=true # Set to true to enable authorization for all job file deployments ( Requires additional configuration for TAC and Studio. ) org.talend.remote.jobserver.commons.config.JobServerConfiguration.FILESERVER_AUTHORIZATION=false # Maximum number of file listeners, 0 = No limit org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_FILE_LISTENERS=6000 # Maximum number of library dependencies embedded in a job org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_LIBRARY_DESC_NB=1000 # Maximum size of all library dependency names for a job org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_LIBRARY_DESC_SIZE=100KB # Maximum number of deployed jobs, 0 = No Limit org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_JOB_NB=6000 # Max size that a job archive is allowed to be. The default is 1G, 0 = No limit org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_JOB_FILE_SIZE=1G # Maximum size of TalendJobServersFiles/archiveJobs folder, 0 = No limit, 0 = No Limit org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_ARCHIVES_DIR_SIZE=100G # Activate job archive signature, 1 or more values separated by comma (','). # Possible values are: # - 'ON_DEPLOY' (legacy & default if no correct value provided) # - 'ON_UPLOAD' (advised) org.talend.remote.jobserver.commons.config.JobServerConfiguration.JOB_ARCHIVE_SIGNATURE_CHECK=ON_DEPLOY ``` * TPS-4318: JobServer memory leak related to ZeroMQ mailbox (TPSVC-12728) requires configuration in `{container}/etc/org.talend.remote.jobserver.server.cfg`: ``` org.talend.remote.jobserver.server.TalendJobServer.ENABLED_PROCESS_MESSAGE=false ``` * TPRUN-1846: feature `tesb-jmx-http-agent` based on jolokia has been removed due to security reasons. If jolokia is still needed, please manually use secured jolokia feature: ``` feature:install jolokia ``` Authorized users are declared in `{container}/etc/users.properties` * TPRUN-3009: default configuration in `{container}/etc/org.talend.esb.auxiliary.storage.service.cfg` for key `security.signature.properties` is: ``` security.signature.properties = file:${tesb.home}/etc/keystores/serviceKeystore.properties ``` if custom changes have been made, ensure the value references an absolute path. For instance, if expected keystore is `{container}/etc/customKeystore.properties`, this previous declaration: ``` security.signature.properties = customKeystore.properties ``` should be updated to: ``` security.signature.properties = file:${tesb.home}/etc/keystores/customKeystore.properties ``` * The patch replaces the files `{container}/bin/trun`, `{container}/bin/trun.bat`, `{container}/bin/setmem`, `{container}/bin/setmem.bat`, and `{container}/bin/inc`. If you have made previous changes to one of these files, you should move them to the file `{container}/bin/setenv` respectively `{container}/bin/setenv.bat`. These files are meant for customizations and will not be replaced during patch application. ## Installation ### Container * Start Runtime Container * Extract & replace the **content** of ZIP directory `container` **into** `{container}` directory Structure after extract & replace should be : ``` {container} ├───bin : existing dir ├───deploy : existing dir ├───etc : existing dir ├───... ├───patches : dir from current or previous patch │ └───Patch_20221219_R2022-11_v1-RT-7.3.1.R2022-09-RT │ patch.bat │ patch.commands │ patch.sh │ logs : directory for logs installation ├───system : existing dir │ ├───... : existing dir ├───... ``` * Ensure username/password are right in `{container}/patches/Patch_20221219_R2022-11_v1-RT-7.3.1.R2022-09-RT/patch.bat` or `{container}/patches/Patch_20221219_R2022-11_v1-RT-7.3.1.R2022-09-RT/patch.sh` ``` ... -u {username} -p {password} -f patch.commands ... ``` * Execute `{container}/patches/Patch_20221219_R2022-11_v1-RT-7.3.1.R2022-09-RT/patch.bat` or `{container}/patches/Patch_20221219_R2022-11_v1-RT-7.3.1.R2022-09-RT/patch.sh` * Ensure directory `{container}/patches/Patch_20221219_R2022-11_v1-RT-7.3.1.R2022-09-RT/logs` contains new log files : * `xxx-installation.log`: patch installation log * `xxx-init.log`: state before patch installation * `xxx-installed.log`: state after patch installation ``` Please note that Routes using cMap (TDM feature) are not automatically restarted by the patch procedure. You will need to restart the Runtime Container for changes to take effect. ``` Runtime patches may contain Java keystore updates. In this case, the previous keystores are preserved in the following places: * Keystores from `etc/keystores` are backed up in directory `{container}/patches/Patch_20221219_R2022-11_v1-RT-7.3.1.R2022-09-RT/backup/etc/keystores/`. * Example keystores are backed up in the directory where they are found with the suffix `-backup-TIMESTAMP`. ## Notes ### Enhancement of the SAP connector add-on The configuration of the "talend-sapjco3-connector" in version 5.5.1 allows to define additional SAP endpoints adding prefixed properties. Here is a sample for an endpoint named "PEER*CONNECTION*POOL": ``` jco.client.ashost = myfirsthost.example.org jco.client.sysnr = 00 jco.client.client = 800 jco.client.user = DEVUSRA jco.client.passwd = *** jco.client.lang = EN jco.destination.peak_limit = 10 jco.destination.pool_capacity = 3 endpoint.SAP_PEER_CONNECTION_POOL.jco.client.ashost = mysecondhost.example.org endpoint.SAP_PEER_CONNECTION_POOL.jco.client.sysnr = 00 endpoint.SAP_PEER_CONNECTION_POOL.jco.client.client = 100 endpoint.SAP_PEER_CONNECTION_POOL.jco.client.user = DEVUSRB endpoint.SAP_PEER_CONNECTION_POOL.jco.client.passwd = *** endpoint.SAP_PEER_CONNECTION_POOL.jco.client.lang = EN endpoint.SAP_PEER_CONNECTION_POOL.jco.destination.peak_limit = 10 endpoint.SAP_PEER_CONNECTION_POOL.jco.destination.pool_capacity = 3 ``` ### Default AlgorithmSuite from Basic128Sha256 to Basic256Sha256 (TPRUN-2631) All AlgorithmSuites of policies with SAML, are updated from Basic128Sha256 to Basic256Sha256 for these features: * `talend-job-controller` * `tesb-locator-soap-service` * `tesb-sam-service-soap` Configuration can be checked on these files, having value set to `SAML`: | Configuration file | Configuration key/value with SAML | Impacted endpoint | | --- | --- | --- | | etc/org.talend.esb.locator.service.cfg | locator.authentication = SAML | http://localhost:8040/services/ServiceLocatorService | | etc/org.talend.esb.sam.service.soap.cfg | sam.service.soap.authentication = SAML | http://localhost:8040/services/MonitoringServiceSOAP | If services are configured to use SAML: * you need to ensure external clients (executing out of container) use an updated policy when reaching these endpoints * you need to manually redeploy artifacts generated from Studio for models exposing/consuming endpoints using `Service Locator` or `Service Activity Monitoring` ### Default Algorithm for password encryption/decryption (TPRUN-2601) Algorithm encryption for all `ENC(xxx)` passwords is upgraded by default to `PBEWITHSHA256AND256BITAES-CBC-BC`. All passwords declared as `ENC(xxx)` in configuration files or Talend Administration Center must be regenerated through these commands in Runtime console (please ensure environment variable `TESB_ENV_PASSWORD` is set): ``` karaf@trun()> feature:install tesb-encryptor-command karaf@trun()> tesb:encrypt-text {textToEncrypt} ``` Algorithm can be configured by setting environment variable `TESB_ENV_ALGORITHM`. If old `ENC(xxx)` values are still needed, update the algorithm to previous one by setting environment variable `TESB_ENV_ALGORITHM` to `PBEWITHSHA256AND128BITAES-CBC-BC` and restart Runtime.