# Upgrading Remote Engine v2.11.8 onwards for Data Services and Routes tasks Upgrade your Remote Engine without the need to undeploy or redeploy any data service Job or Route tasks currently running on the engine. Microservice tasks are automatically halted with the first Remote Engine shutdown and resumed upon the startup of the new Remote Engine version. ## Before you begin Information noteImportant: If the Talend Remote Engine will be unavailable for at least 6 hours, consider undeploying Route or Data Service tasks before maintenance and redeploying them afterward to keep their status accurate. A periodic check marks a running Route or Data Service task as Failed in the Talend Management Console when the engine has been unavailable for at least 6 hours. After the engine is available again, the task runs normally, but its status remains Failed until the next check. To understand why this happens and plan your maintenance, see [Status of tasks after a prolonged engine outage](https://help.qlik.com/talend/en-US/remote-engine-user-guide-linux/Cloud/data-service-and-route-deployment-without-network-connection#status-after-prolonged-engine-outage). Before upgrading, check the support status of your target version in [Talend Remote Engine lifecycle](https://help.qlik.com/talend/en-US/customer-support-statements/Cloud/talend-remote-engine-lifecyle). * The Remote Engine to upgrade must be v2.11.8 and onwards. If you are using v2.11.7 or an earlier version, upgrade them to the latest version before starting this procedure as explained in [Upgrading Talend Remote Engine](https://help.qlik.com/talend/en-US/installation-guide-linux/Cloud/upgrading-remote-engine-intro "Upgrade your Talend Remote Engine to retrieve the latest features."). * Generate a service account token (SAT) to use a service account to perform this operation. For more information on how to generate this token, see [Generating a service account token](https://api.talend.com/use-cases/service-accounts/generating-service-account-token/). Using a service account is a widely accepted security practice for automation in a production environment. While using a personal access token (PAT) is technically possible, it is recommended to utilize a service account for enhanced security. For more information on how to generate a personal access token, see [Generating a Personal Access Token](https://help.qlik.com/talend/en-US/management-console-user-guide/Cloud/cloud-access-token). * Gather the IDs of the engines to be upgraded, using either the related API endpoint or the Talend Management Console web page: * API endpoint: [Get available Remote Engines](https://api.talend.com/apis/processing/2021-03/#operation_get-all-available-remote-engines). In the response, the engine IDs are found in the id field. Copy the IDs of the engines to be upgraded and keep them in a place of your choice for later use. * Talend Management Console web page: a Remote Engine ID is found in the Overview tab of the Engine details blade. ![Engine details dialog box](https://help.qlik.com/talend/en-US/installation-guide-linux/Resources/images/tmc-engine-id.png) If the engine in question is in a cluster, this URL reads like ``` https://app.eu.cloud.talend.com/tmc/processing/engines/5bf0452b853ce935a13ed9e6/cluster/646c7931d5380f70c1d217fc/assigned-engine/646c790a2373b74412c26047/details/info ``` In this case, the ID is 646c790a2373b74412c26047. If the engine in question is in a cluster, this URL reads like ``` https://app.eu.cloud.talend.com/tmc/processing/engines/5bf0452b853ce935a13ed9e6/cluster/646c7931d5380f70c1d217fc/assigned-engine/646c790a2373b74412c26047/details/info ``` `https://app.eu.cloud.talend.com/tmc/processing/engines/5bf0452b853ce935a13ed9e6/cluster/646c7931d5380f70c1d217fc/assigned-engine/646c790a2373b74412c26047/details/info` ``` https://app.eu.cloud.talend.com/tmc/processing/engines/5bf0452b853ce935a13ed9e6/cluster/646c7931d5380f70c1d217fc/assigned-engine/646c790a2373b74412c26047/details/info ``` `https://app.eu.cloud.talend.com/tmc/processing/engines/5bf0452b853ce935a13ed9e6/cluster/646c7931d5380f70c1d217fc/assigned-engine/646c790a2373b74412c26047/details/info``https://app.eu.cloud.talend.com/tmc/processing/engines/5bf0452b853ce935a13ed9e6/cluster/646c7931d5380f70c1d217fc/assigned-engine/646c790a2373b74412c26047/details/info`https://app.eu.cloud.talend.com/tmc/processing/engines/5bf0452b853ce935a13ed9e6/cluster/646c7931d5380f70c1d217fc/assigned-engine/646c790a2373b74412c26047/details/infoIn this case, the ID is 646c790a2373b74412c26047646c790a2373b74412c26047. ## Procedure 1. Use the specific API endpoint or the Talend Management Console web UI to unpair the Remote Engine to be upgraded from the designated engine logical entity on Talend Management Console. ### Example * API endpoint: ``` curl -X DELETE "$RE_APFIRST_ENDPOINT/processing/runtimes/remote-engines/$RE_ENGINE_ID/pairing?keepOsgiDeployments=true&keepMicroserviceDeployments=true" -H "Authorization: Bearer $RE_TOKEN" ``` In this command: * $RE\_APFIRST\_ENDPOINT represents the URL of Talend Management Console, for example, https://api.us.cloud.talend.com/. For the API endpoint URL of your region, see [Endpoint regions](https://api.talend.com/apis/processing/2021-03/#endpoints). * $RE\_ENGINE\_ID is for the ID of the engine to be unpaired. * $RE\_TOKEN is the service account token, which is recommended, or the personal access token of the account to be used. You can either replace these variables with their actual values in your command, or define them in the environment of your API requests. If you use Talend API Tester to issue your API endpoint, see [Creating an environment](https://help.qlik.com/talend/en-US/api-tester-user-guide/Cloud/creating-environment) to learn about how to define variables. When the command is executed, a new preauthorized key is automatically generated. Keep this key in a place of your choice, because you need to use it to pair the new Remote Engine in the following steps. * Talend Management Console web page: follow [this procedure](https://help.qlik.com/talend/en-US/remote-engine-user-guide-linux/Cloud/pairing-remote-engines-using-web-service) to find the Reset the connection section in the Engine details blade of the engine to be unpaired, and from that section, click Reset the key to unpair the engine and get a new pairing key. 2. Wait for at least 2 minutes to ensure that the engine is correctly unpaired. ### Example The following message from the engine logs is an example that indicates the unpairing has been correctly completed. ![Logs that indicate the engine state is retired with the line "re_state": "RETIRED".](https://help.qlik.com/talend/en-US/installation-guide-linux/Resources/images/re-unpair-successful-log.png) 3. On the host machine of the unpaired engine, stop the engine using the following command: ``` /bin/stop ``` If this engine has been installed as a system service, follow your OS procedure to shut down the service. 4. Install and configure the new Remote Engine version. * If you use the installer to install this new engine, do not use the actual preauthorized key yet when you are prompted to provide a key; provide a fake authorized key instead. * For more information about how to install a Remote Engine, see [Installing Talend Remote Engine](https://help.qlik.com/talend/en-US/installation-guide-linux/Cloud/installing-talend-remote-engine "Talend Remote Engines allow you to run tasks on premises."). * For more information about how to configure a Remote Engine and its runtime, see [Configuring Remote Engine](https://help.qlik.com/talend/en-US/remote-engine-user-guide-linux/Cloud/configuring-talend-remote-engine) and [Configuring the Data Service Runner](https://help.qlik.com/talend/en-US/remote-engine-user-guide-linux/Cloud/configuring-data-service-runner). * If you have installed the new engine as a system service, stop the service before proceeding to the following steps. 5. Copy data from the old engine to the new engine. This copy helps ensure the continuous running of your data service Job tasks or Route tasks. For this reason, copy each type of data to their designated folders as expected by Remote Engine. 1. Copy-paste the dsrunner folder from Old\_RemoteEngine\_installation/data/ to New\_RemoteEngine\_installation/data/. This folder contains the task information. 2. Copy-paste the cls folder from Old\_RemoteEngine\_installation/data/ to New\_RemoteEngine\_installation/data/. This folder contains the task logs. 3. On the new Remote Engine, update the path to the new engine to be used in every .metadata file that is stored in each of the task folders in New\_RemoteEngine\_installation/data/dsrunner/exec/osgi. ## Example ![Path to the osgi folder content.](https://help.qlik.com/talend/en-US/installation-guide-linux/Resources/images/re-task-metadata-file.png) ![Path to the osgi folder content.](https://help.qlik.com/talend/en-US/installation-guide-linux/Resources/images/re-task-metadata-file.png) For example, in this folder New\_RemoteEngine\_installation/data/dsrunner/osgi/02141bff-bffe-4975-8835-7dd9ff77b62a, open the .metadata file and find the following line, which is the path to the engine to be used: ``` "artifactFile":"Old_RemoteEngine_installation/data/dsrunner/exec/osgi/5f69d7c4-e3d8-4d41-b248-b94a20d7cbf0/SimpleRecipientList_OSGI_0.1.0.20231303094541.kar" ``` Then replace it with the path to the new Remote Engine: ``` "artifactFile":"New_RemoteEngine_installation/data/dsrunner/exec/osgi/5f69d7c4-e3d8-4d41-b248-b94a20d7cbf0/SimpleRecipientList_OSGI_0.1.0.20231303094541.kar" ``` If you are using Windows, these paths look like ``` RemoteEngine_installation\\data\\dsrunner\\exec\\osgi\\02141bff-bffe-4975-8835-7dd9ff77b62a\\ro_20230209_SimpleTimerRoute_801_OSGi_0.1.2.20232505013343.kar ``` 4. Still on the new Remote Engine, update the path to the log files for each execution. This information is found in every <TASKID>-<EXECUTION\_ID>-state.dat file that is stored in New\_RemoteEngine\_installation/data/cls. ## Example For example, open the New\_RemoteEngine\_installation/data/cls/640ef0c6a45e743a0b2580ca-5f69d7c4-e3d8-4d41-b248-b94a20d7cbf0-state.dat file and find the following line, which is the path you need to update: ``` "currentFilePath":"Old_RemoteEngine_installation/data/cls/640ef0c6a45e743a0b2580ca-5f69d7c4-e3d8-4d41-b248-b94a20d7cbf0-2023-05-10-14-001.json.log" ``` Then replace it with the new path on the new Remote Engine: ``` "currentFilePath":"New_RemoteEngine_installation/data/cls/640ef0c6a45e743a0b2580ca-5f69d7c4-e3d8-4d41-b248-b94a20d7cbf0-2023-05-10-14-001.json.log" ``` On Windows, this path look like ``` "currentFilePath":"RemoteEngine_installation\\data\\cls\\647867abadf89464d65ab1f2-02141bff-bffe-4975-8835-7dd9ff77b62a-2023-06-01-11-001.json.log ``` 6. Pair the new Remote Engine to the same Talend Management Console engine item that you previously unpaired from the old Remote Engine. 1. In the New\_RemoteEngine\_installation/etc/org.talend.ipaas.rt.pairing.client.cfg file, specify the endpoint for pairing of Talend Management Console in the following line: ``` pairing.service.url=https://pair..cloud.talend.com ``` For the exact endpoint URL to be used, see the Remote Engine pairing entry of your region in [Accessing Talend Cloud applications](https://help.qlik.com/talend/en-US/installation-guide-linux/Cloud/talend-cloud-application-url "The URLs to the Talend Cloud applications differ from a data center to another."). If you have installed the new engine using the installer, you can skip this step because the endpoint should have been automatically specified. 2. In New\_RemoteEngine\_installation/etc/preauthorized.key.cfg, replace the fake preauthorized key with the actual one generated by the unpairing call at the beginning of the procedure. If you have lost this preauthorized key, find it back in the Remote Engine key field of the engine item to be paired on Talend Management Console. For further information about how to find this key, see [Creating a Remote Engine](https://help.qlik.com/talend/en-US/remote-engine-user-guide-linux/Cloud/creating-remote-engines). 7. Start the new Remote Engine. See further information at [Starting a Remote Engine](https://help.qlik.com/talend/en-US/remote-engine-user-guide-linux/Cloud/starting-talend-remote-engine). ## Results Verify the task status on Talend Management Console by checking that Data Services and Routes tasks are up and running. Logs and metrics should also be available for all these tasks.