Migrating to Talend Data Preparation 8.0.x R2025-02 with an embedded MongoDB

Before you begin

If you have installed and used Talend Dictionary Service to create custom semantic types or update the predefined ones, you must migrate Talend Dictionary Service before migrating Talend Data Preparation. See Migrating Talend Dictionary Service to version 8.0.x R2022-07 or later.

Download the hybrid Talend Data Preparation 8.0.x archive from Talend Cloud's Downloads page.

Procedure

Stop your current Talend Data Preparation instance but keep the MongoDB instance running.

Back up your MongoDB by running the following command:

mongodump -h <source_mongodb_host>:<source_mongo_port> -d <source_database> -u <source_mongodb_user> -p <source_mongodb_password> -o <dump_output>

The default value for the embedded MongoDB properties are the following:

Property	Value
mongodb.host	localhost
mongodb.port	27017
mongodb.database	dataprep
mongodb.user	dataprep-user
mongodb.password	duser

Stop the MongoDB instance from the previous version of Talend Data Preparation.

Information noteNote: For Windows users, you must uninstall the current MongoDB before you install Talend Data Preparation.
Install Talend Data Preparation and select the embedded MongoDB option. Do not start the instance.
If you have installed Talend Data Preparation in service mode, stop the instance and MongoDB
Delete the database:
1. Open a mongo shell as a Talend Data Preparation database user and execute the following commands:
```
cd <install_dir>/mongodb/bin
./mongo dataprep -u dataprep-user -p dprep --port 27017
```
2. Switch to the Talend Data Preparation database: use dataprep
3. Drop the database: db.runCommand( { dropDatabase: 1 } )
4. Exit mongo shell: exit

Restore the MongoDB backup on the new 8.0 MongoDB instance by using the following command:

mongo_path>/bin/mongorestore -h <mongo_host>:<mongo_port> -u <mongo_tdp_username> -p <mongo_tdp_password> -d <mongo_tdp_database> <backup_location>

To migrate from 8.0.x R2023-11 or before, update the <Data_Preparation_Path>/config/application.properties file:

Backup the <Data_Preparation_Path>/config/application.properties file.

Edit the file and update the properties:

Remove the following properties:

 # Service documentation
 service.documentation=true
 service.documentation.name=Talend Data Preparation - API
 service.documentation.description=This service exposes high level services that may involve services orchestration.
 service.paths=api
 springfox.documentation.swagger.v2.host=${public.ip}:${server.port}${gateway-api.service.path}
 springfox.resources.prefix.url=${gateway-api.service.path}
 # see https://github.com/springfox/springfox/issues/2265
 logging.level.io.swagger.models.parameters.AbstractSerializableParameter=error

 security.basic.enabled=false
 security.oidc.client.expectedIssuer=${iam.uri}/oidc
 iam.license.url=${iam.uri}/oidc/api
 security.oidc.client.keyUri=${iam.uri}/oidc/jwk/keys
 security.oauth2.client.clientId=&lt;security client id>
 security.oauth2.client.clientSecret=&lt;security client secret>
 security.oidc.client.claimIssueAtTolerance=120
 # security.oauth2.resource.serviceId=${PREFIX:}resource
 security.oauth2.resource.tokenInfoUri=${iam.uri}/oidc/oauth2/introspect
 security.oauth2.resource.uri=/v2/api-docs,/api/**,/folders/**,/datasets/**,/dataset/**,/preparations/**,/transform/**,/version/**,/acl/**,/apply/**,/export,/export/**,/aggregate,/sampling/**,/receivers/**,/error,/datastores/**,/preparation/**,/actions/**,/suggest/**,/dictionary/**,/transformation/preparations/**,/transformation/v2/**,/sharing/**
 security.oauth2.resource.filter-order=3
 security.scim.enabled=true

 security.oauth2.client.access-token-uri=${iam.uri}/oidc/oauth2/token
 security.oauth2.client.scope=openid refreshToken
 security.oauth2.client.user-authorization-uri=${iam.uri}/oidc/idp/authorize
 security.oauth2.sso.login-use-forward=false

 security.oauth2.resource.tokenInfoUriCache.enabled=true

 # FIXME: temporary fix related to dataquality-semantic-consumer version = 11.2.0
 # FIXME: that will need to be removed as soon as we bump to spring boot 2.7 and bump dq library at the same time
 spring.cloud.stream.binders.kafka-binder-consumer.environment.spring.cloud.stream.kafka.binder.brokers=${spring.cloud.stream.kafka.binder.brokers}

 # Zuul properties
 gateway-api.service.url=http://${public.ip}:${server.port}
 zuul.servletPath=/gateway/upload

 zuul.routes.dq.path=/gateway/dq/semanticservice/**
 zuul.routes.dq.sensitiveHeaders=${zuul.sensitiveHeaders}
 zuul.routes.dq.url=${dataquality.server.url}/
 proxy.auth.routes.dq=oauth2

 zuul.routes.me.path=/gateway/api/v1/scim/me/**
 zuul.routes.me.url=${iam.scim.url}/Me
 proxy.auth.routes.me=oauth2

 zuul.routes.pendo.path=/gateway/api/iam-server/pendo/**
 zuul.routes.pendo.url=${iam.scim.url}/pendo
 proxy.auth.routes.pendo=oauth2

 zuul.routes.sharingset.path=/gateway/api/v1/sharingset/**
 zuul.routes.sharingset.sensitiveHeaders=${zuul.sensitiveHeaders}
 zuul.routes.sharingset.url=http://${public.ip}:${server.port}/sharing/v1/sharingset
 proxy.auth.routes.sharingset=oauth2

 zuul.routes.sharing.path=/gateway/api/v1/sharing/**
 zuul.routes.sharing.sensitiveHeaders=${zuul.sensitiveHeaders}
 zuul.routes.sharing.url=http://${public.ip}:${server.port}/sharing/v1/sharing
 proxy.auth.routes.sharing=oauth2

 zuul.routes.sharings.path=/gateway/api/v1/sharings/**
 zuul.routes.sharings.sensitiveHeaders=${zuul.sensitiveHeaders}
 zuul.routes.sharings.url=http://${public.ip}:${server.port}/sharing/v1/sharings
 proxy.auth.routes.sharings=oauth2

 zuul.routes.api.path=/gateway/api/**
 zuul.routes.api.sensitiveHeaders=${zuul.sensitiveHeaders}
 zuul.routes.api.url=http://${public.ip}:${server.port}/api
 proxy.auth.routes.api=oauth2

 spring.cloud.gateway.mvc.routes[0].id=api
 spring.cloud.gateway.mvc.routes[0].uri=http://${public.ip}:${server.port}
 spring.cloud.gateway.mvc.routes[0].predicates[0]=Path=/gateway/api/**
 spring.cloud.gateway.mvc.routes[0].filters[0]=RewritePath=/gateway/api/(?<segment>.*), /api/$\{segment}

 zuul.routes.upload.path=/api/upload/**
 zuul.routes.upload.sensitiveHeaders=${zuul.sensitiveHeaders}
 zuul.routes.upload.url=http://${public.ip}:${server.port}/api
 proxy.auth.routes.upload=oauth2

 zuul.routes.swagger1.path=/gateway/v3/api-docs/**
 zuul.routes.swagger1.sensitiveHeaders=${zuul.sensitiveHeaders}
 zuul.routes.swagger1.url=http://${public.ip}:${server.port}/v3/api-docs
 proxy.auth.routes.swagger1=oauth2

 zuul.ignoredPatterns=/login,/logout,/signOut,/signIn
 zuul.sensitiveHeaders=Cookie,Set-Cookie
 zuul.host.socket-timeout-millis=300000
 zuul.host.connect-timeout-millis=5000

Add the following properties:

# Service documentation
  springdoc.server.url=${gateway-api.service.path}
  springdoc.swagger-ui.path=/swagger-ui/
  springdoc.swagger-ui.disable-swagger-default-url=true
  springdoc.swagger-ui.docExpansion=none
  springdoc.swagger-ui.operationsSorter=alpha
  springdoc.swagger-ui.tagsSorter=alpha

  springdoc.swagger-ui.urls[0].name=semanticservice
  springdoc.swagger-ui.urls[0].display-name=Talend Data Preparation - Semantic Service
  springdoc.swagger-ui.urls[0].url=${gateway-api.service.path}/dq/semanticservice/api-docs

  # Auth0 issuer uri  - Used only for daikon configuration
  spring.security.oauth2.resourceserver.auth0.jwt.issuer-uri=https://talend.auth0.com/
  # IAM authentication validation
  spring.security.oauth2.resourceserver.iam.jwt.jwk-set-uri=${iam.uri}/oidc/jwk/keys
  spring.security.oauth2.resourceserver.iam.opaque-token.introspection-uri=${iam.uri}/oidc/oauth2/introspect
  spring.security.oauth2.resourceserver.iam.opaque-token.query-entitlements=true

  # IAM authorization code flow
  spring.security.oauth2.client.registration.oidc.clientId=&lt;security client id>
  spring.security.oauth2.client.registration.oidc.clientSecret=&lt;security client secret>
  spring.security.oauth2.client.registration.oidc.authorization-grant-type=authorization_code
  spring.security.oauth2.client.registration.oidc.scope=openid,refreshToken
  spring.security.oauth2.client.registration.oidc.redirectUri=http://${public.ip}:${server.port}/signIn
  spring.security.oauth2.client.registration.oidc.provider=oidc

  spring.security.oauth2.client.provider.oidc.authorization-uri=${iam.uri}/oidc/idp/authorize
  spring.security.oauth2.client.provider.oidc.token-uri=${iam.uri}/oidc/oauth2/token
  spring.security.oauth2.client.provider.oidc.jwk-set-uri=${iam.uri}/oidc/jwk/keys

  # Kafka configuration
  spring.cloud.function.definition=internalMessage;internalMessageDatasetModule;tsdEventConsumer;tsdSemanticCacheUpdate

  spring.cloud.gateway.mvc.http-client.connect-timeout=5000

Replace the client ID and client secret in new the properties. You can get them from the backup file.

Remove the following properties:

## Enable JWT needed in hybrid mode:
  security.oauth2.resource.jwt.key-uri=${iam.uri}/oidc/jwk/keys
  ## Redefine security scope by adding entitlements for hybrid
  security.oauth2.client.scope=openid refreshToken entitlements

Add the scope entitlements to the property spring.security.oauth2.client.registration.oidc.scope.
```
  spring.security.oauth2.client.registration.oidc.scope=openid,refreshToken,entitlements
```

To migrate from 8.0.x R2024-05 or before, update the <Data_Preparation_Path>/config/application.properties file:

Backup the <Data_Preparation_Path>/config/application.properties file.

Edit the file and update the properties:

Remove the following property:

# Proxy for search purpose
 help.search.url=https://www.talendforge.org/find/api/THC.php

Replace the following property:

# Support url
 support.url=https://www.talend.com/services/technical-support/
 # Community link
 community.url=https://community.talend.com/s/group/0F93p0000000VNyCAM/data-quality-preparation-stewardship

with the following property:

# Support url
 support.url=https://community.qlik.com/t5/Support/ct-p/qlikSupport
 # Community link
 community.url=https://community.qlik.com/t5/Data-Quality-Preparation-Stewardship/bd-p/qlik-t-data-quality-prep-stewardship

To retrieve the content of the data-set storage folder used for the previous Talend Data Preparation and migrate it to version 8.0, complete the next step according to the your situation:
- If you have set a custom value for the dataset.content.store.file.location property, make sure that you set the same value in the application.properties file for the 8.0 version.
- If you have never modified the value of the dataset.content.store.file.location property, copy the content of the <Data_Preparation_Path>/data folder used in 1.x/2.x, and paste it in the 8.0 installation folder, following the same path.
Start Talend Data Preparation.

Results

You have migrated your data to Talend Data Preparation 8.0. To configure Talend Data Preparation, follow the instructions in Configuring Talend Data Preparation.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here