Configuring a custom Docker registry for Talend Management Console tasks

Configure an external Docker registry for data integration Jobs, big data Jobs, Data Services, and Routes in a Dynamic Engine environment.

External Docker registries are configured on a per-environment basis.

Tip: For Dynamic Engine versions older than 1.6.0, this procedure applies to Data Service and Route tasks only.

Before you begin

Ensure the registry is accessible through HTTPS.
The dynamic-engine-crd custom resource definitions must have been installed using the oci://ghcr.io/talend/helm/dynamic-engine-crd Helm chart. If not, run the following commands for the installation:
1. Find the chart version to be used:
  - Run the following Helm command:
    helm show chart oci://ghcr.io/talend/helm/dynamic-engine-crd --version <engine_version>
  - See the version directly from Talend Management Console or check the Dynamic Engine changelog for the chart version included in your Dynamic Engine version.
  - Use an API call to the Dynamic Engine version endpoint.
2. Run the following command to install the Helm chart of a given version:
```
helm install dynamic-engine-crd oci://ghcr.io/talend/helm/dynamic-engine-crd --version <helm_chart_version>
```
  Replace <helm_chart_version> with the chart version supported by your Dynamic Engine version.
  Without specifying the version, you install the latest available dynamic-engine-crd chart version.

Procedure

Run the following command to create a custom values file for the external registry connection.
```
cat <<EOF > custom-artifact-registry-values.yaml
configuration:
  registry:
    url: myregistry.example.com
    path: my-specific-registry
    username: foo
    password: bar
    secretName: my-specific-registry
EOF
```
This command creates a values file to provide the credentials for the registry connection. A secret is a Kubernetes object for storing sensitive information.

If credentials are not required, leave the username, password, and secretName fields empty.

Run the following command to install or upgrade the Dynamic Engine environment with the custom values file.

helm upgrade --install dynamic-engine-environment-$DYNAMIC_ENGINE_ENVIRONMENT_ID \
 -f $DYNAMIC_ENGINE_ENVIRONMENT_ID-values.yaml \
 -f custom-artifact-registry-values.yaml \
 oci://ghcr.io/talend/helm/dynamic-engine-environment \
 --version $DYNAMIC_ENGINE_VERSION

This command upgrades or installs the Dynamic Engine environment, depending on whether it has been installed.

Validate the deployment:
- In the corresponding Dynamic Engine environment namespace, verify:
  - The dpe-data-service-route-deployer pod is running.
  - If the registry requires credentials, the my-specific-registry and dpe-data-service-route-deployer secrets exist, with my-specific-registry containing a .dockerconfigjson value representing the registry authentication and dpe-data-service-route-deployer containing a REGISTRY_CREDENTIALS value.
- After you deploy a task, confirm the image exists in the registry by running this command:
```
docker pull myregistry.example.com/my-specific-registry/<artifact_name>:<artifact_version>
```

Results

Dynamic Engine can now connect to and use your own Docker registry for task images, but your registry service needs to be configured outside Dynamic Engine. Dynamic Engine does not provide the mechanism to configure your registry.

Tip: Dynamic Engine ships with an internal Docker registry intended for quick-start deployments. This registry is not recommended for production use and should not be customized for autoscaling in production environments.

If you have customized the internal Docker registry, such as, setting up autoscaling, in a production environment, move the configuration to the values file for your own external Docker registry and deploy the file with the desired Dynamic Engine environments. For example

cat <<EOF > autoscaling-external-registry.yaml
configuration:
  dockerRegistry:
    replicaCount: 2
    autoscaling:
      enabled: true
      maxReplicas: 5
EOF

If you need to keep that configuration in the internal Docker registry, bear in mind that the setup should not be used for production environment and in many use cases, for example, multiple replicas for autoscaling, you must keep the persistent shared storage configuration for Dynamic Engine to be true (configuration.persistence.enabled=true). In this case, persistent shared storage becomes a requirement for deployments of Dynamic Engine and all its environments.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here