Configuring a custom Docker registry for Data Services and Routes

Configure an internal or external Docker registry for Data Services and Routes in a Dynamic Engine environment.

By default, a Docker registry service (docker-registry) is deployed in your Dynamic Engine infrastructure to store images built from Data Services and Routes.

You can customize the setup of the default internal registry service during the deployment or upgrade of your Dynamic Engine instance.
You can also customize the setup of your Dynamic Engine environment to use a private external registry of your own.

Before you begin

The dynamic-engine-crd custom resource definitions must have been installed using the oci://ghcr.io/talend/helm/dynamic-engine-crd helm chart. If not, run the following commands for the installation:
1. Find the chart version to be used:
  - Run the following Helm command:
    helm show chart oci://ghcr.io/talend/helm/dynamic-engine-crd --version <engine_version>
  - See the version directly from Talend Management Console or check the Dynamic Engine changelog for the chart version included in your Dynamic Engine version.
  - Use an API call to the Dynamic Engine version endpoint.
2. Run the following command to install the Helm chart of a given version:
```
helm install dynamic-engine-crd oci://ghcr.io/talend/helm/dynamic-engine-crd --version <helm_chart_version>
```
  Replace <helm_chart_version> with the chart version supported by your Dynamic Engine version.
  Without specifying the version, you install the latest available dynamic-engine-crd chart version.

About this task

The internal Docker registry is shared across all Dynamic Engine environments and is deployed in the qlik-dynamic-engine namespace with global configuration values. Therefore, it needs to be customized with the deployment or upgrade of your Dynamic Engine instance, rather than with a specific Dynamic Engine environment. However, external Docker registries are configured on a per-environment basis.

This architecture allows you to implement different registry strategies for different environments. For example, you can configure:

Development environment: Use the shared internal registry.
Staging environment: Use a dedicated external registry.
Production environment: Use the same external registry as staging.

Docker registry configuration options
Registry type	Customization procedure
Internal registry	See Configuring internal Docker registry for Data Services and Routes. This example shows how to configure pod autoscaling to ensure the smooth deployment of large numbers of Data Services and Routes.
External registry	See Connecting to an external Docker registry for Data Services and Routes. Dynamic Engine connects to and uses your own Docker registry, but your registry service needs to be configured outside Dynamic Engine. Dynamic Engine does not provide the mechanism to configure your registry.

Configuring internal Docker registry for Data Services and Routes

Customize the internal Docker registry that stores images for Data Service and Route services when deploying or upgrading your Dynamic Engine instance.

About this task

In the example below, you activate HPA on the internal Docker registry and set a custom value for the number of replicas to be created and the maximum number of replicas. HPA stands for horizontal pod autoscaling, a Kubernetes feature that automatically adjusts the number of pods in response to demand. A pod is a Kubernetes object that represents a group of one or more containers.

Procedure

Create a custom values file for autoscaling and replica settings:

cat <<EOF > $DYNAMIC_ENGINE_ID-custom-internal-registry.yaml
configuration:
  dockerRegistry:
    replicaCount: 2
    autoscaling:
      enabled: true
      maxReplicas: 5
EOF

Apply this custom values file:

helm install -i dynamic-engine-$DYNAMIC_ENGINE_ID -f $DYNAMIC_ENGINE_ID-values.yaml \
-f $DYNAMIC_ENGINE_ID-custom-internal-registry.yaml \
oci://ghcr.io/talend/helm/dynamic-engine \
--version $DYNAMIC_ENGINE_VERSION

If you are upgrading the engine, use the following command:

helm upgrade -i dynamic-engine-$DYNAMIC_ENGINE_ID -f $DYNAMIC_ENGINE_ID-values.yaml \
-f $DYNAMIC_ENGINE_ID-custom-internal-registry.yaml \
oci://ghcr.io/talend/helm/dynamic-engine \
--version $DYNAMIC_ENGINE_VERSION

What to do next

If not done yet, deploy or upgrade the Dynamic Engine environment assigned to this Dynamic Engine instance, and apply the default or customized values files accordingly. For more customization cases, see the relevant sections in this guide

Connecting to an external Docker registry for Data Services and Routes

Connect to a private external Docker registry to store images for Data Services and Routes in a Dynamic Engine environment.

Before you begin

Ensure the registry is accessible through HTTPS.

Procedure

Run the following command to create a custom values file for the external registry connection.
```
cat <<EOF > custom-routes-registry-values.yaml
configuration:
  registry:
    url: myregistry.example.com
    path: my-specific-registry
    username: foo
    password: bar
    secretName: my-specific-registry
EOF         
```
This command creates a values file to provide the credentials for the registry connection. A secret is a Kubernetes object for storing sensitive information.

If credentials are not required, leave the username, password, and secretName fields empty.

Run the following command to install or upgrade the Dynamic Engine environment with the custom values file.

helm upgrade --install dynamic-engine-environment-$DYNAMIC_ENGINE_ENVIRONMENT_ID \
-f $DYNAMIC_ENGINE_ENVIRONMENT_ID-values.yaml \
-f custom-routes-registry-values.yaml \
oci://ghcr.io/talend/helm/dynamic-engine-environment \
--version $DYNAMIC_ENGINE_VERSION

This command upgrades or installs the Dynamic Engine environment, depending on whether it has been installed.

Validate the deployment:
- In the corresponding Dynamic Engine environment namespace, verify:
  - The dpe-data-service-route-deployer pod is running.
  - If the registry requires credentials, the my-specific-registry and dpe-data-service-route-deployer secrets exist, with my-specific-registry containing a .dockerconfigjson value representing the registry authentication and dpe-data-service-route-deployer containing a REGISTRY_CREDENTIALS value.
- After you deploy a Route or a Data Service, confirm the image exists in the registry by running this command:
```
docker pull myregistry.example.com/my-specific-registry/<artifact_name>:<artifact_version>
```

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here