Using a private image registry for Dynamic Engine deployments

Configure Dynamic Engine to use a private image registry for deployments.

Before you begin

Do not change the image path or tag for each Dynamic Engine service.

About this task

When the public registry is not accessible, you can configure an alternative image registry to get the charts to deploy or upgrade Dynamic Engine. This allows Kubernetes clusters to pull required images from a private or internal registry.

This procedure demonstrates how to configure a private image registry as a cache or proxy for Dynamic Engine deployments.

In this example, the private image registry is accessible at myregistry.example.com and requires credentials.
The proxy repository to ghcr.io is accessible at myregistry.example.com/docker-ghcr-io-remote.

Procedure

Create the Helm values file for this private registry.

cat <<EOF > custom-image-registry-values.yaml
global:
  registry:
    dockerRegistry: myregistry.example.com/docker-ghcr-io-remote
    dockerUsername: <registry_username>
    dockerPassword: <registry_password>
EOF

Authenticate with the private registry.
```
helm registry login myregistry.example.com
```
This command allows Helm to access the credentials to pull charts or images during Dynamic Engine deployments.

Install or upgrade a Dynamic Engine instance and its environment using the private registry.

helm upgrade --install dynamic-engine-crd \
  oci://myregistry.example.com/docker-ghcr-io-remote/talend/helm/dynamic-engine-crd \
  --version $DYNAMIC_ENGINE_VERSION

helm upgrade --install dynamic-engine-$DYNAMIC_ENGINE_ID \
  -f $DYNAMIC_ENGINE_ID-values.yaml \
  -f custom-image-registry-values.yaml \
  oci://myregistry.example.com/docker-ghcr-io-remote/talend/helm/dynamic-engine \
  --version $DYNAMIC_ENGINE_VERSION

helm upgrade --install dynamic-engine-environment-$DYNAMIC_ENGINE_ENVIRONMENT_ID \
  -f $DYNAMIC_ENGINE_ENVIRONMENT_ID-values.yaml \
  -f custom-image-registry-values.yaml \
  oci://myregistry.example.com/docker-ghcr-io-remote/talend/helm/dynamic-engine-environment \
  --version $DYNAMIC_ENGINE_VERSION

Verify the deployment.
- The imagePullSecrets specification contains the secret the pods use for pulling images from your private registry. The default secret is talend-registry.
```
spec:
  imagePullSecrets:
    - name: talend-registry
```
- All image paths start with myregistry.example.com/docker-ghcr-io-remote.
  You can run this command to check the image paths:
  kubectl get deployment -A -l 'app.qlik.com/part-of=qlik-dynamic-engine' \ -o jsonpath="{range .items[*]}{.metadata.name}{'\t'}{..image}{'\n'}{end}"

Results

Once complete, all Dynamic Engine Helm releases should be installed, that is to say, all named deployments of the Helm charts such as dynamic-engine-crd, dynamic-engine, and dynamic-engine-environment are present.

All pods in the namespace should be running.

In Talend Management Console, the status of this environment becomes Ready, confirming that it is ready to run tasks or plans.

If the image registry is misconfigured or images cannot be pulled, pods will show ImagePullBackoff errors.

Troubleshooting:

This table lists common errors when installing or upgrading Dynamic Engine charts: dynamic-engine-crd, dynamic-engine, and dynamic-engine-environment.

Dynamic Engine chart installation errors
Error	Cause	Remediation
Chart is not found	Wrong chart version Misconfigured proxy	Ensure to use the versions provided in the official documentation. Use the commands provided below this table to verify your proxy setup.
User is not authenticated	Login command not executed or missing credentials	Log in to the configured private image registry.
User is not authorized	Insufficient permissions for the private registry	Ensure your user can pull images from the configured registry.

For a correctly configured proxy, the following commands should return the same result:

Show charts from the private registry:

helm show chart oci://myregistry.example.com/docker-ghcr-io-remote/talend/helm/dynamic-engine \
  --version $DYNAMIC_ENGINE_VERSION

Show charts from the public registry:

helm show chart oci://ghcr.io/talend/helm/dynamic-engine --version $DYNAMIC_ENGINE_VERSION

This table lists common errors when the dynamic-engine-operator attempts to install or upgrade a services chart in the Dynamic Engine or Dynamic Engine environment namespace.

Service chart installation errors
Error	Cause	Remediation
Chart is not found	Docker image registry is not correctly configured	Verify the image registry configuration.
User is not authenticated	Docker image registry credentials are not configured	Verify the image registry configuration.
User is not authorized	Docker image registry credentials do not have read permissions	Verify the image registry configuration.

What to do next

After successful deployment, you can add tasks to the Dynamic Engine environment as you would for standard engines. For details, see Adding a Job task in a Dynamic Engine environment.

Tip:

If your Kubernetes cluster is air-gapped, image pulls are routed through the private registry, so the cluster does not require direct outbound HTTP or HTTPS access.

However, if Dynamic Engine services need to access external HTTP or HTTPS endpoints, which is often the case when working with Qlik Talend Cloud, you must also configure an HTTP proxy in your Helm values file.

For instructions, see Using an HTTP proxy for Dynamic Engine services in air-gapped environments.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here