Declarative pipelines

Declarative pipelines let you define, configure, and deploy Qlik Talend Data Integration pipeline projects entirely in code using YAML files. You edit the files in any integrated development environment or version control system, then deploy them to create or update projects.

When to use declarative pipelines

Use declarative pipelines to promote:

• Collaboration

• Continuous Integration / Continuous Delivery

• Repeatable deployment to different environments

• API-driven automation

Here are some typical use cases:

• Define pipelines in code for continuous integration and development automation.

• Deploy identical pipeline configurations across multiple environments (development, verification, production).

• Reuse and share pipeline definitions across projects or teams.

• Automate large-scale project creation without manual work.

How it works

A pipeline project is represented as a set of YAML files organized in a folder structure that mirrors the project's tasks and datasets. YAML files are easier to read, see diffs, review, reuse, and create automation.

You can export an existing project as YAML, edit the files, and import them back to apply your changes. If the project is connected to version control, the YAML files are directly available in your GitHub repository.

For more information about the YAML file structure, see Retrieving and editing a pipeline configuration.

Typical workflow

Before you start, you must set up your development environment and version control. For more information, see Getting started with declarative pipelines.

A typical declarative pipeline workflow looks like this:

1. Retrieve the pipeline configuration

   You can export an existing project as YAML, retrieve it from version control, or start from a template.

   Retrieving and editing a pipeline configuration

2. Edit the pipeline configuration

   Modify the YAML files in your development environment. You can add tasks, change source selections, and update settings.

   Developer guide: Declarative pipelines

3. Validate the pipeline configuration

   check the files for syntax and semantic errors before deploying the pipeline.

   Validating a pipeline configuration

4. Deploy the pipeline configuration

   Import the files to create or update the project in Qlik Talend Data Integration.

   You can also commit the files to version control and perform apply from version control in the project in Qlik Talend Data Integration.

   Deploying a pipeline configuration

Relationship to version control

When a project is connected to version control, the YAML files are stored directly in your GitHub repository. You can edit the YAML files in GitHub or in any IDE, and then apply the changes back to the project.

For information about connecting a project to version control, see Manage your pipeline projects with version control.

Using AI assistants to configure pipelines

Use AI assistants (such as GitHub Copilot or Anthropic Claude) to streamline YAML configuration creation and editing. To ensure accuracy and adherence to project standards, follow these best practices:

1. Select a high-capability model — Use a premium, large-context model (for example, Anthropic Claude Sonnet 4.6 or higher) rather than a standard or "auto" model for complex YAML generation and schema adherence. Advanced models correctly interpret complex pipeline schemas and produce more accurate configurations.

2. Configure project-specific AI instructions — Improve AI-generated configuration quality by providing project context to the model. Create an instruction file in the root of your VS Code workspace. Choose one of the following:

   • CLAUDE.md (for Anthropic Claude Code extension)

   • .github/copilot-instructions.md (for GitHub Copilot)

   The content of your instruction file should be as follows:

   This is a Qlik Talend Cloud Pipeline (QTCP) project. Before any project advice or YAML edits, I MUST read the source of truth first:
   https://raw.githubusercontent.com/qlik-oss/schemas/refs/heads/main/qtcp/ai-instructions.md

Using AI assistants with Qlik MCP tools

When you deploy a Qlik MCP server in your tenant, AI assistants can use dedicated MCP tools to help manage pipeline projects programmatically.

Prerequisites

• Qlik MCP server is deployed in your tenant

• You have the Qlik MCP: Allowed permission assigned through a custom role

• You have permissions to create or edit pipeline projects

• An AI client with MCP support (such as VS Code with GitHub Copilot) is configured to use the Qlik MCP server

Available MCP capabilities for data pipelines

AI assistants can use the following capabilities to configure and manage pipeline projects:

• Search for spaces — Get a list of available spaces.

• Search for connections — Get a list of available data connections.

• Search for projects — Get a list of existing pipeline projects.

• Search for connection objects — Discover available tables and views from a data source.

• Get pipeline project details — Retrieve project binding values.

Related resources

• To deploy the MCP server, see Qlik MCP server

• For detailed MCP tool documentation, see Qlik MCP tools

• For permission requirements, see Qlik MCP server access and permissions