Skip to main content Skip to complementary content

Creating pipeline tasks

Pipeline tasks are executable integration processes that contain a pipeline artifact published from Talend Cloud Pipeline Designer.

Before you begin

  • You must have the Author permission on the workspace where the task is located to create or edit a pipeline task and manage its configuration.

    The Author permission allows you to configure the Artifact and Parameters steps when creating or editing a task.

  • You must have the Execute permission on the workspace where the task is located to manage the run configuration and schedule, such as setting engine, log level or execution timeout.

    The Execute permission allows you to configure the Processing and Schedule steps when creating or editing a task.

Procedure

  1. Open the Management page.
  2. Go to the Tasks tab.
  3. Select the environment and the workspace using the faceted search at the top of the page.
  4. Click Add task.
  5. In the Artifact step, select the Pipeline artifact type.
  6. Select the artifact you want to execute from the drop-down list.
    The details and description of the artifact appears.
  7. Select the version of the artifact to be used in the task.
    The details and description of the artifact are refreshed automatically.
  8. Select the workspace for the task.
  9. Give a name to the new task.
    By default, the task name is the name of the artifact from which it is created.
    Information noteNote: Tasks must have a unique name within a workspace.
  10. Optional: Add a tag to the task.
    1. Select an existing tag from the drop-down list.
    2. Create a new tag by typing in the name and clicking on it in the drop-down list.
  11. Click Continue.
  12. Optional: In the Parameters step, enter new values then click Continue.
    You can overwrite context variables if you have configured them in your pipeline before publishing it from Talend Cloud Pipeline Designer.
  13. In the Processing step, select the engine on which to run your task from the Engine drop-down list.
    • Cloud Engine for Design
    • A pre-configured Remote Engine Gen2
  14. Select a Run Profile for the execution.
  15. Click Continue.
  16. In the Schedule step, select the execution frequency.
    You can add one or more triggers to schedule your task or you can manually start the task without a trigger.
    For more information, see Scheduling pipeline tasks.
  17. Click Save and close.

Results

Your task is ready to run on Talend Management Console.

Information noteNote:

Exponential back-off retry pattern has been adopted to provide not only faster first and second retries (1 minute and 3 minutes respectively after the initially failed attempt), but also better global behaviour on peak workload situation.

For example, when you try to execute a task that is already running, the execution fails. Talend Management Console retries the execution with incrementally increased intervals, that is to say, the first retry takes place at the end of 1 minute only, the second one 2 minutes, then subsequently 4 minutes and 8 minutes.

If the fourth retry fails, you will get the error message Exceeded the limit of deployment attempts: another instance of the same task was running on the engine at same time. Try to run the task later.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here