Scheduling tasks

Scheduling is used to keep the target or pipeline up-to-date with the latest changes to the source datasets. You can schedule two types of task in Qlik Talend Data Integration: Data movement tasks and ELT tasks. Although this topic focuses on Data movement tasks, links to scheduling instructions for ELT tasks are provided at the end of the topic.

Data movement tasks that can be scheduled are determined by the pipeline project type:

• Replication: Replicate data and Land data in data lake tasks

• Data pipeline: Landing, Lake landing tasks

ELT tasks that can be scheduled are Storage, Transformation, and Data mart tasks, which are only available with data pipeline projects.

Scheduling data movement tasks

Depending on the source connector type, your Qlik subscription, and whether or not you are using Passerelle de déplacement des données, you might need to schedule your task to retrieve changes from the source datasets.

In the following cases, scheduling is required to retrieve changes from the source datasets:

• Using any data source connector without Passerelle de déplacement des données

• Using SaaS application connectors with or without Passerelle de déplacement des données

• Using the SAP OData connector together with the As scheduled option in the task settings.

• Running a data movement task configured with the following update methods:

  • Full load only: Replicate data task.

  • Reload: Land data in data lake and lake landing tasks.

  • Reload and compare: Landing task

• Source datasets that do not support CDC

The schedule determines how often the target datasets will be updated with changes to the source datasets. Sometimes, however, not all of the source datasets will support CDC.

If the dataset supports CDC and the task update method is CDC-based (for example, Apply changes), only the changes to that dataset will be retrieved and propagated to the corresponding target table. If a dataset does not support CDC (for example, a View) or if the task update method is not CDC-based (for example, Reload), changes will be propagated by reloading the entire dataset.

Scheduling when using SaaS application connectors

With SaaS application connectors, a single task will be created with the ability to schedule CDC intervals and reload intervals.

Scheduling when using other connector types

When using other connector types (such as database connectors) with a task defined with a CDC-based update method, if some of the selected datasets support CDC and some do not, two separate sub-tasks will be created: one for retrieving the changes to datasets that support CDC, and the other for reloading datasets that do not support CDC.

Scheduling consideration for CDC datasets versus Full load only datasets

Datasets that do not support CDC are usually updated less frequently than those that do. In this case, best practice is to set a longer scheduling interval for the datasets that do not support CDC. However, if all of the datasets are updated with a similar frequency, it is recommended to maintain the same scheduling interval for both tasks.

Pour des informations sur les intervalles de planification minimaux en fonction du type de source de données et du niveau d'abonnement, consultez Intervalles de planification minimaux autorisés.

Setting scheduling

To set the scheduling:

1. Open your pipeline project and then do one of the following:

   • In tasks view, click on the data task and select Scheduling.
   • In pipeline view, click on the data task and select Scheduling.
   • Open the data task and click the Scheduling toolbar button.

2. The following options are available:

   • Toggle the scheduling on or off.

   • Choose Hourly, Daily, Weekly, or Monthly from the drop-down.

   • Set the Interval.

   • Reload tables that do not support CDC every: This option will only be shown if you are using a SaaS application source and CDC is enabled in the task settings. The specified value will be multiplied by the value of the Interval field. For example, if Interval is set to Every 1 hour and Reload tables that do not support CDC every is set to 4, datasets that do not support CDC will be reloaded every 4 hours.

      

     If all datasets support CDC, this option will be ignored. But if datasets that do not support CDC are later added to the task (either manually or by matching the dataset selection pattern), they will be reloaded according to the specified interval.

     Note Informations

     The term(s) used for CDC differ according to task type:

     • Standard replication task: "Apply changes" and "Store changes"

     • Lake landing and Landing tasks: "Change data capture"

   • Set a Start time and Start date for the scheduling.

3. Click OK to save your settings.

Executing a missed run for a task based on Passerelle de déplacement des données

Il peut arriver qu'un problème réseau entraîne la perte de la connexion à Passerelle de déplacement des données. Si la connexion à Passerelle de déplacement des données n'est pas rétablie avant l'exécution planifiée suivante, la tâche de données ne pourra pas s'exécuter conformément à la planification. Dans ce cas, vous pouvez choisir d'exécuter ou non un cycle immédiatement après le rétablissement de la connexion.

The default settings for all Passerelle de déplacement des donnéess are defined in the Administration activity center. You can override these settings for individual tasks as described below.

To do this

1. Open your project and then do one of the following:

   • In tasks view, click on the data task and select Scheduling.

   • In pipeline view, click on the data task and select Scheduling.

   • Open the data task and click the Scheduling toolbar button.

   The Scheduling - <task> dialog opens.

2. At the bottom of the dialog, toggle on Use custom settings for this task.

3. Choose one of the following Run missed scheduled tasks options.

   • As soon as possible and then as scheduled if it's important to run a task before the next scheduled instance

   • As scheduled to run the task at the next scheduled instance

4. Save your settings.

See also: Exécution d'une tâche après une planification manquée.

Scheduling other task types

In data pipeline projects, you also need to schedule Storage, Transformation, and Data Mart tasks to update them with changes made to the input datasets.

For more information, see:

• Planification d'une tâche de stockage

• Planification d'une tâche de transformation

• Planification d'une tâche de datamart