Skip to main content Skip to complementary content

Invoking a REST API to retrieve road traffic data

Availability-noteDeprecated

This scenario aims at helping you set up and use connectors in a pipeline. You are advised to adapt it to your environment and use case.

Example of a pipeline created from the instructions below.

Procedure

  1. Click Connections > Add connection.
  2. In the panel that opens, select the type of connection you want to create.

    Example

    REST
  3. Select your engine in the Engine list.
    Information noteNote:
    • It is recommended to use the Remote Engine Gen2 rather than the Cloud Engine for Design for advanced processing of data.
    • If no Remote Engine Gen2 has been created from Talend Management Console or if it exists but appears as unavailable which means it is not up and running, you will not be able to select a Connection type in the list nor to save the new connection.
    • The list of available connection types depends on the engine you have selected.
  4. Select the type of connection you want to create.
    Here, select REST.
  5. Fill in the connection properties and URL address of the REST server to be invoked as described in REST properties, check the connection and click Add dataset. In this example, the https://opendata.paris.fr/api/datasets/1.0/ service listing Paris open data is invoked.
  6. In the Add a new dataset panel, name your dataset. The REST connector returns a JSON with a parsed body entry.

    Example

    Paris road traffic
  7. Fill in the required properties to invoke the service and click View sample to see a preview of your dataset sample:
    1. In the Type field, select Batch as you want to invoke the service only once.
    2. In the Resource field, type in search.
    3. In the HTTP method field, select GET.
    4. In the Answer body format field, select Text.
    5. Enable the Set URL parameter option and add the query parameters: dataset (parameter) and referentiel-comptages-routiers (value).
  8. Click Validate to save your dataset.
  9. Do the same to add the Test dataset that will be used as a destination in your pipeline to receive logs. Fill in the connection properties as described in Test connection properties.
  10. Click Add pipeline on the Pipelines page. Your new pipeline opens.
  11. Give the pipeline a meaningful name.

    Example

    From Http Rest to Log output - invoke & process
  12. Click ADD SOURCE and select your source dataset, Paris road traffic in the panel that opens.
  13. Click the ADD DESTINATION item on the pipeline to open the panel allowing to select the output dataset in which your output logs will be stored.
  14. In the Configuration tab of the destination, enable the Log records to STDOUT option to capture the logs into this output destination.
  15. Click Save to save your configuration.
  16. On the top toolbar of Talend Cloud Pipeline Designer, click the Run button to open the panel allowing you to select your run profile.
  17. Select your run profile in the list (for more information, see Run profiles), then click Run to run your pipeline.

Results

Your pipeline is being executed, the data that was invoked from a REST API service has been processed and the output logs are stored in the target dataset you have defined.
Log output of the data invoked and processed by the REST API.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here