Retrieving Microsoft Azure web applications, filtering them and creating gists
This scenario aims at helping you set up and use connectors in a pipeline. You are advised to adapt it to your environment and use case.
Before you begin
- You have created and registered Microsoft Azure applications.
- You have configured an OAuth 2.0 client credentials grant flow access.
- You have created a personal access token (PAT) in order to use the GitHub gist API.
Procedure
- Click Connections > Add connection.
-
In the panel that opens, select the type of connection you
want to create.
Example
Azure OAuth Client credentials -
Select your engine
in the Engine list.
Information noteNote:
- It is recommended to use the Remote Engine Gen2 rather than the Cloud Engine for Design for advanced processing of data.
- If no Remote Engine Gen2 has been created from Talend Management Console or if it exists but appears as unavailable which means it is not up and running, you will not be able to select a Connection type in the list nor to save the new connection.
- The list of available connection types depends on the engine you have selected.
-
Select the type of connection you want to create.
Here, select HTTP Client.
-
Fill in the connection properties and URL address of the server to be invoked as
described in HTTP Client properties, check the connection and
click Validate: In this example, the
https://graph.microsoft.com/v1.0/ service listing Microsoft
Azure apps is invoked.
- Base URL: https://graph.microsoft.com/v1.0/
- Authentication type: Select OAuth 2.0.
- Flow: Select Client credentials.
- Authentication mode: Select FORM.
- Token endpoint: Enter https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/token.
- Client ID and Client secret: Enter the Azure web application ID and client secret.
- Additional parameters: Enter https://graph.microsoft.com/.default as the scope attribute.
- Click Add dataset to validate your connection and create the corresponding dataset.
-
In the Add a new dataset panel, name your dataset.
Example
Azure apps -
Fill in the required properties to invoke the service and click View
sample to see a preview of your dataset sample:
- Type: Select Batch as you want to invoke the service only once.
- HTTP method: Select GET.
- Path: Enter Applications so that the final URL will be https://graph.microsoft.com/v1.0/Applications.
- Disable the Parameters option.
-
Enable the Query parameters option and add these query
parameters:
("$select", "appId,displayName,createdDateTime,publisherDomain")
and
("$filter", "createdDateTime ge 2022-12-01T00:00:00.000Z")
- Disable the Request headers and Request body options.
- Response body format: Select JSON.
- Extract Sub-part of the JSON: Enter .value to retrieve records from this JSON field.
- Returned content: select Body.
- Click Validate to save your dataset.
- Click Connections > Add connection.
-
In the panel that opens, select the type of connection you
want to create.
Example
Git PAT - Select your engine in the Engine list.
-
Fill in the connection properties and URL address of the server to be invoked as
described in HTTP Client properties, check the connection and
click Add dataset: In this example, the
https://api.github.com service is invoked.
- Base URL: https://api.github.com
- Authentication type: Select Bearer token.
- Bearer Access token: Enter the personal access token you have previously created.
- Click Add dataset to validate your connection and create the corresponding dataset.
-
In the Add a new dataset panel, name your dataset.
Example
GIST CREATE -
Fill in the required properties to invoke the service and click View
sample to see a preview of your dataset sample:
- HTTP method: Select POST.
- Path: Enter gists so that the final URL will be https://api.github.com/gists.
- Disable the Parameters and Query parameters options.
-
Enable the Request headers option and add these lines
for the Main query:
("Accept", "application/vnd.github+json")
and
("X-GitHub-Api-Version", "2022-11-28")
-
Enable the Request body option, select
JSON and paste these lines:
{ "description":"Azure application {.input.displayName}", "public":false, "files":{ "AzureApp_{.input.appId}.xml":{"content":"<?xml version=\"1.0\"?>\n<application>\n <id>{.input.appId}</id>\n <name>{.input.displayName}</name>\n <createdDate>{.input.createdDateTime}</createdDate>\n <domain>{.input.publisherDomain}</domain>\n</application>" } } }
- Answer body format: Select TEXT.
- Returned content: select Body.
- Click Validate to save your dataset.
- Click Add pipeline on the Pipelines page. Your new pipeline opens.
-
Give the pipeline a meaningful name.
Example
Retrieve Azure apps & filter them on creation date -
Click ADD SOURCE and select your source dataset,
Azure apps, in the panel that opens.
You can see a sample of the retrieved Microsoft Azure apps.
- Click the ADD DESTINATION item and select the destination dataset, GIST CREATE, in the panel that opens.
- On the top toolbar of Talend Cloud Pipeline Designer, click the Run button to open the panel allowing you to select your run profile.
- Select your run profile in the list (for more information, see Run profiles), then click Run to run your pipeline.
Results
Your pipeline is being executed, the Azure apps that were invoked from the Microsoft
Graph service have been filtered and sent to GitHub GIST where XML content for each
application has been created:
Did this page help you?
If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!