HTTP Client properties

Type in the base URL address you want to access.

Example: https://www.example.com/v1.0/

The second part of the URL needs to be set in the Path parameter of the HTTP Client dataset configuration.

You can also define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

Base URL = "https://{.input.job_url}" and Path = "{.input.job_url_path}"

• No authentication: no authentication is expected to access the server.
• Basic: a username and a password are expected (see the corresponding documentation).
• Digest: a username and a password are expected (see the corresponding documentation).
• Bearer Token: a raw access token is expected. It will be carried into the HTTP Request header as Authorization: Bearer <your token>.
• NTLM: a username (that may contain a domain name) and a password are expected (see the corresponding documentation).
• API Key: uses a flexible way to pass an API key token to the server, including the ability to select where to pass it, its name and a prefix:
  • Destination: Select where the token will be set: either in a HTTP header with the given name, or in a HTTP query parameter with the given name (not recommended as the token might be visible in the logs).
  • Name: Enter the name of the header or query parameter.
  • Prefix (optional): Enter the prefix to be added in front of the token (only if the Destination is Request header).
  • Token: Enter the authentication token.
• OAuth 2.0 automatically manages the retrieval and renewal of the access token against the OAuth server then passes it to the target endpoint as a Bearer token:
  • Flow: The OAuth flow you want to execute. Currently, only the Client Credentials flow is supported.
  • Authentication mode: For all authentication methods supported, the flow and scope parameters are set in the body using the format 'application/x-www-form-urlencoded' with keys 'grant_type=xxx&scope=xxxx'.
  • Token endpoint:
  • Enter the authentication token using the format 'oauth2/mydomain.com/token'.
  • Client ID and Client secret: Enter the client ID and client secret.
  • Additional parameters: Enter the additional attributes you want to add, like the scope attribute for example.

Type in the base URL address you want to access.

Example: https://www.example.com/v1.0/

The second part of the URL needs to be set in the Path parameter of the HTTP Client dataset configuration.

Enable this toggle so that the connection between the client and the server is established through an HTTP or SOCKS proxy:

• Proxy type: select the type of proxy you want to use. The HTTP proxy support basic authentication.
• Proxy host and Proxy port: enter the address and port of the proxy.
• Proxy login and Proxy password (HTTP only): enter the credentials needed to authenticate to the proxy.

• Initial backoff (ms): Enter the duration to wait, in milliseconds, before the first retry of the HTTP call.
• Backoff factor: When an attempt fails, the duration to wait is multiplied by this factor in order to increase the delay between each call. If the factor is 1, all delays will be the same.
• Maximum number of retries: Enter the maximum number of attempts for a single HTTP query. The initial call should not be included in this number.

Example for the following configuration:

Intial backoff: 300 ms, Backoff factor: 2, and Maximum number of retries: 4

The HTTP Client connector does a first GET operation on the server (first call). A connection timeout error is returned, the retry mechanism is activated, and the connector waits for 300 milliseconds. A second attempt is then made (first retry). A 503 HTTP response is returned, the wait duration is multiplied by 2, so the connector waits for 600 milliseconds. A third attempt is made (second retry). A 500 Internal server error is returned, the wait duration is multiplied by 2, so the connector waits for 1200 milliseconds. A fourth attempt is made (third retry), and this time it is successful. No additional attempt is made, and the response is returned.

• Batch if you want to execute the HTTP query once. The pipeline that uses this dataset will be a batch pipeline.
• Streaming if you want to execute the HTTP query every N milliseconds in a streaming pipeline. The pipeline that uses this dataset will be a streaming pipeline and you will be able to define the polling interval in milliseconds in the Min poll interval (ms) field of the source dataset.

Enter the second part of the URL you previously set in the configuration of the connection on which this dataset is created. The concatenation of both designates the resource you are targeting with this dataset.

The Base URL (connection) and Path (dataset) values are concatenated, and a / character is added when needed.

You can also define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

Base URL = "https://{.input.job_url}" and Path = "{.input.job_url_path}"

Enable this option to specify the extra parameters needed to complete the Base URL or Path in the form of name-value pairs.

For example, with the base URL https://www.example.com and the path /{api_version}, you can set the path by adding a parameter in this table, setting Name to api_version and Value to v1.0.

• Name: Enter the name of the parameter.
• Value: Enter the value of the parameter. It can be a static value like UUID-1234567, a value extracted from the incoming record, like {.input.user.id}, or a mixture of both, like UUID-{.input.user.id}.

Example: Query parameter name = entityId and query parameter value = UUID-1234567

• Name: Enter the name of the header.
• Value: Enter the value of the header. It can be a static value like text/html; charset=utf-8, a value extracted from the incoming record, like {.input.meta.content}, or a mixture of both, like text/{.input.document.format}; charset={.input.document.charset}.
• Query: Select the query on which this header configuration will be applied.

Example: Header name = Content-Type and header value = text/html;charset=utf-8

• Text: Enter the body of the message as plain text in the text frame under the drop-down list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

  id={.input.user.id}
  name={.input.user.name}  

• JSON: Enter the body of the message in the JSON format in the text frame under the drop-down list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

  {
   "id": "{.input.user.id}",
   "name": "{.input.user.name}",
   "age": "{.input.user.age}",
   }

• XML: Enter the body of the message in the XML format in the text frame under the dropdown list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

  <user>
   <id>{.input.user.id}</id>
   <name>{.input.user.name}</name>
   <age>{.input.user.age}</age>
  </user>

• Form data and x_www_urlencoded: Create the body using a multipart form with attribute name-value pairs or using a URL encoded form with attribute name-value pairs. Enter the attribute names and values in rows of the table under the drop-down list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record.

• Binary: Select this option if you want to upload data files larger than 500MB in chunks. Once selected, the File path field is displayed to let you specify a local upload folder.

• Text: Select this format to return a plain text message. In that case the response payload is not parsed and it will not be possible to extract a sub-part from it.
• JSON: Select this format if the response received has a JSON format. In that case the payload is translated into correctly parsed JSON records.

This field is optional and must remain empty to retrieve the entire JSON response.

For an example on how to use it, read Additional information about how to extract data with HTTP Client.

• Body: If Text is selected from the Response body format dropdown list, a record with one string attribute named body is generated and all the payload is copied in it; If JSON is selected from the Response body format dropdown list, a record that represents the hierarchical structure of the parsed document is generated. Limitation: If the body part in the main flow exceeds 200MB, the record will be truncated and a warning will be thrown.

• Status, headers, and body: A record is generated with a status attribute (INT type) that contains the HTTP status code, a headers attribute that is a record with all response headers as nested attributes, and a body attribute that is either a simple string containing the full payload as text (if Text is selected from the Response body format dropdown list) or a hierarchical object representing the parsed document (if JSON is selected from the Response body format dropdown list). Limitation: If the body part in the main flow exceeds 200MB, the record will be truncated and a warning will be thrown.
• Download file only: Select this option to limit the returned content in the main flow and only download the HTTP response body as a file or to cache, without building a record in the main flow to save resources. This option is meant to be used in the on-premises component in Talend Studio and is not recommended in Talend Cloud.

The value can be from the component input or from the HTTP response. Enter "{.input.<dssl_path>}" in the Value field if the value is from the component input or "{.response.<dssl_path>}" if the value is from the HTTP response.

When retrieving data from an HTTP server, you can specify the format and content of the data retrieved using the schema, the Guess schema button, the Response body format option, the Returned content option, the Extract a sub-part of the response option, and the Output key/value pairs option. For more information, see tHTTPClient: configuration and output.

This option is available when Output key/value pairs is selected.

• Maximum number of redirects: Set the maximum number of redirects that the connector should follow. If there are more redirections than configured, the last one is returned as result of the query.
• Redirect only on same host: Enable this option if you want redirections to be performed only when using the same host.

Note that pagination is only compliant with JSON payload and the desired elements must be in an array in the JSON payload.

You need to set the following options for pagination to work properly.

• Pagination strategy: Select the pagination strategy to be used:
  • Offset/limit: This strategy retrieves the elements starting at a specific position in pages of a specific size. The offset parameter specifies the beginning element in terms of the offset from the first element; the limit parameter specifies the maximum size of a page in terms of the number of the elements. With this strategy applied, the pagination stops when the last call returns 0 elements.
  • Marker: This strategy uses a "marker" (often the ID of the last item retrieved) to fetch the next set of results. The marker acts as a pointer to indicate where the next page of data starts, allowing the API to return results that come after the marker. With this strategy applied, the pagination stops when the response contains the marker element.
  • Next link: This strategy provides a "next link" (or URL) in the response, which the client can use to retrieve the next set of results. The "next link" includes an offset or a page number that the client can use to fetch the next page. With this strategy applied, the pagination stops when the response contains the next link element.
• Location: Specify the way to set the offset and the limit parameters, which can be Query parameters or In headers.
• Name of the offset: Set the name of the parameter that will be used as the offset.
• Value of the offset: Set the offset. To start from the first element, set it to 0. You can obtain the offset of a query from the incoming records using the DSSL expression '{.input.last_page_info.offset}'.
• Name of the limit: Set the name of the parameter that defines the maximum number of elements allowed in a page.
• Value of the limit: Set the limit, the maximum number of elements that can be returned in a page. The pagination will stop when a page is returned with 0 elements.
• Path to elements: Set the path to the JSON array that contains the desired elements. It must be a JSON array. With offset/limit strategy applied, the element specified by this parameter becomes the root regardless of the setting in Extract a sub-part of the response option. For instance, with JIRA in the Preset field, setting Path to elements to .issue retrieves the issues even if Extract a sub-part of the response is set to .key (assuming that key is an attribute of the elements present in the issues).
• Preset: Select preset configuration for specific service that supports pagination strategy by selecting the corresponding service. To select a service, click the [...] button next to this field and select the service. You can select one of the following services.
  • BOX: see Offset-based Pagination for more information.
  • JIRA: see JIRA Server platform REST API reference for more information.
  • OData: see OData Version 4.01. Part 1: Protocol for more information.
• Load selected preset: Click this button to load the preset configuration selected in the Preset field.

• Normalize attributes: allows you to convert a value to the expected data type.
• Normalize arrays: allows you to filter on some elements of a JSON array.

  Example:

  If you configure the component to retrieve this JSON document, these JSON inconsistencies will be returned in the Job execution console:

  Some JSON inconsistencies has been identified:
                                      .users.*.addresses is not consistently present in all objects.
                                      .users.*.active is not consistently present in all objects.
                                      .users.*.email is not consistently present in all objects.
                                      .users.*.addresses is an array that contains items of several types: STRING, OBJECT.
                                      .users.*.addresses.*.country is not consistently present in all objects.
                                      .users.*.addresses.*.street is not consistently present in all objects.
                                      .users.*.addresses.*.zipcode is found with varying types: STRING, INT.
                                      Please, have a look to advanced settings JSON normalization to fix it.

  To prevent this, you could for example use the Normalize attributes option to cast the attribute with the path .users.*.addresses as an ARRAY type, and the attribute with the path users.*.addresses.*.zipcode as an INT type even though some returned values have a String type. You could also add a missing attribute, such as .users.*.active (here you can cast it as a BOOLEAN since it returns either 'true' or 'false').

  You could also use the Normalize arrays option to only keep the OBJECT elements in .users.*.addresses even though addresses also contain some String types.

This option is available when JSON is selected in the Response body format option in the Basic settings of the component.