HTTP Client properties

Type in the base URL address you want to access.

Example: https://www.example.com/v1.0/

The second part of the URL needs to be set in the Path parameter of the HTTP Client dataset configuration.

You can also define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

Base URL = "https://{.input.job_url}" and Path = "{.input.job_url_path}"

• No authentication: no authentication is expected to access the server.
• Basic: a username and a password are expected (see the corresponding documentation).
• Digest: a username and a password are expected (see the corresponding documentation).
• Bearer Token: a raw access token is expected. It will be carried into the HTTP Request header as Authorization: Bearer <your token>.
• NTLM: a username (that may contain a domain name) and a password are expected (see the corresponding documentation).
• API Key: uses a flexible way to pass an API key token to the server, including the ability to select where to pass it, its name and a prefix:
  • Destination: Select where the token will be set: either in a HTTP header with the given name, or in a HTTP query parameter with the given name (not recommended as the token might be visible in the logs).
  • Name: Enter the name of the header or query parameter.
  • Prefix (optional): Enter the prefix to be added in front of the token (only if the Destination is Request header).
  • Token: Enter the authentication token.
• OAuth 2.0 automatically manages the retrieval and renewal of the access token against the OAuth server then passes it to the target endpoint as a Bearer token:
  • Flow: The OAuth flow you want to execute. Currently, only the Client Credentials flow is supported.
  • Authentication mode: For all authentication methods supported, the flow and scope parameters are set in the body using the format 'application/x-www-form-urlencoded' with keys 'grant_type=xxx&scope=xxxx'.
  • Token endpoint:
  • Enter the authentication token using the format 'oauth2/mydomain.com/token'.
  • Client ID and Client secret: Enter the client ID and client secret.
  • Additional parameters: Enter the additional attributes you want to add, like the scope attribute for example.

Type in the base URL address you want to access.

Example: https://www.example.com/v1.0/

The second part of the URL needs to be set in the Path parameter of the HTTP Client dataset configuration.

Enable this toggle so that the connection between the client and the server is established through an HTTP or SOCKS proxy:

• Proxy type: select the type of proxy you want to use. The HTTP proxy support basic authentication.
• Proxy host and Proxy port: enter the address and port of the proxy.
• Proxy login and Proxy password (HTTP only): enter the credentials needed to authenticate to the proxy.

• Initial backoff (ms): Enter the duration to wait, in milliseconds, before the first retry of the HTTP call.
• Backoff factor: When an attempt fails, the duration to wait is multiplied by this factor in order to increase the delay between each call. If the factor is 1, all delays will be the same.
• Maximum number of retries: Enter the maximum number of attempts for a single HTTP query. The initial call should not be included in this number.

Example for the following configuration:

Intial backoff: 300 ms, Backoff factor: 2, and Maximum number of retries: 4

The HTTP Client connector does a first GET operation on the server (first call). A connection timeout error is returned, the retry mechanism is activated, and the connector waits for 300 milliseconds. A second attempt is then made (first retry). A 503 HTTP response is returned, the wait duration is multiplied by 2, so the connector waits for 600 milliseconds. A third attempt is made (second retry). A 500 Internal server error is returned, the wait duration is multiplied by 2, so the connector waits for 1200 milliseconds. A fourth attempt is made (third retry), and this time it is successful. No additional attempt is made, and the response is returned.

• Batch if you want to execute the HTTP query once. The pipeline that uses this dataset will be a batch pipeline.
• Streaming if you want to execute the HTTP query every N milliseconds in a streaming pipeline. The pipeline that uses this dataset will be a streaming pipeline and you will be able to define the polling interval in milliseconds in the Min poll interval (ms) field of the source dataset.

Enter the second part of the URL you previously set in the configuration of the connection on which this dataset is created. The concatenation of both designates the resource you are targeting with this dataset.

The Base URL (connection) and Path (dataset) values are concatenated, and a / character is added when needed.

You can also define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

Base URL = "https://{.input.job_url}" and Path = "{.input.job_url_path}"

Enable this option to specify the extra parameters needed to complete the Base URL or Path in the form of name-value pairs.

Example:

Base URL = https://www.example.com

Path = /{api_version}

Parameter name = api_version

Parameter value = v1.0

• Name: Enter the name of the parameter.
• Value: Enter the value of the parameter. It can be a static value like UUID-1234567, a value extracted from the incoming record, like {.input.user.id}, or a mixture of both, like UUID-{.input.user.id}.

Example:

Query parameter name = entityId

Query parameter value = UUID-1234567

• Name: Enter the name of the header.
• Value: Enter the value of the header. It can be a static value like text/html; charset=utf-8, a value extracted from the incoming record, like {.input.meta.content}, or a mixture of both, like text/{.input.document.format}; charset={.input.document.charset}.
• Query: Select the query on which this header configuration will be applied.

Example:

Header name = Content-Type

Header value = text/html; charset=utf-8

Query = Main

• TEXT: Enter the body of the message as plain text. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

  id={.input.user.id}
  name={.input.user.name}

• JSON: Enter the body of the message in the JSON format. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

  {
      "id": "{.input.user.id}",
      "name": "{.input.user.name}",
      "age": {.input.user.age},
  }

• XML: Enter the body of the message in the XML format. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

  <user>
     <id>{.input.user.id}</id>
     <name>{.input.user.name}</name>
     <age>{.input.user.age}</age>
  </user>

• Form data: Create the body using a multipart form with attribute name and value pairs. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record.
• X_WWW_FORM_URLENCODED: Create the body using a URL encoded form with attribute name and value pairs. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record.

• TEXT: Select this format to return a plain text message. In that case the response payload is not parsed and it will not be possible to extract a sub-part from it nor to use {.response...} placeholders in the Output key/value pair parameters.
• JSON: Select this format if the response received has a JSON format. In that case the payload is translated into correctly parsed JSON records.

(Available only if the answer body format is JSON or XML)

If the body response is parsed as JSON or XML, it is possible to only return a sub-part of the response payload. The Data Shaping Selector Language syntax is used to extract the value. If an array is detected, items are returned one after the other.

• Body: If the Answer body format is TEXT, a record with one String attribute named body is generated and all the payload is copied in it.

  If the Answer body format is JSON, a record that represents the hierarchical structure of the parsed document is generated.

• Status, headers and body: A record is generated with a status attribute (Int type) that contains the HTTP status code, a headers attribute that is a record with all response headers as nested attributes, and a body attribute that is either a simple string containing the full payload as text (Answer body format is TEXT) or a hierarchical object representing the parsed document.

• Name: Enter the name of the attribute to output.
• Value: Enter the value of the attribute. It can be a static value, a value extracted from the incoming record, or the result of the current HTTP query. The input record value or the response of the current HTTPCall are identified by their prefix:
  • {.input}: selector on input record
  • {.response}: selector on response

Example:

Name = id

Value = {.response.user.id}

• Maximum number of redirects: Set the maximum number of redirects that the connector should follow. If there are more redirections than configured, the last one is returned as result of the query.
• Redirect only on same host: Enable this option if you want redirections to be performed only when using the same host.

• Preset: Select the predefined configuration for the popular services supporting pagination. Once selected, click Load selected preset to update the pagination form for the selected environment.

  For more information, read the Box, JIRA, and OData documentation.

• Pagination strategy: It is the pagination strategy that has been implemented in the HTTP service you are querying. Offset/limit is the only supported strategy for now.
• Location: Select where the offset and limit parameters are set. Either in Query parameters or in Headers.
• Name of the offset: Enter the parameter's name that will be used as offset.
• Value of the offset: Set the initial offset value. Enter 0 to start at the very first set.

  You can retrieve this value from the incoming record using the Data Shaping Selector Language syntax.

  Example: {.input.last_page_info.offset}

• Name of the limit: Enter the parameter's name that will be used to define the maximum number of returned elements by page.
• Value of the limit: Set the maximum number of returned elements in a page. The pagination will stop when a page is returned with 0 elements.
• Path to elements: Define the path to the JSON array that contains the returned elements. It must be a JSON array. The Offset/limit will count the number of elements in this array and continue or stop the pagination when it is empty. When the Offset/limit pagination is enabled and this parameter is set, it becomes the root of the Extract a sub-part of the response in the Main configuration.

  Example:

  If you select JIRA/search pagination, issues are returned in the .issues array by default, so you have to enter this value in Path to elements. However, if you want only the Jira keys to be returned, set .key in Extract a sub-part of the response and keys will be returned even if key is an attribute of elements present in issues.

• Normalize attributes: allows you to convert a value to the expected data type.
• Normalize arrays: allows you to filter on some elements of a JSON array.

  Example:

  If you configure the component to retrieve this JSON document, these JSON inconsistencies will be returned in the Job execution console:

  Some JSON inconsistencies has been identified:
                                      .users.*.addresses is not consistently present in all objects.
                                      .users.*.active is not consistently present in all objects.
                                      .users.*.email is not consistently present in all objects.
                                      .users.*.addresses is an array that contains items of several types: STRING, OBJECT.
                                      .users.*.addresses.*.country is not consistently present in all objects.
                                      .users.*.addresses.*.street is not consistently present in all objects.
                                      .users.*.addresses.*.zipcode is found with varying types: STRING, INT.
                                      Please, have a look to advanced settings JSON normalization to fix it.

  To prevent this, you could for example use the Normalize attributes option to cast the attribute with the path .users.*.addresses as an ARRAY type, and the attribute with the path users.*.addresses.*.zipcode as an INT type even though some returned values have a String type. You could also add a missing attribute, such as .users.*.active (here you can cast it as a BOOLEAN since it returns either 'true' or 'false').

  You could also use the Normalize arrays option to only keep the OBJECT elements in .users.*.addresses even though addresses also contain some String types.

This option is available when JSON is selected in the Response body format option in the Basic settings of the component.