tHTTPClient Standard properties

• Built-in: No property data stored centrally.
• Repository: Select the repository file in which the properties are stored. The fields that follow are completed automatically using the data retrieved.

When retrieving data from an HTTP server, you can specify the format and content of the data retrieved using the schema, the Guess schema button, the Response body format option, the Returned content option, the Extract a sub-part of the JSON option, and the Output key/value pairs option. For more information, see tHTTPClient: configuration and output.

• Built-in: You create and store the schema locally for this component only. For more information about a component schema in its Basic settings tab, see Basic settings tab.

• Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs. For more information about a component schema in its Basic settings tab, see Basic settings tab.

The component generates the schema automatically according to the related settings of the component. However, you need to define the schema manually if you select Body from the Returned content drop-down list and JSON from the Response body format drop-down list. In this case, you need to add a column for each of the first-level attributes and set the column type to String for attributes with nested JSON objects.

{
    "id": 1,
    "name": "Talend",
    "address": {
       "city": "San Mateo",
       "state": "California",
       "zipcode": "94402"
    }
}

You need to add these columns in the schema: id (type Int), name (type String), and address (type String). The component can then extract zipcode from the address column using Data Shaping Selector Language (DSSL) (for example, input.address.zipcode).

By default, the columns in the schema depend on the settings of the Response body format and the Returned content options as follows.

• When the Status, headers, and body entry is selected from the Returned content drop-down list, the schema contains three columns: body (type String), headers (type String), and status (type Int).
• When the Body entry is selected from the Returned content drop-down list and TEXT is selected from the Response body format drop-down list, the schema contains one column: body (type String).

When retrieving data from an HTTP server, you can specify the format and content of the data retrieved using the schema, the Guess schema button, the Response body format option, the Returned content option, the Extract a sub-part of the JSON option, and the Output key/value pairs option. For more information, see tHTTPClient: configuration and output.

Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

• View schema: choose this option to view the schema only.

• Change to built-in property: choose this option to change the schema to Built-in for local changes.

• Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion.

  If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.

You can also define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

Base URL = "https://{.input.job_url}" and Path = "{.input.job_url_path}"

Authentication type

• No authentication: no authentication is expected to access the server.
• Basic: A username and a password are expected. For more information, see RFC 2617.
• Digest: A username and a password are expected. For more information, see RFC 2617.
• Bearer token: A raw access token is expected. It will be carried into the HTTP Request header as Authorization: Bearer <your token>.
• NTLM: A username (that may contain a domain name) and a password are expected. For more information, see NT LAN Manager (NTLM) Authentication Protocol.
• API key uses a flexible way to pass an API key token to the server, including the ability to select where to pass it, its name and a prefix:
  • Destination: Select where the token will be set: either in an HTTP header with the given name, or in an HTTP query parameter with the given name (not recommended as the token might be visible in the logs).
  • Name: Enter the name of the header or query parameter.
  • Prefix (optional): Enter the prefix to be added in front of the token (only if the Destination is Request header).
  • Token: Enter the authentication token.
• OAuth 2.0 automatically manages the retrieval and renewal of the access token against the OAuth server then passes it to the target endpoint as a bearer token:
  • Flow: The OAuth flow you want to execute. Currently, only the Client Credentials flow is supported.
  • Authentication mode: For all authentication methods supported, the flow and scope parameters are set in the body in the format application/x-www-form-urlencoded, with keys grant_type=xxx&scope=xxxx.
  • Token endpoint: Enter the authentication token using the format oauth2/mydomain.com/token.
  • Client ID and Client secret: Enter the client ID and client secret.
  • Additional parameters: Enter the additional attributes you want to add, like the scope attribute for example.

To select an HTTP method, click the [...] button next to this field and select the desired HTTP method in the dialog box that appears.

Refer to the description on the Base URL field for related information.

You can also define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

Base URL = "https://{.input.job_url}" and Path = "{.input.job_url_path}"

If the base URL or path contains placeholders, you can add parameters in this table and set the parameter name and parameter value in the following way to replace the placeholders.

• Name: Enter the name of the placeholder to be replaced.

• Value: Enter the value you want to replace the placeholder with. It can be a static value (for example, contactEntity), a value extracted from the incoming record (for example, {.input.entity.id}), or a mixture of both (for example, version{.input.api.version}).

For example, with the base URL https://www.example.com and the path /{api_version}, you can set the path by adding a parameter in this table, setting Name to api_version and Value to v1.0.

• Name: Enter the name of the parameter.
• Value: Enter the value of the parameter. It can be a static value (for example, UUID-1234567), a value extracted from the incoming record (for example, {.input.user.id}), or a mixture of both (for example, UUID-{.input.user.id}).

Example: Query parameter name = entityId and query parameter value = UUID-1234567

• Name: Enter the name of the header.
• Value: Enter the value of the header. It can be a static value (for example, text/html;charset=utf-8), a value extracted from the incoming record (for example, {.input.meta.content}), or a mixture of both (for example, text/{.input.document.format};charset={.input.document.charset}).

  Note that headers with an AUTHENT value can only be sent with OAuth 2.0 authentication queries.

• Query: Select the query on which this header configuration will be applied.

Example: Header name = Content-Type and header value = text/html;charset=utf-8

• Text: Enter the body of the message as plain text in the text frame under the drop-down list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

  id={.input.user.id}
  name={.input.user.name}

• JSON: Enter the body of the message in the JSON format in the text frame under the drop-down list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

  {
   "id": "{.input.user.id}",
   "name": "{.input.user.name}",
   "age": "{.input.user.age}",
   }

• XML: Enter the body of the message in the XML format in the text frame under the drop-down list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

  <user>
   <id>{.input.user.id}</id>
   <name>{.input.user.name}</name>
   <age>{.input.user.age}</age>
  </user>

• Form data and x_www_urlencoded: Create the body using a multipart form with attribute name-value pairs or using a URL encoded form with attribute name-value pairs. Enter the attribute names and values in rows of the table under the drop-down list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record.

The Body type drop-down list is available when you select the Request body option.

• Text: Select this format to return a plain text message. In that case the response payload is not parsed and it will not be possible to extract a sub-part from it.
• JSON: Select this format if the response received has a JSON format. In that case the payload is translated into correctly parsed JSON records.

When retrieving data from an HTTP server, you can specify the format and content of the data retrieved using the schema, the Guess schema button, the Response body format option, the Returned content option, the Extract a sub-part of the JSON option, and the Output key/value pairs option. For more information, see tHTTPClient: configuration and output.

• Body: If Text is selected from the Response body format drop-down list, a record with one string attribute named body is generated and all the payload is copied in it; If JSON is selected from the Response body format drop-down list, a record that represents the hierarchical structure of the parsed document is generated.

• Status, headers, and body: A record is generated with a status attribute (INT type) that contains the HTTP status code, a headers attribute that is a record with all response headers as nested attributes, and a body attribute that is either a simple string containing the full payload as text (if Text is selected from the Response body format drop-down list) or a hierarchical object representing the parsed document (if JSON is selected from the Response body format drop-down list).

When retrieving data from an HTTP server, you can specify the format and content of the data retrieved using the schema, the Guess schema button, the Response body format option, the Returned content option, the Extract a sub-part of the JSON option, and the Output key/value pairs option. For more information, see tHTTPClient: configuration and output.

This field is optional and must remain empty to retrieve the entire JSON response.

{
   "description": "List of some famous geologists",
   "content_length": 5,
   "author": "A. Smith",
   "content":[
      {
         "name":"Maurice Krafft",
         "age":35,
         "female":false,
         "address":{
            "city":"Mulhouse",
            "zipcode":68100
         }
      },
      {
         "name":"Katia Krafft",
         "age":49,
         "female":true,
         "address":{
            "city":"Soultz-Haut-Rhin",
            "zipcode":6860
         }
      },
      {
         "name":"Pline l'Ancien",
         "age":55,
         "female":false,
         "address":{
            "city":"Côme",
            "zipcode":22100
         }
      },
      {
         "name":"James Hutton",
         "age":70,
         "female":false,
         "address":{
            "city":"Édimbourg",
            "zipcode":0
         }
      },
      {
         "name":"Aline Peltier",
         "age":40,
         "female":false,
         "address":{
            "city":"Saint-Pierre",
            "zipcode":97410
         }
      }
   ]
}

{"name":"Katia Krafft","age":49.0,"female":true, ...
{"name":"Pline l'Ancien","age":55.0,"female":false, ...
{"name":"James Hutton","age":70.0,"female":false, ...

|=-----------------=|
|field              |
|=-----------------=|
|Soultz-Haut-Rhin   |
|Côme               |
|Édimbourg          |
'-------------------'

When retrieving data from an HTTP server, you can specify the format and content of the data retrieved using the schema, the Guess schema button, the Response body format option, the Returned content option, the Extract a sub-part of the JSON option, and the Output key/value pairs option. For more information, see tHTTPClient: configuration and output.

This field is available if you select JSON from the Response body format drop-down list.

The value can be from the component input or from the HTTP response. Enter "{.input.<dssl_path>}" in the Value field if the value is from the component input or "{.response.<dssl_path>}" if the value is from the HTTP response.

When retrieving data from an HTTP server, you can specify the format and content of the data retrieved using the schema, the Guess schema button, the Response body format option, the Returned content option, the Extract a sub-part of the JSON option, and the Output key/value pairs option. For more information, see tHTTPClient: configuration and output.

This option is available when Output key/value pairs is selected.

For a MIME multipart response with attachments, the first part that is not an attachment in the body will be treated as the output record body; the attachment part is marked with Content-Disposition=attachment;... header and the attachment filename is taken from its `name=...` value. If the name field is not provided in header metadata, the file will be named using the service name (that is, the Path value). If no part has the Content-Disposition=attachment header, the first part of the multipart is treated as the body for the output record and other parts are treated as the attachments.

For binary files, the tHTTPClient component will save the HTTP response body into a file as is. If there is no name specified in the Content-Disposition header, the result file will be named after the resource name with a _0 suffix.

This option is not recommended in production environments.

• Proxy type: Select the type of proxy you want to use: HTTP or SOCKS. The HTTP proxy support basic authentication.
• Proxy host and Proxy port: Enter the address and port of the proxy.
• Proxy login and Proxy password: Enter the credentials needed to authenticate to the proxy. These two fields are available when HTTP is selected from the Proxy type drop-down list.

• Initial backoff (ms): Enter the duration to wait, in milliseconds, before the first retry of the HTTP call.
• Backoff factor: When an attempt fails, the duration to wait is multiplied by this factor in order to increase the delay between each call. If the factor is 1, all delays will be the same.
• Maximum number of retries: Enter the maximum number of attempts for a single HTTP query. The initial call should not be included in this number.

Example for the following configuration:

Intial backoff: 300 ms, Backoff factor: 2, and Maximum number of retries: 4

The HTTPClient component does a first GET operation on the server (first call). A connection timeout error is returned, the retry mechanism is activated, and the component waits for 300 milliseconds. A second attempt is then made (first retry). A 503 HTTP response is returned, the wait duration is multiplied by 2, so the component waits for 600 milliseconds. A third attempt is made (second retry). A 500 Internal server error is returned, the wait duration is multiplied by 2, so the component waits for 1200 milliseconds. A fourth attempt is made (third retry), and this time it is successful. No additional attempt is made, and the response is returned.

• Maximum number of redirects: Set the maximum number of redirects that the connector should follow. If there are more redirections than configured, the last one is returned as the result of the query.
• Redirect only on same host: Enable this option if you want redirections to be performed only when using the same host.

Note that pagination is only compliant with JSON payload and the desired elements must be in an array in the JSON payload.

You need to set the following options for pagination to work properly.

• Preset: Select preset configuration for specific service that supports pagination strategy by selecting the corresponding service. To select a service, click the [...] button next to this field and select the service. You can select one of the following services.
  • BOX: see Offset-based Pagination for more information.
  • JIRA: see JIRA Server platform REST API reference for more information.
  • OData: see OData Version 4.01. Part 1: Protocol for more information.
• Load selected preset: Click this button to load the preset configuration selected in the Preset field.
• Pagination strategy: Select the pagination strategy to be used. Currently, only Offset/limit strategy is available. The offset/limit strategy retrieves the elements starting at a specific position in pages of a specific size. The offset parameter specifies the beginning element in terms of the offset from the first element; the limit parameter specifies the maximum size of a page in terms of the number of the elements. With Pagination strategy applied, a query stops when the last call returns 0 elements.
• Location: Specify the way to set the offset and the limit parameters, which can be Query parameters or In headers.
• Name of the offset: Set the name of the parameter that will be used as the offset.
• Value of the offset: Set the offset. To start from the first element, set it to 0. You can obtain the offset of a query from the incoming records using the DSSL expression '{.input.last_page_info.offset}'.
• Name of the limit: Set the name of the parameter that defines the maximum number of elements allowed in a page.
• Value of the limit: Set the limit, the maximum number of elements that can be returned in a page. The pagination will stop when a page is returned with 0 elements.
• Path to elements: Set the path to the JSON array that contains the desired elements. It must be a JSON array. With offset/limit strategy applied, the element specified by this parameter becomes the root regardless of the setting in Extract a sub-part of the response option. For instance, with JIRA in the Preset field, setting Path to elements to .issue retrieves the issues even if Extract a sub-part of the response is set to .key (assuming that key is an attribute of the elements present in the issues).

• Normalize attributes: allows you to convert a value to the expected data type.
• Normalize arrays: allows you to filter on some elements of a JSON array.

  Example:

  If you configure the component to retrieve this JSON document, these JSON inconsistencies will be returned in the Job execution console:

  Some JSON inconsistencies has been identified:
  .users.*.addresses is not consistently present in all objects.
  .users.*.active is not consistently present in all objects.
  .users.*.email is not consistently present in all objects.
  .users.*.addresses is an array that contains items of several types: STRING, OBJECT.
  .users.*.addresses.*.country is not consistently present in all objects.
  .users.*.addresses.*.street is not consistently present in all objects.
  .users.*.addresses.*.zipcode is found with varying types: STRING, INT.
  Please, have a look to advanced settings JSON normalization to fix it.

  To prevent this, you could for example use the Normalize attributes option to cast the attribute with the path .users.*.addresses as an ARRAY type, and the attribute with the path users.*.addresses.*.zipcode as an INT type even though some returned values have a String type. You could also add a missing attribute, such as .users.*.active (here you can cast it as a BOOLEAN since it returns either 'true' or 'false').

  You could also use the Normalize arrays option to only keep the OBJECT elements in .users.*.addresses even though addresses also contain some String types.

This option is available when JSON is selected in the Response body format option in the Basic settings of the component.

• Content type: Set the content type of the file defined by HTTP. See https://www.w3.org/Protocols/rfc1341/4_Content-Type.html for related information.
• Encoding: Set the file encoding.
• Path to file: Enter the full path to the file to be uploaded, including the file name.
• Attachment name: Set the HTTP-formed ID of the attachment.

The Add attachments option is available when you select the Request body option and select Form data from the Body type drop-down list in the Basic settings view.

For instance, a geographic location matrix, that should contain only Double, may have one of its item inferred as Integer "[47, -1.567075]". It is error prone, since the connector will try to generate an Integer array, and then will try to add a Double inside. Forcing all numbers to Double will prevent these errors.

This option is available when JSON is selected from the Response body format drop-down list in the Basic settings view.

ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box.

NB_LINE: the number of rows read by an input component or transferred to an output component. This is an After variable and it returns an integer.