# HTTP Client properties Properties to configure to be able to send an HTTP request to the server end and get the corresponding response information from the server end. ## HTTP Client connection Select HTTP Client in the list and configure the connection. ## Configuration Select your engine from the list and set the main and advanced settings. Main settingsMain settings | Property | Configuration | | --- | --- | | Base URL | Type in the base URL address you want to access.

Example: https://www.example.com/v1.0/

The second part of the URL needs to be set in the
Path parameter of the HTTP Client dataset
configuration.

You can also define a placeholder using the [Data Shaping Selector Language
syntax](https://help.qlik.com/talend/r/en-US/8.0/data-shaping-language-reference-guide/using-data-shaping-selector-language) to have some parts dynamically filled with the value
extracted from the incoming record. Example:

Base URL =
"https://{.input.job\_url}" and
Path =
"{.input.job\_url\_path}" | | Authentication | Select one of these authentication methods depending on the server
security requirements:

No authentication: no authentication is expected to access the server.
Basic: a username and a password are expected (see the corresponding documentation).
Digest: a username and a password are expected (see the corresponding documentation).
Bearer Token: a raw access token is expected. It will be carried into the HTTP Request header as Authorization: Bearer <your token>.
NTLM: a username (that may contain a domain name) and a password are expected (see the corresponding documentation).
API Key: uses a flexible way to pass an API key token to the server, including the ability to select where to pass it, its name and a prefix:
- Destination: Select where the token will be set: either in a HTTP header with the given name, or in a HTTP query parameter with the given name (not recommended as the token might be visible in the logs).
- Name: Enter the name of the header or query parameter.
- Prefix (optional): Enter the prefix to be added in front of the token (only if the Destination is Request header).
- Token: Enter the authentication token.
OAuth 2.0 automatically manages the retrieval and renewal of the access token against the OAuth server then passes it to the target endpoint as a Bearer token:
- Flow: The OAuth flow you want to execute. Currently, only the Client Credentials flow is supported.
- Authentication mode: For all authentication methods supported, the flow and scope parameters are set in the body using the format 'application/x-www-form-urlencoded' with keys 'grant_type=xxx&scope=xxxx'.
- Token endpoint:
- Enter the authentication token using the format 'oauth2/mydomain.com/token'.
- Client ID and Client secret: Enter the client ID and client secret.
- Additional parameters: Enter the additional attributes you want to add, like the scope attribute for example.

| Advanced settingsAdvanced settings | Property | Configuration | | --- | --- | | Connection timeout (ms) | Type in the base URL address you want to access.

Example: https://www.example.com/v1.0/

The second part of the URL needs to be set in the
Path parameter of the HTTP Client dataset
configuration. | | Read timeout (ms) | Sets the maximum waiting time in milliseconds for the response payload
to be received. An exception will occur if the timeout expires before there
is data available to read. | | Bypass server certificate validation | When enabled, the server certificate is not validated by the client. It
is intended only for test purposes and thus should be disabled in production
environments. | | Use a proxy | Enable this toggle so that the connection between the client and the
server is established through an HTTP or SOCKS proxy:

Proxy type: select the type of proxy you want to use. The HTTP proxy support basic authentication.
Proxy host and Proxy port: enter the address and port of the proxy.
Proxy login and Proxy password (HTTP only): enter the credentials needed to authenticate to the proxy.

| | Retry with exponential backoff | Select this option to automatically retry failed HTTP calls. If
selected, HTTP calls are retried if there is a timeout, or if the HTTP
status code is higher or equal to 400, however no retries are done for
401/403/511 errors which are authentication errors.

Initial backoff (ms): Enter the duration to wait, in milliseconds, before the first retry of the HTTP call.
Backoff factor: When an attempt fails, the duration to wait is multiplied by this factor in order to increase the delay between each call. If the factor is 1, all delays will be the same.
Maximum number of retries: Enter the maximum number of attempts for a single HTTP query. The initial call should not be included in this number.

Example for the following configuration:

Intial backoff:
300 ms, Backoff factor:
2, and Maximum number of retries:
4

The HTTP Client connector does a first
GET operation on the server (first call). A connection timeout error is
returned, the retry mechanism is activated, and the connector waits for
300 milliseconds. A second attempt is then made (first retry). A 503 HTTP
response is returned, the wait duration is multiplied by 2, so the
connector waits for 600 milliseconds. A third attempt is made (second
retry). A 500 Internal server error is returned, the wait duration is
multiplied by 2, so the connector waits for 1200 milliseconds. A fourth
attempt is made (third retry), and this time it is successful. No
additional attempt is made, and the response is returned. | After configuring the connection, give it a display name (mandatory) and a description (optional). ## HTTP Client dataset Dataset configurationDataset configuration | Property | Configuration | | --- | --- | | Dataset name | Enter a
display name for the dataset. This name will be used as a unique identifier of
the dataset in all Talend Cloud apps. | | Connection | Select
your connection in the list. If you are creating a dataset based on an existing
connection, this field is read-only. | | Type | Select the type of dataset you want to create:

Batch if you want to execute the HTTP query once. The pipeline that uses this dataset will be a batch pipeline.
Streaming if you want to execute the HTTP query every N milliseconds in a streaming pipeline. The pipeline that uses this dataset will be a streaming pipeline and you will be able to define the polling interval in milliseconds in the Min poll interval (ms) field of the source dataset.

| Main settingsMain settings | Property | Configuration | | --- | --- | | HTTP method | Select an HTTP method in the list to specify the action to be
performed. | | Path | Enter the second part of the URL you previously set in the configuration
of the connection on which this dataset is created. The concatenation of
both designates the resource you are targeting with this dataset.

The Base URL (connection) and
Path (dataset) values are concatenated, and a
/ character is added when needed.

You can also define a placeholder using the [Data Shaping Selector Language
syntax](https://help.qlik.com/talend/r/en-US/8.0/data-shaping-language-reference-guide/using-data-shaping-selector-language) to have some parts dynamically filled with the value
extracted from the incoming record. Example:

Base URL =
"https://{.input.job\_url}" and
Path =
"{.input.job\_url\_path}" | | Path parameters | Enable this option to specify the extra parameters needed to complete the
Base URL or Path in the form of name-value pairs.

If the Base URL or Path contains a placeholder, you can define parameters
to replace these placeholders:

Name: Enter the name of the placeholder to be replaced.
Value: Enter the value you want to replace the placeholder with. It can be a static value like contactEntity, a value extracted from the incoming record, like {.input.entity.id}, or a mixture of both, like version{.input.api.version}.

For example, with the base URL https://www.example.com
and the path /{api\_version}, you can set the path by
adding a parameter in this table, setting Name to
api\_version and Value to
v1.0. | | Query parameters | Enable this option to specify the parameters that will be set in the
query URL after the ? character in the form of
name-value pairs. These values are automatically encoded.

Name: Enter the name of the parameter.
Value: Enter the value of the parameter. It can be a static value like UUID-1234567, a value extracted from the incoming record, like {.input.user.id}, or a mixture of both, like UUID-{.input.user.id}.

Example: Query parameter name = entityId and query
parameter value = UUID-1234567 | | Request headers | Enable this option to define some HTTP request headers as name-value
pairs. You can define each header to be either part of the main HTTP query
(Main), the authentication query
(Authentication, only available with
OAuth 2.0 authentication), or both queries
(Both).

Name: Enter the name of the header.
Value: Enter the value of the header. It can be a static value like text/html; charset=utf-8, a value extracted from the incoming record, like {.input.meta.content}, or a mixture of both, like text/{.input.document.format}; charset={.input.document.charset}.
Query: Select the query on which this header configuration will be applied.

Example: Header name = Content-Type and header
value = text/html;charset=utf-8 | | Request body | Enable this option if you want to include a message body in the request:

Text: Enter the body of the message as plain text in the text frame under the drop-down list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:
```
id={.input.user.id}name={.input.user.name}  
```
JSON: Enter the body of the message in the JSON format in the text frame under the drop-down list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:
```
{ "id": "{.input.user.id}", "name": "{.input.user.name}", "age": "{.input.user.age}", }
```
XML: Enter the body of the message in the XML format in the text frame under the dropdown list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:
```
<user> <id>{.input.user.id}</id> <name>{.input.user.name}</name> <age>{.input.user.age}</age></user>
```
Form data and x_www_urlencoded: Create the body using a multipart form with attribute name-value pairs or using a URL encoded form with attribute name-value pairs. Enter the attribute names and values in rows of the table under the drop-down list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record.
Binary: Select this option if you want to upload data files larger than 500MB in chunks. Once selected, the File path field is displayed to let you specify a local upload folder.

| | Response body format | Select the response body format from the drop-down list. The right
format allows the connector to parse and apply operations to the response.
Currently, text and JSON formats are supported.

Text: Select this format to return a plain text message. In that case the response payload is not parsed and it will not be possible to extract a sub-part from it.
JSON: Select this format if the response received has a JSON format. In that case the payload is translated into correctly parsed JSON records.

| | Extract a sub-part of the response | Enter the path of a node to select a sub-element of the response. If the
element is an array, each element of the array will be looped over. For more
information about the syntax for entering the node name, see [Data Shaping Selector Language
syntax](https://help.qlik.com/talend/r/en-US/8.0/data-shaping-language-reference-guide/using-data-shaping-selector-language).

This field is optional and must remain empty to
retrieve the entire JSON response.

For an example on how to use it,
read [Additional information about how to extract data with HTTP Client](https://help.qlik.com/talend/en-US/connectors-guide/Cloud/additional-info-about-extracting-data-with-httpclient "You can configure the HTTP Client connector in order to return a specific part of the response body instead of the whole response payload using the Extract a sub-part of the response and Output key/value pairs parameters."). | | Returned content | Select one of the following two options depending on the data returned
by the server.

Body: If Text is selected from the Response body format dropdown list, a record with one string attribute named body is generated and all the payload is copied in it; If JSON is selected from the Response body format dropdown list, a record that represents the hierarchical structure of the parsed document is generated. Limitation: If the body part in the main flow exceeds 200MB, the record will be truncated and a warning will be thrown.
Status, headers, and body: A record is generated with a status attribute (INT type) that contains the HTTP status code, a headers attribute that is a record with all response headers as nested attributes, and a body attribute that is either a simple string containing the full payload as text (if Text is selected from the Response body format dropdown list) or a hierarchical object representing the parsed document (if JSON is selected from the Response body format dropdown list). Limitation: If the body part in the main flow exceeds 200MB, the record will be truncated and a warning will be thrown.
Download file only: Select this option to limit the returned content in the main flow and only download the HTTP response body as a file or to cache, without building a record in the main flow to save resources. This option is meant to be used in the on-premises component in Talend Studio and is not recommended in Talend Cloud.

| | Output key/value pair | Select this option to return key/value pairs instead of the raw body of
the HTTP response. To enter a value for a node, add a row in the table by
clicking the plus button under the table, enter the name of the node in the
Name field, and enter the value in the
Value field.

The value can be from the
component input or from the HTTP response. Enter
"{.input.<dssl\_path>}" in the
Value field if the value is from the component
input or "{.response.<dssl\_path>}" if the
value is from the HTTP response.

In the example for the
Extract a sub-part of the JSON option, you can
retrieve the name and city values of the geologists by entering
.content in the Extract a sub-part
of the JSON field and adding the following two key/value
pairs in the table.

"Name":"{response.name}"
"City": "{response.address.city}"

When retrieving data from an HTTP server, you can specify the
format and content of the data retrieved using the schema, the
Guess schema button, the Response
body format option, the Returned
content option, the Extract a sub-part of the
response option, and the Output key/value
pairs option. For more information, see [tHTTPClient: configuration and output](https://help.qlik.com/talend/en-US/components/8.0/http/thttpclient-schema-and-output). | | Forward input values | Select this option to pass the values received from the component input
to the subsequent component.

This option is available when
Output key/value pairs is selected. | Advanced settingsAdvanced settings | Property | Configuration | | --- | --- | | Accept redirections | Enable this option to apply [the HTTP Redirection rules](https://www.rfc-editor.org/rfc/rfc7231#page-54) on your resources.

Maximum number of redirects: Set the maximum number of redirects that the connector should follow. If there are more redirections than configured, the last one is returned as result of the query.
Redirect only on same host: Enable this option if you want redirections to be performed only when using the same host.

| | Pagination | Select this option to enable the pagination strategy. For more
information about pagination strategy, see *Pagination* section
of [JIRA Server platform REST API
reference](https://docs.atlassian.com/software/jira/docs/api/REST/7.6.1/).

Note that pagination is only compliant with JSON
payload and the desired elements must be in an array in the JSON
payload.

You need to set the following options for pagination to
work properly.

Pagination strategy: Select the pagination strategy to be used:
- Offset/limit: This strategy retrieves the elements starting at a specific position in pages of a specific size. The offset parameter specifies the beginning element in terms of the offset from the first element; the limit parameter specifies the maximum size of a page in terms of the number of the elements. With this strategy applied, the pagination stops when the last call returns 0 elements.
- Marker: This strategy uses a "marker" (often the ID of the last item retrieved) to fetch the next set of results. The marker acts as a pointer to indicate where the next page of data starts, allowing the API to return results that come after the marker. With this strategy applied, the pagination stops when the response contains the marker element.
- Next link: This strategy provides a "next link" (or URL) in the response, which the client can use to retrieve the next set of results. The "next link" includes an offset or a page number that the client can use to fetch the next page. With this strategy applied, the pagination stops when the response contains the next link element.
Location: Specify the way to set the offset and the limit parameters, which can be Query parameters or In headers.
Name of the offset: Set the name of the parameter that will be used as the offset.
Value of the offset: Set the offset. To start from the first element, set it to 0. You can obtain the offset of a query from the incoming records using the DSSL expression '{.input.last_page_info.offset}'.
Name of the limit: Set the name of the parameter that defines the maximum number of elements allowed in a page.
Value of the limit: Set the limit, the maximum number of elements that can be returned in a page. The pagination will stop when a page is returned with 0 elements.
Path to elements: Set the path to the JSON array that contains the desired elements. It must be a JSON array. With offset/limit strategy applied, the element specified by this parameter becomes the root regardless of the setting in Extract a sub-part of the response option. For instance, with JIRA in the Preset field, setting Path to elements to .issue retrieves the issues even if Extract a sub-part of the response is set to .key (assuming that key is an attribute of the elements present in the issues).
Preset: Select preset configuration for specific service that supports pagination strategy by selecting the corresponding service. To select a service, click the [...] button next to this field and select the service. You can select one of the following services.
- BOX: see Offset-based Pagination for more information.
- JIRA: see JIRA Server platform REST API reference for more information.
- OData: see OData Version 4.01. Part 1: Protocol for more information.
Load selected preset: Click this button to load the preset configuration selected in the Preset field.

| | Normalize the JSON HTTP response | Select this option to normalize inconsistencies in the JSON payload in
order for the component to correctly parse these documents:

Normalize attributes: allows you to convert a value to the expected data type.

Normalize arrays: allows you to filter on some elements of a JSON array.

Example:

If you configure the component to retrieve this JSON document, these JSON inconsistencies will be returned in the Job execution console:

Some JSON inconsistencies has been identified:                                    .users.*.addresses is not consistently present in all objects.                                    .users.*.active is not consistently present in all objects.                                    .users.*.email is not consistently present in all objects.                                    .users.*.addresses is an array that contains items of several types: STRING, OBJECT.                                    .users.*.addresses.*.country is not consistently present in all objects.                                    .users.*.addresses.*.street is not consistently present in all objects.                                    .users.*.addresses.*.zipcode is found with varying types: STRING, INT.                                    Please, have a look to advanced settings JSON normalization to fix it.

To prevent this, you could for example use the Normalize attributes option to cast the attribute with the path .users.*.addresses as an ARRAY type, and the attribute with the path users.*.addresses.*.zipcode as an INT type even though some returned values have a String type. You could also add a missing attribute, such as .users.*.active (here you can cast it as a BOOLEAN since it returns either 'true' or 'false').

You could also use the Normalize arrays option to only keep the OBJECT elements in .users.*.addresses even though addresses also contain some String types.

This option is available when JSON is
selected in the Response body format option in the
Basic settings of the component. | ## HTTP Client source/destination configuration When HTTP Client is used as a source or destination dataset in your pipeline, an additional option can be enabled while retrieving/sending data from/to your web server: * Die on error (status code different from 2xx): Enable this option if you want unsuccessful HTTP response status codes (different from 2xx) to raise an error at runtime. This option is disabled by default. ### In this section * [Additional information about how to extract data with HTTP Client](https://help.qlik.com/talend/en-US/connectors-guide/Cloud/additional-info-about-extracting-data-with-httpclient)