General Web Connector

The Qlik General Web connector can extract data from any URL. The connector enables you to easily connect to many web services in a straight forward and structured way. As it is general in its approach it means that you need to understand the API you're trying to connect to, and ensure that you abide by their terms of use.

The connector provides the following features:

• Allows you to make GET, POST, PUT or DELETE requests to any URL.
• Allows you to read files (for example, JSON files) directly from your local machine.
• Allows JSON data to be automatically converted to XML data.
• Allows you to make requests to SOAP Web Services.
• Allows the request URL to be encrypted to avoid distributing sensitive parameters in your Qlik load scripts.
• Allows a user set a cookie to be sent.
• Allows a custom http headers to be set.
• Allows a minimum time to be set between requests.

• Allows optional basic authentication credentials to be specified.

  Information noteIf you are trying to connect to an OAuth2 based API, this may be possible with this connector if the service provides a mechanism for you to generate your own access token, in their web interface, which you can then send, for example, as a header or URL parameter, using this connector.
• Allows the response to be cached up to a user specified time to minimize requests to the API, for example, where the results change infrequently.
• Allows the content type of the request to be configured.
• Allows the User Agent of the request to be configured.
• Allows high volumes of requests to be made on parallel threads using the asynchronous feature, for the XXXXToTable tables only.
• Allows the response of an API, or any http request, to be saved directly to the local file system. For example you can download images and then binary load them into your QlikView or Qlik Sense application.
• Allows you to access the headers returned from the response using the ResponseMetaData table.
• Allows you to set a timeout for the request, that is, increase or decrease it from the default of 180000 ms or 3 minutes.
• Allows you to configure a retry for the request in the event of a failure after a specified number of milliseconds.

The data that is returned can then be streamed directly into QlikView or Qlik Senseas XML, making use of Qlik's native XML parsing functionality. Alternatively there are tables in the connector which will attempt to convert the data into QVX tables.

Ways to access your data

You can use this connector to connect to many APIs. The following are two examples using the Google Geocoding API.

Working with the Raw API Output

Do the following:

1. Select the JsonToXmlRawtable and click on Parameters.

2. Enter the following URL in the Request URL field:

   http://maps.googleapis.com/maps/api/geocode/json? address=1600+Amphitheatre+Parkway,+Mountain+View,+CA&sensor=false

3. Select GET in the Request Method and click on Save Inputs & Run Table. The processed response appears in the Data Preview.
4. Select either the QlikView or Qlik Sense tab.
5. Copy the generated URL and paste it into the URL field in the Web File dialog. This can be found in the QlikView Script Editor or the Qlik Sense Create new connection selections.

You now have access to the XML parsing and loading functionality.

Using Qlik Web Connectors to generate a table

The other option we have is to use Qlik Web Connectors ability to attempt to convert the response into a table. Using the same URL and GET request as in the XML example, If you now select the JsonToTable table and run the table Qlik Web Connectors extracts the data of a single table.

If you copy and paste the generated URL into a browser, the observed response is the JSON response from the API after it has been converted to XML.

The generated load script can be copied and pasted into QlikView or Qlik Sensein the normal way.

Troubleshooting

xmlParseCharRef Error

If you get an error which is similar to:

error on line XXXX at column Y: xmlParseCharRef: invalid xmlChar value Z

with one of the tables, for example, the JsonToXmlRawtable when viewing the response in either a browser or attempting to load it into QlikView, it may be because the API is returning a an invalid character which is not allowed in XML.

SSL Issues (System.Net.WebException/System.IO.IOException)

If you see the following error:

System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. System.IO.IOException: Received an unexpected EOF or 0 bytes from the transport stream.

along with the following stack trace in your Qlik Web Connectors log file:

System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a

send. ---> System.IO.IOException: Received an unexpected EOF or 0 bytes from the transport stream.

at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count) at

System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest

asyncRequest) at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest

asyncRequest) at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count,

AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.ForceAuthentication(Boolean

receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest) at

System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)

this may be because the service you are trying to connect to uses an older version of SSL, for example, SSL2. It is recommended that only the latest TLS version of SSL is used.

SSL Issues

If you see either of the following:

The remote certificate is invalid according to the validation procedure

or

Could not establish trust relationship for the SSL/TLS secure channel

in your log files or error messages, then Qlik Web Connectors is unable to ensure the validity of the SSL certificate.