Appending data to the index cluster

You can append data from the data sources to the Qlik Big Data Index cluster. You can append data with the management console, or by using the Qlik Big Data Index REST API.

When you append data in Qlik Big Data Index it is automatically detected by Qlik Sense Engine Service. This happens within five minutes after the new data is available to the QSL Manager. The new data is applied the next time you do a selection in Qlik Sense.

Appending data with the management console

You need to do the following to append data with the management console:

1. Add the new parquet file or ORC file to the source data folder.

2. Click Start scanning in the management console to rescan the data source.

   Wait until the scan is complete, and check that the schema was updated with the new data.

3. Click Start syncing to sync the updated data to all nodes and index the updated data.

   Wait until all files are synced and all indexing tasks are completed.

The index is now appended with the new data. If you have a Qlik Sense app open, you need to make a selection to update charts with appended data.

Updating the index with appended data using the Qlik Big Data Index REST API

The Qlik Big Data Index REST API provides a REST gateway to access the public RPCs of Qlik Big Data Index. If you want to notify the indexing cluster that there are new parquet or ORC files available in the data source, you need to execute REST API calls.

You can find out which endpoints are supported by calling:

{url}:{port}/v1/bdi/about/api

Exposing the REST API

You can expose the REST API using port-forwarding with the kubectl command:

kubectl port-forward service/<helm-release-name>-restapi [LOCAL_PORT:]8080

LOCAL_PORT is optional, so if you exclude it, 8080 is used.

Example:  

Appending data

There are two calls required to append data.

1. http://localhost:8080/v1/bdi/indexing/data_scan

2. http://localhost:8080/v1/bdi/indexing/task/add

   Data:

   { "full_flow": true }

Example: Using curl as client

You can add Data Append REST API endpoint calls like these in your data pipeline flow, or execute them manually.

Setting QSL Manager data update frequency

1. If QSL services are running, stop QSL services.
2. Delete qsl_registry_config.json from /output/config/qsl_processor.

3. Edit /dist/runtime/config/template/qsl_registry_config.json and change the idxltUpdateCheckFreqMS setting.

4. Restart QSL services.