Creating knowledge marts

Creating knowledge marts lets you embed and store your structured and unstructured data in a vector database. This allows the augmented context to be retrieved with semantic search features to be used as a context for Retrieval Augmented Generation (RAG) applications.

RAG optimizes the LLM output by providing additional context to the LLM with the query.

Installing the Qlik Data Gateway - Data Movement

To use knowledge marts, you need to connect to vector databases and LLM connections which requires to install a specific Qlik Data Gateway - Data Movement. For more information, see Setting up Qlik Data Gateway - Data Movement for knowledge marts.

Viewing and downloading the logs

You can view and download the logs for the knowledge marts. For more information, see Troubleshooting Data Movement gateway.

Limitations

There are limitations when you use source datasets matching all these conditions:

• Created by SQL transformation or a transformation flow

• Non-materialized

• Historical Data Store (Type 2) turned off

These datasets are considered updated on every run which may affect efficiency and cost. You can mitigate this by:

• Changing the source datasets to be materialized.

• Using explicit dataset transformations.

• Creating global rules that transform multiple datasets.

Supported encoding format

Your files must be properly encoded in UTF-8. Other formats may be wrongly interpreted.

Supported characters

The file and folder names can contain the following characters:

• [0-9], [a-Z], [A-Z]
• ! - _ . * ' ()

Other special characters might be supported but, because of significant special character handling, it is recommended to only use the characters from the list above.

Relationships

• It is not possible to relate data from two datasets. Create a transform task where you define the relationship in the data model, and use the transform task as source for the task.

• When two datasets are related in the data model, both datasets will be available in the task, even if you only selected one of the datasets.