Troubleshooting Qlik Answers responses with the reasoning trace

The reasoning trace shows the steps Qlik Answers took to process a question and generate a response. Reviewing the trace helps you verify that the correct data was used and identify where to make improvements when an answer is not what you expected.

For Qlik Answers users, it can help validate your questions or guide you in asking follow up questions. For owners of content such as assistants and applications, it can help you verify that the application data or knowledge bases is providing answers as expected or if you need to make improvements

Accessing the reasoning trace

After Qlik Answers returns a response, click View source below the answer and then click Show reasoning. The trace expands to show each step the pipeline executed.

Understanding the trace structure

The reasoning trace is a log of the agents that ran and the decisions they made. The agents involved, and how many times each one runs, depend on the question asked and the data sources configured in your application. The trace does not follow a fixed sequence. A simple question may involve fewer agents, while a complex multi-part question may call the same agent several times before producing a final response.

The following agents may appear in a trace:

• Answers Agent

• Data Analyst Agent

• Knowledge Base Agent

Answers Agent

The Answers Agent appears at the start and end of every trace. At the start, it reads the question, determines what data sources are needed, and routes the work to the appropriate specialist agents. At the end, it assembles the sub-results into the final response shown to the user.

The opening Answers Agent block describes what the agent understood about the question.

Verify that the subject and any entities are correctly identified. If the agent misread the question at this stage, the specialist agents downstream will query against the wrong context.

Data Analyst Agent

The Data Analyst Agent handles questions involving structured data. It searches the data model for relevant fields and measures, builds calculation expressions, and creates the chart objects that appear in the response.

For complex questions, the Data Analyst Agent may appear more than once in the trace. For example, it may run a first pass to discover fields, a second pass to verify that a specific value exists in the data, and a third pass to build the visualizations.

Knowledge Base Agent

The Knowledge Base Agent handles questions that involve unstructured content, such as internal documents, meeting transcripts, memos, or PDFs connected to your application as a knowledge base data source.

When a question requires both structured and unstructured data, the Answers Agent routes work to both the Knowledge Base Agent and the Data Analyst Agent. Each agent returns its results independently, and the Answers Agent combines them in the final response.

If a user asks a question that clearly references a document or meeting and the Knowledge Base Agent does not appear in the trace, verify that a knowledge base data source is connected to your application and that the content has been indexed.

Reading similarity scores

The most important section of any trace is the field discovery output inside the Data Analyst Agent. This section lists every field and master item the agent considered, along with a similarity score for each one.

The similarity score indicates how well the semantic search matched a field or master item to the user's question. The agent uses the highest-scoring fields and master items to build the response. If a field or master item you expected to be used is absent from the list, or is present but scored lower than a competing field or master item, the answer was likely built using different data than you intended.

Interpreting scores

What to look for

When reviewing the field and master item list, check the following:

• Is your intended field or master item present?

  If it is missing entirely, the field or master item label and description may not match the language used in the question.

• Does your intended field or master item have a higher score than competing fields or master items?

  If a different field or master item scored higher and was used instead, the answer may appear correct but be drawing from the wrong data.

• Are there fields or master items in the list that should not be there?

  A field or master item that scores unexpectedly high may be pulling focus away from the correct field or master item.

The fields and master items most important to answering user questions should consistently score higher than any other fields and master items the agent might confuse them with.

Improving similarity scores

Similarity scores are computed from field and master item names, descriptions, and metadata. Improving these directly raises the score for relevant questions.

Updating field and master item metadata

Do the following:

1. In your application, open Data manager or Data load editor.

2. Locate the field or master item with a low or missing similarity score.

3. Update the field or master item name to use plain business language that reflects how users will ask about it.

   For Data manager, see Editing a table

   For Data load editor, see Renaming fields.

   For master items, see:

   • Editing a master dimension

   • Editing a master measure

4. Add or expand the master item description to clearly explain what the master item contains, what business concept it represents, and when it should be used. See:

   • Editing a master dimension

   • Editing a master measure

5. Reload the application.

6. Ask the same question again and open the reasoning trace to verify the score has improved.

Example:  

A field named NETWR with no description will score significantly lower than a field labeled Net Sales Amount with a description that reads "Total revenue from completed sales transactions, after discounts have been applied." The more closely your metadata reflects natural business language, the higher the field will score when users ask relevant questions.

Resolving field and master item conflicts

When two fields or master items cover similar concepts and compete for the same query, the agent selects the one with the higher score. If the wrong field or master item is being used, you should revise it.

Do the following:

1. Review the metadata for both fields or master items.

2. Make the intended field or master item name and description more specific and more directly aligned with the business term users are searching for.

   For field names:

   For Data manager, see Editing a table

   For Data load editor, see Renaming fields.

   For master items, edit the name or description. See:

   • Editing a master dimension

   • Editing a master measure

3. If the competing field or master item is less relevant, update its name or description to reflect its actual scope more accurately so it scores lower on unrelated queries.

Diagnosing a wrong or incomplete answer

Use the following guidance when the response from Qlik Answers does not match what you expected.

The answer used the wrong field or master item

Do the following:

1. Open the reasoning trace and locate the field discovery output in the Data Analyst Agent.

2. Find the field or master item that was used and note its similarity score.

3. Find the field or master item you expected to be used. If it is in the list, compare its score to the field or master item that was selected.

4. If your intended field or master item scored lower, update its name or description to improve its relevance to the question.

   For master items, edit the description. See:

   • Editing a master dimension

   • Editing a master measure

5. If your intended field or master item is not in the list at all, review its name and description. It is likely described in technical terms that do not match how users phrase questions.

A field or master item is missing from the list entirely

Do the following:

1. Open the field discovery output in the trace.

2. Confirm the field or master item exists in the data model.

3. If you are using a business logic logical model, confirm the field or master item is not ungrouped.

4. Review the field or master item name and description.

   If either is absent or uses internal naming conventions (such as SAP field codes for fields), add a plain-language name and description.

   See:

   • Editing a master dimension

   • Editing a master measure

5. Reload the application and re-run the question to verify the field or master item now appears in the trace.

The answer is correct but incomplete

The Data Analyst Agent's planning output describes which charts were built and which dimensions were selected. If a dimension or measure is missing from the planned charts, its similarity score was likely high enough to appear in the field and master item list but below the threshold for chart selection. Improving the related field or master item metadata will bring it above the selection threshold.

The narrative and the chart show different numbers

The Answers Agent's final synthesis includes bracketed evidence IDs that link each claim in the text to the chart object it came from. For example:

Brand 1 sold 24 vehicles generating total revenue of $1,584,041.82 [d1f09de1-4a5a...]

If the number in the text does not match the chart, the synthesis referenced a different chart object than the one displayed. Locate the referenced object using the ID and compare its expression to the visible chart.

Reference: Agents and their roles