Telemetry logging

With the February 2018 release of Qlik Sense, you can capture granular usage metrics from the Qlik in-memory engine based on configurable thresholds. This provides, among other things, the ability to capture CPU and RAM usage of individual chart objects and CPU and RAM usage of reload tasks.

Enabling telemetry logging

Do the following:

1. Open the Qlik Management Console (QMC) by entering the QMC address in your browser.
   By default, the QMC address is https://<QPS server name>/qmc.

2. In the QMC, navigate to Engines, select an engine and go the setting QIX performance log level under Logging.

3. Choose one of the following values:

   • Off: No logging will occur.
   • Error: Activity meeting the error threshold is logged. Recommended option.
   • Warning: Activity meeting the error and warning thresholds is logged. Recommended option.
   • Info: All activity is logged. Only recommended during troubleshooting because the log files may become very big.
   Information noteThe options Fatal and Debug are not applicable in this scenario.

4. Repeat steps 1 and 2 for each engine for which you want to enable telemetry.

5. Set threshold parameters. Edit C:\ProgramData\Qlik\Sense\Engine\Settings.ini. If the file does not exist, create it. You may need to open the file as an administrator to make changes.

   Set the values according to the following list. We recommend that you start with these threshold values and increase or decrease them as you become more aware of how your particular environment performs. Too low values will create very large log files.

   [Settings 7]

   ErrorPeakMemory=2147483648

   WarningPeakMemory=1073741824

   ErrorProcessTimeMs=60000

   WarningProcessTimeMs=30000

6. Save and close the file.

7. Restart the Qlik Sense Engine Service.

8. Repeat steps 4-6 for each engine for which you want to enable telemetry.

Parameter descriptions

ErrorPeakMemory: Default 2147483648 bytes (2 Gb). If an engine operation requires more than this value of Peak Memory, a record is logged with log level error. Peak Memory is the maximum, transient amount of RAM an operation uses.

WarningPeakMemory: Default 1073741824 bytes (1 Gb). If an engine operation requires more than this value of Peak Memory, a record is logged with log level warning. Peak Memory is the maximum, transient amount of RAM an operation uses.

ErrorProcessTimeMs: Default 60000 millisecond (60 seconds). If an engine operation requires more than this value of Process Time, a record is logged with log level error. Process Time is the end-to-end clock time of a request.

WarningProcessTimeMs: Default 30000 millisecond (30 seconds). If an engine operation requires more than this value of Process Time, a record is logged with log level warning. Process Time is the end-to-end clock time of a request.

Reading the logs

Telemetry data is logged to C:\ProgramData\Qlik\Sense\Log\Engine\Trace\<hostname>_QixPerformance_Engine.txt and rolls to the ArchiveLog folder in your ServiceCluster share.

In addition to the common fields found described, fields relevant to telemetry are the following:

Level: The logging level threshold the engine operation met.

ActiveUserId: The User ID of the user performing the operation.

Method: The engine operation itself, seeImportant Engine Operations.

DocId: The ID of the Qlik application.

ObjectId: For chart objects, the Object ID of chart object.

PeakRAM: The maximum RAM an engine operation used.

NetRAM: The net RAM an engine operation used. For hypercubes that support a chart object, the net RAM is often lower than Peak RAM as temporary RAM can be used to perform set analysis, intermediate aggregations, and other calculations.

ProcessTime: The end-to-end clock time for a request including internal engine operations to return the result.

WorkTime: Effectively the same as ProcessTime, excluding internal engine operations to return the result. Will report very slightly shorter time than ProcessTime.

TraverseTime: Time spent running the inference engine (that is, the green, white, and gray).

Important Engine Operations

The Method column details each engine operation and are too numerous to completely detail. The most relevant methods to investigate are as follows and will be the most common methods that show up in the logs if a Warning or Error log entry is written.

Method Description

Global::OpenApp - Opening an application

Doc::DoReload, Doc::DoReloadEx - Reloading an application

Doc::DoSave - Saving an application

GenericObject::GetLayout - Calculating a hypercube (that is, a chart object)

Comments

For best overall representation of the time it takes for an operation to complete, use ProcessTime.

About Error and Warning log level designations: These designations were used because they conveniently fit into the existing logging and QMC frameworks. A row of telemetry information written out as an error or warning does not at all mean the engine had a warning or error condition that should require investigation or remedy unless you are interested in optimizing performance. It is simply a means of reporting on the thresholds set within the engine Settings.ini file and it provides a means to log relevant information without generating overly verbose log files.

In addition to the above information, once the logs mentioned above are created, the Telemetry Dashboard for Qlik Sense can be downloaded and installed to read the log files and analyze the information.

The Telemetry Dashboard provides the ability to capture CPU and RAM usage of individual chart objects, CPU and RAM usage of reload tasks, and more.

The dashboard can be downloaded at: Telemetry Dashboard for Qlik Sense.

Do the following:

1. Run the installer. The files are installed at C:\Program Files\Qlik\Sense.

2. Once installed, you will see two new tasks, two data connections, and one new app in the QMC.

3. In the QMC, change the ownership of the application to yourself, or the user you want to open the app with.

4. Open the Tasks section in the QMC, select TelemetryDashboard-1-Generate-Metadata, and click Start at the bottom. This task will run and automatically reload the app upon completion.

5. Use the application from the hub to browse the information by sheets.