Qlik Logging Service – configuration and integration

This topic describes the configuration and integration features that are available for the Qlik Logging Service. These features should be used in the context of troubleshooting issues with the assistance of Qlik technical support.

Dynamic configuration

Most of the functionality provided by the Qlik Logging Service is controlled by the configuration file: QlikCentralizedLogging.config typically located here: C:\ProgramData\Qlik\Sense\Log. This file contains all the settings recognized by the Qlik Logging Service in xml format and is typically modified directly by the Qlik Logging Service when executing the update or setup commands during command line invocation.

The "Dynamic Configuration" feature introduces a new setting: CentralizedLoggingConfigurationNotificationEnabled, its default value is "False". This setting is always dynamic, meaning that changing its value by directly modifying the configuration file and saving it will cause the Qlik Logging Service to read and use its new value. When this setting is set to "False", the default value, changes to other settings in the configuration file are not detected by the Qlik Logging Service and will require a service restart to take effect. When the setting is set to "True", changes to any of the settings below will be detected and applied immediately by the Qlik Logging Service after saving the configuration file.

Qlogs statistics

The Qlik Logging Service periodically collects statistics about the Qlogs database, which is the database used for all logging entries. How often these statistics are collected is controlled by the CentralizedLoggingDBStatsCheckPeriod setting. Setting its value to zero(0) disables collection and its default value is ten (10) minutes expressed in milliseconds (10 * 60 * 1000 = 600000). The information collected contains record counts and size information and can be viewed with the following query:

SELECT * FROM public.view_db_stats;

Windows Event Log integration

Integration with the Windows Event Log is a new feature that allows the Qlik Logging Service to collect information about configured Windows events. The set of events monitored by the Qlik Logging Service is set via the CentralizedLoggingPlatformEvents setting. Clearing its value disables the feature. Its default value is:

<add key="CentralizedLoggingPlatformEvents" value="System:1074,6013,36888,36874,2013,7031,4202|Application, Engine:300|Application, .NET Runtime:1026|Application, PostgreSQL:0" />

This is an explanation of the events configured by default.

Additional events may be monitor by modifying the value of the CentralizedLoggingPlatformEvents setting as follows:

As shown above, the Event Log is one (1), the Event Source is two (2) and the Event ID is three (3). To capture the event highlighted above, the value "Application, Perflib: 1008" would need to be appended to the CentralizedLoggingPlatformEvents setting as shown below:

<add key="CentralizedLoggingPlatformEvents" value="System:1074,6013,36888,36874,2013,7031,4202|Application, Engine:300|Application, .NET Runtime:1026|Application, PostgreSQL:0|Application, Perflib: 1008" />

 

Notice that the Event Source is optional, but useful when multiple event sources may use the same event ID. For the example above, omitting the event source "Perflib" would have resulted in the monitoring of event ID 1008 for ALL applications. Multiple event IDs may be monitored for the same Event Log and Event Source and they are separated by commas "," and multiple Event Log and Event Sources may be monitored and they are separated by the bar "|" character.

Any Windows events captured by the Qlik Logging Service may be viewed by executing the following query against the Qlogs database:

SELECT * FROM public.view_platform_events;

Windows Performance Monitor integration

The Qlik Logging Service integrates with the Windows Performance Monitor, it periodically queries the system for configured metrics. The integration is divided into two sections, one that monitors configured platform metrics and one that monitors configured processes.

Platform metrics

The period used to monitor platform metrics is controlled by the CentralizedLoggingPlatformPerformanceCheckPeriod setting. A value of zero (0) disables this functionality and its default value is five (5) minutes expressed in milliseconds (5 * 60 * 1000 = 300000).

The metrics or performance counters collected are configured in the setting CentralizedLoggingPlatformPerformanceCounters and its default value is:

Default value table

The format of the CentralizedLoggingPlatformPerformanceCounters setting value follows the naming and structure used by the Windows Performance Monitor as shown below:

Additional performance counters may be added by modifying the value of the CentralizedLoggingPlatformPerformanceCounters setting. The value of the setting uses the following format:

• Each performance counter is separated by the bar "|" character.
• Each performance counter is made up of four fields separated by the semicolon ";" character and whitespace can be added freely to improve readability.
  • The first field is optional and corresponds to the name that will be used to store the performance counter in the database. If the field is not provided, the name of the counter is used instead.
  • The second field is the counter category and corresponds to one (1) on the image above.
  • The third field is the counter and corresponds to two (2) on the image above.
  • The fourth field is the counter instance and corresponds to three (3) on the image above, this field is optional and when set to the wildcard character "*" or not provided all instances available are collected.

To add the counter shown above, the following would need to be appended to the CentralizedLoggingPlatformPerformanceCounters setting: "| Test1 ; ICMP ; Messages/sec ; *". The performance counter will now be collected based on the period set by the CentralizedLoggingPlatformPerformanceCheckPeriod setting and can be retrieved from the database using the following query against the Qlogs database:

Platform metrics may be viewed by executing the following query against the Qlogs database:

SELECT * FROM public.view_platform_metrics;

Process metrics

The period used to monitor process metrics is controlled by the CentralizedLoggingProcessPerformanceCheckPeriod setting. A value of zero(0) disables this functionality and its default value is five (5) minutes expressed in milliseconds (5 * 60 * 1000 = 300000).

Performance metrics are collected for the processes configured via the CentralizedLoggingProcessPerformance setting and its default value is:

The format of the setting also follows the naming and structure used by the Windows Performance Monitor as shown below:

All performance counters available are collected for each process listed with each process separated by the bar "|" character. To collect metrics for additional processes, the "simple name" of the process would need to be appended to the CentralizedLoggingProcessPerformance setting, for example to add chrome as shown above, the string "| chrome" would be appended to the CentralizedLoggingProcessPerformance setting.

Process metrics may be viewed by executing the following query against the QLogs database:

SELECT * FROM public.view_process_metrics;

Export

The Qlik Logging Service is able to export the log data it manages. This complementary feature will make it easier to share log data with Qlik's tech support. The export command is available when using the Qlik Logging Service as a command line tool.

Usage: Qlik.Logging.Service.exe <export> [[export_options]]

Although the exported data is not encrypted, it is in binary format due to compression and not readable by tools other than the Qlik Logging Service.

Export

The "export" command will copy all log data managed by the Qlik Logging Service to a destination. The following command options can be used to control what gets exported and where to:

--output_file or -o (Required): Identifies the location where exported data should be persisted. The location can be a disk file or the address of a Qlik Logging Service that is listening for exported data. When specifying a file, the export process will create or replace the target file. When specifying a waiting service the format is: <hostname or host ip address>[: port], that is the name of the host or its ip address optionally followed by a colon (":") and the port number used by the listening host. If no port is specified, the default 7081 will be used.

--hostname or -h: Limits the exported data to the subset that originated from the specified host. The comparison is case insensitive.

--level or -l: Limits the exported data to the subset that is equal to the specified level. The comparison is case insensitive.

--to_timestamp: Limits the exported data to the subset created before the specified date. See "Date Format" section below for more details.

--from_timestamp: Limits the exported data to the subset created after the specified date. See "Date Format" section below for more details.

--process_name: Limits the exported data to the subset that originated from the specified process. The comparison is case insensitive.

Examples

These commands where tested from a "Command Prompt" without elevated permissions.

To export all contents to a file:

To export all content originating from host "Node1" between 09/01/2018 and 09/02/2018 at 10:00PM to a waiting network reachable host listening on port 80:

Date format

The format expected is "YYYY-MM-DD [[HH:MM:SS]]", the date portion is required while the time portion is optional and when omitted defaults to "00:00:00".

• Date
  • YYYY: year, 4 digits
  • MM: month, 2 digits, 01 through 12
  • DD: day of the month, 2 digits, 01 through 31
• Time
  • HH: hour, 2 digits, 00 through 23
  • MM: minute, 2 digits, 00 through 59
  • SS: second, 2 digits, 00 through 59

Technical notes

Unicode null character

Processes using the Qlik Logging Service may under very rare circumstances log entries containing the Unicode character NULL (\u0000). This character is problematic for many tools as it is most commonly used to denote the end of a string. When a Unicode character has been inserted into the QLogs database, you may see error messages like this one from PostgreSQL:

ERROR: unsupported Unicode escape sequence

DETAIL: \u0000 cannot be converted to text.

CONTEXT: JSON data, line 1: {"Message":...

STATEMENT: fetch 200 in "SQL_CUR4"

Executing the following script against the Qlogs database will replace all occurrences of the Unicode NULL character "\u0000" with the string "<UNICODE_NULL>"

Configuration file reset

If for some reason the Qlik Logging Service configuration file QlikCentralizedLogging.config becomes corrupt or in some way unreadable, it can be deleted and a new file with default values will be created the next time the service is started. The same is true for individual settings, removing a setting from the configuration file will trigger its replacement with default values during the next service start sequence.

Multi-node configuration

In a multi-node environment, the usually runs on all nodes.

In a multi-node environment, the Qlik Logging Service usually runs on all nodes. With the default configuration, all nodes will be executing database management functions. This is not a problem, but a more efficient configuration would designate a single node as the one in charge of maintaining the database. For example, on a three (3) node deployment, the Qlik Logging Service running on each node could be configured as follows:

Central node and rim node 1

<!-- Centralized Logging Configuration -->

 

<!-- DB Size Management disabled -->

<add key="CentralizedLoggingDBSizeCheckPeriod" value="0" />

 

<!-- Archive Management disabled -->

<add key="CentralizedLoggingDBArchiveCheckPeriod" value="0" />

 

<!-- Purge Management disabled -->

<add key="CentralizedLoggingDBPurgeCheckPeriod" value="0" />

 

<!-- DB Stats Collection disabled -->

<add key="CentralizedLoggingDBStatsCheckPeriod" value="0" />

 

<!-- Platform Performance Collection enabled (1 minute) -->

<add key="CentralizedLoggingPlatformPerformanceCheckPeriod" value="60000" />

 

<!-- Process Performance Collection enabled (1 minute) -->

<add key="CentralizedLoggingProcessPerformanceCheckPeriod" value="60000" />

Rim node 2 (Database management)

<!-- Centralized Logging Configuration -->

 

<!-- DB Size Management enabled (5 minutes, 6GB max size) -->

<add key="CentralizedLoggingDBSizeCheckPeriod" value="300000" />

<add key="MaximumDatabaseSizeInGB" value="6" />

 

<!-- Archive Management enabled (60 minutes) -->

<add key="CentralizedLoggingDBArchiveCheckPeriod" value="3600000" />

 

<!-- Purge Management enabled (60 minutes) -->

<add key="CentralizedLoggingDBPurgeCheckPeriod" value="3600000" />

 

<!-- DB Stats Collection enabled (5 minutes) -->

<add key="CentralizedLoggingDBStatsCheckPeriod" value="300000" />

 

<!-- Platform Performance Collection enabled (1 minute) -->

<add key="CentralizedLoggingPlatformPerformanceCheckPeriod" value="60000" />

 

<!-- Process Performance Collection enabled (1 minute) -->

<add key="CentralizedLoggingProcessPerformanceCheckPeriod" value="60000" />