Qlik Logging Service – configuration and integration

This topic describes the configuration and integration features that are available for the Qlik Logging Service. The features should be considered "Advanced Features" to be used in the context of troubleshooting issues with the assistance of Qlik technical support.

Dynamic configuration

Most of the functionality provided by the Qlik Logging Service is controlled by the configuration file: QlikCentralizedLogging.config typically located here: C:\ProgramData\Qlik\Sense\Log. This file contains all the settings recognized by the Qlik Logging Service in xml format and is typically modified directly by the Qlik Logging Service when executing the update or setup commands during command line invocation.

The "Dynamic Configuration" feature introduces a new setting: CentralizedLoggingConfigurationNotificationEnabled, its default value is "False". This setting is always dynamic, meaning that changing its value by directly modifying the configuration file and saving it will cause the Qlik Logging Service to read and use its new value. When this setting is set to "False", the default value, changes to other settings in the configuration file are not detected by the Qlik Logging Service and will require a service restart to take effect. When the setting is set to "True", changes to any of the settings below will be detected and applied immediately by the Qlik Logging Service after saving the configuration file.

Key Description Range
CentralizedLoggingLogMaxFileSizeMb The logging service logs to a file, this setting controls the maximum size in MB of the log file before it is rolled over. 1...
CentralizedLoggingLogEnabled Turns on/off logging to file by the logging service. True, False
CentralizedLoggingEnabled Enables or disables the logging service functionality. When True, the service accepts incoming log entries, when False, the service continues to run, but it does not offer any logging functionality. True, False
CentralizedLoggingServerBufferSize Sets the size in log entries of the internal buffer. Writes to the database occur when the internal buffer is full. For example if set to 10, writes to the database will occur when 10 log entries are received. 1...
CentralizedLoggingServerTimeoutSeconds Sets the timeout in seconds required to trigger a database write. On an idle system log entries may not accumulate often. This threshold will force a database write even if the internal buffer is not full. 1...
CentralizedLoggingLogLevel Controls the verbosity level of the log. Off, Fatal, Error, Warn, Info, Debug, All
MaximumDatabaseSizeInGB

Sets the maximum size for the database in GB, when the size of the database exceeds this number, the logging service will trim/remove entries from the database until the size of the database goes below the maximum set.

Opmerking: A value less than 2 disables the functionality.
0...
CentralizedLoggingConfigurationNotificationEnabled Enables or disables dynamic configuration settings. When set to True, changes to a dynamic configuration setting take effect immediately without requiring a service re-start. When False, a re-start is required for the new setting value to take effect. Notice that this setting is always dynamic. True, False
CentralizedLoggingDBSizeCheckPeriod

Sets how often to check the size of the database. The quantity is expressed in milliseconds.

Opmerking: A value of zero (0) disables the functionality, 600000 (ten minutes) is the minimum value accepted. Values less than that are automatically set to that minimum and 2147483646 (~596 hours) is the maximum.
0, 600000..2147483646
CentralizedLoggingDBArchiveCheckPeriod

Sets how often to check for the need to archive log entries, the quantity is expressed in milliseconds.

Opmerking: A value of zero (0) disables the functionality, 600000 (ten minutes) is the minimum value accepted. Values less than that are automatically set to that minimum and 2147483646 (~596 hours) is the maximum.
0, 600000..2147483646
CentralizedLoggingDBPurgeCheckPeriod

Sets how often to check for the need to purge archived entries, the quantity is expressed in milliseconds.

Opmerking: A value of zero (0) disables the functionality, 600000 (ten minutes) is the minimum value accepted. Values less than that are automatically set to that minimum and 2147483646 (~596 hours) is the maximum.
0, 600000..2147483646
CentralizedLoggingDBStatsCheckPeriod

Sets how often to retrieve database statistics. The quantity is expressed in milliseconds.

Opmerking: A value of zero(0) disables the functionality, 300000 (five minutes) is the minimum value accepted. Values less than that are automatically set to that minimum and 2147483646 (~596 hours) is the maximum.
0, 300000..2147483646
CentralizedLoggingPlatformPerformanceCheckPeriod

Sets how often to retrieve platform performance metrics, the quantity is expressed in milliseconds.

Opmerking: A value of zero (0) disables the functionality, 1000 (one second) is the minimum value accepted. Values less than that are automatically set to that minimum and 2147483646 (~596 hours) is the maximum.
0, 1000..2147483646
CentralizedLoggingProcessPerformanceCheckPeriod

Sets how often to retrieve process performance metrics, the quantity is expressed in milliseconds.

Opmerking: A value of zero (0) disables the functionality, 1000 (one second) is the minimum value accepted. Values less than that are automatically set to that minimum and 2147483646 (~596 hours) is the maximum.
0, 1000..2147483646
CentralizedLoggingPlatformPerformanceCounters

Set of platform performance counters to collect every CentralizedLoggingPlatformPerformanceCheckPeriod period. The format expected is:

Database payload key; Counter Category; Counter Name [;] [Counter Instance] |

Database payload key...

 
CentralizedLoggingProcessPerformance

Set of processes to collect performance counter for every CentralizedLoggingProcessPerformanceCheckPeriod period.

The format expected is:

Process Name | Process Name | ...

Process Name is the simple name for the process for example the default value of this setting is: engine|proxy|repository|scheduler

 
CentralizedLoggingPlatformEvents

Set of platform events to monitor The format expected is:

EventLogX, [EventSourceX]:EventID0, EventID1, ..., EventIDn|

EventLogY, [EventSourceY]:EventID0, ..., EventIDn

 

Qlogs statistics

The Qlik Logging Service periodically collects statistics about the Qlogs database, which is the database used for all logging entries. How often these statistics are collected is controlled by the CentralizedLoggingDBStatsCheckPeriod setting. Setting its value to zero(0) disables collection and its default value is ten (10) minutes expressed in milliseconds (10 * 60 * 1000 = 600000). The information collected contains record counts and size information and can be viewed with the following query:

SELECT * FROM public.view_db_stats;

Windows Event Log integration

Integration with the Windows Event Log is a new feature that allows the Qlik Logging Service to collect information about configured Windows events. The set of events monitored by the Qlik Logging Service is set via the CentralizedLoggingPlatformEvents setting. Clearing its value disables the feature. Its default value is:

<add key="CentralizedLoggingPlatformEvents" value="System:1074,6013,36888,36874,2013,7031,4202|Application, Engine:300|Application, .NET Runtime:1026|Application, PostgreSQL:0" />

This is an explanation of the events configured by default.

Event log Event source Event ID Description
System   1074 User initiated system re-start
System   6013 System uptime
System   36888 Schannel (TLS protocol) errors
System   36874 Schannel (TLS protocol) errors
System   2013 Low disk space
System   7031 Service terminated unexpectedly
System   4202 TCP/IP network configuration issue
Application Engine 300 Engine caught unexpected exception
Application PostgreSQL 0 PostgreSQL error
Application .NET Runtime 1026 .NET process terminated

Additional events may be monitor by modifying the value of the CentralizedLoggingPlatformEvents setting as follows:

As shown above, the Event Log is one (1), the Event Source is two (2) and the Event ID is three (3). To capture the event highlighted above, the value "Application, Perflib: 1008" would need to be appended to the CentralizedLoggingPlatformEvents setting as shown below:

<add key="CentralizedLoggingPlatformEvents" value="System:1074,6013,36888,36874,2013,7031,4202|Application, Engine:300|Application, .NET Runtime:1026|Application, PostgreSQL:0|Application, Perflib: 1008" />

 

Notice that the Event Source is optional, but useful when multiple event sources may use the same event ID. For the example above, omitting the event source "Perflib" would have resulted in the monitoring of event ID 1008 for ALL applications. Multiple event IDs may be monitored for the same Event Log and Event Source and they are separated by commas "," and multiple Event Log and Event Sources may be monitored and they are separated by the bar "|" character.

Any Windows events captured by the Qlik Logging Service may be viewed by executing the following query against the Qlogs database:

SELECT * FROM public.view_platform_events;

Windows Performance Monitor integration

The Qlik Logging Service integrates with the Windows Performance Monitor, it periodically queries the system for configured metrics. The integration is divided into two sections, one that monitors configured platform metrics and one that monitors configured processes.

Platform metrics

The period used to monitor platform metrics is controlled by the CentralizedLoggingPlatformPerformanceCheckPeriod setting. A value of zero (0) disables this functionality and its default value is five (5) minutes expressed in milliseconds (5 * 60 * 1000 = 300000).

The metrics or performance counters collected are configured in the setting CentralizedLoggingPlatformPerformanceCounters and its default value is:

<add key="CentralizedLoggingPlatformPerformanceCounters" value="CPUUtilizationPercent;Processor;% Processor Time;_Total|CPUUserUtilizationPercent;Processor;% User Time;_Total|CPUInterruptPercent;Processor;% Interrupt Time;_Total|CPUQueueLength;System;Processor Queue Length|DiskFreeSpacePercent;LogicalDisk;% Free Space;_Total|DiskIdleTimePercent;PhysicalDisk;% Idle Time;_Total|DiskTimePerReadSeconds;PhysicalDisk;Avg. Disk sec/Read;_Total|DiskTimePerWriteSeconds;PhysicalDisk;Avg. Disk sec/Write;_Total|DiskIOQueueLength;PhysicalDisk;Avg. Disk Queue Length;_Total|MemoryCacheBytes;Memory;Cache Bytes|MemoryCommittedBytesInUsePercent;Memory;% Committed Bytes In Use|MemoryAvailableMBytes;Memory;Available MBytes|MemoryFreePageEntries;Memory;Free System Page Table Entries|MemoryPoolNonPagedBytes;Memory;Pool Nonpaged Bytes|MemoryPoolPagedBytes;Memory;Pool Paged Bytes|MemoryPagesPerSecond;Memory;Pages/sec|NetworkBytesPerSecond;Network Interface;Bytes Total/sec;*|NetworkOutputQueueLength;Network Interface;Output Queue Length;*|" />

Default value table

Database field Counter Category Counter name Counter instance
CPUInterruptPercent Processor % Interrupt Time _Total
CPUQueueLength System Processor Queue Length  
CPUUserUtilizationPercent Processor % User Time _Total
CPUUtilizationPercent Processor % Processor Time _Total
DiskFreeSpacePercent LogicalDisk % Free Space _Total
DiskIdleTimePercent PhysicalDisk % Idle Time _Total
DiskIOQueueLength PhysicalDisk Avg. Disk Queue Length _Total
DiskTimePerReadSeconds PhysicalDisk Avg. Disk sec/Read _Total
DiskTimePerWriteSeconds PhysicalDisk Avg. Disk sec/Write _Total
MemoryAvailableMBytes Memory Available MBytes  
MemoryCacheBytes Memory Cache Bytes  
MemoryCommittedBytesInUsePercent Memory % Committed Bytes In Use  
MemoryFreePageEntries Memory Free System Page Table Entries  
MemoryPagesPerSecond Memory Pages/sec  
MemoryPoolNonPagedBytes Memory Pool Nonpaged Bytes  
NetworkBytesPerSecond Network Interface Bytes Total/sec *
NetworkOutputQueueLength Network Interface Output Queue Length *

The format of the CentralizedLoggingPlatformPerformanceCounters setting value follows the naming and structure used by the Windows Performance Monitor as shown below:

Additional performance counters may be added by modifying the value of the CentralizedLoggingPlatformPerformanceCounters setting. The value of the setting uses the following format:

  • Each performance counter is separated by the bar "|" character.
  • Each performance counter is made up of four fields separated by the semicolon ";" character and whitespace can be added freely to improve readability.
    • The first field is optional and corresponds to the name that will be used to store the performance counter in the database. If the field is not provided, the name of the counter is used instead.
    • The second field is the counter category and corresponds to one (1) on the image above.
    • The third field is the counter and corresponds to two (2) on the image above.
    • The fourth field is the counter instance and corresponds to three (3) on the image above, this field is optional and when set to the wildcard character "*" or not provided all instances available are collected.

To add the counter shown above, the following would need to be appended to the CentralizedLoggingPlatformPerformanceCounters setting: "| Test1 ; ICMP ; Messages/sec ; *". The performance counter will now be collected based on the period set by the CentralizedLoggingPlatformPerformanceCheckPeriod setting and can be retrieved from the database using the following query against the Qlogs database:

SELECT BTRIM(e.payload->>'Test1', '"')::INTEGER AS icmp_msg_per_sec FROM public.log_entries e WHERE e.logger = 'Qlik.Logging.Service.Platform.Performance'

Platform metrics may be viewed by executing the following query against the Qlogs database:

SELECT * FROM public.view_platform_metrics;

Process metrics

The period used to monitor process metrics is controlled by the CentralizedLoggingProcessPerformanceCheckPeriod setting. A value of zero(0) disables this functionality and its default value is five (5) minutes expressed in milliseconds (5 * 60 * 1000 = 300000).

Performance metrics are collected for the processes configured via the CentralizedLoggingProcessPerformance setting and its default value is:

<add key="CentralizedLoggingProcessPerformance" value="engine|proxy|repository|scheduler" />

The format of the setting also follows the naming and structure used by the Windows Performance Monitor as shown below:

All performance counters available are collected for each process listed with each process separated by the bar "|" character. To collect metrics for additional processes, the "simple name" of the process would need to be appended to the CentralizedLoggingProcessPerformance setting, for example to add chrome as shown above, the string "| chrome" would be appended to the CentralizedLoggingProcessPerformance setting.

Process metrics may be viewed by executing the following query against the QLogs database:

SELECT * FROM public.view_process_metrics;

Export

The Qlik Logging Service is able to export the log data it manages. This complementary feature will make it easier to share log data with Qlik's tech support. The export command is available when using the Qlik Logging Service as a command line tool.

Usage: Qlik.Logging.Service.exe <export> [[export_options]]

Although the exported data is not encrypted, it is in binary format due to compression and not readable by tools other than the Qlik Logging Service.

Export

The "export" command will copy all log data managed by the Qlik Logging Service to a destination. The following command options can be used to control what gets exported and where to:

--output_file or -o (Required): Identifies the location where exported data should be persisted. The location can be a disk file or the address of a Qlik Logging Service that is listening for exported data. When specifying a file, the export process will create or replace the target file. When specifying a waiting service the format is: <hostname or host ip address>[: port], that is the name of the host or its ip address optionally followed by a colon (":") and the port number used by the listening host. If no port is specified, the default 7081 will be used.

--hostname or -h: Limits the exported data to the subset that originated from the specified host. The comparison is case insensitive.

--level or -l: Limits the exported data to the subset that is equal to the specified level. The comparison is case insensitive.

--to_timestamp: Limits the exported data to the subset created before the specified date. See "Date Format" section below for more details.

--from_timestamp: Limits the exported data to the subset created after the specified date. See "Date Format" section below for more details.

--process_name: Limits the exported data to the subset that originated from the specified process. The comparison is case insensitive.

Examples

These commands where tested from a "Command Prompt" without elevated permissions.

To export all contents to a file:

Qlik.Logging.Service export --output_file C:\Temp\qlogs.bin

To export all content originating from host "Node1" between 09/01/2018 and 09/02/2018 at 10:00PM to a waiting network reachable host listening on port 80:

Qlik.Logging.Service export --from_timestamp "2018-09-01" --to_timestamp "2018-09-02 22:00:00" --hostname node1 --otuput_file somehost.domain.com:80

Date format

The format expected is "YYYY-MM-DD [[HH:MM:SS]]", the date portion is required while the time portion is optional and when omitted defaults to "00:00:00".

  • Date
    • YYYY: year, 4 digits
    • MM: month, 2 digits, 01 through 12
    • DD: day of the month, 2 digits, 01 through 31
  • Time
    • HH: hour, 2 digits, 00 through 23
    • MM: minute, 2 digits, 00 through 59
    • SS: second, 2 digits, 00 through 59

Technical notes

Unicode null character

Processes using the Qlik Logging Service may under very rare circumstances log entries containing the Unicode character NULL (\u0000). This character is problematic for many tools as it is most commonly used to denote the end of a string. When a Unicode character has been inserted into the QLogs database, you may see error messages like this one from PostgreSQL:

ERROR: unsupported Unicode escape sequence

DETAIL: \u0000 cannot be converted to text.

CONTEXT: JSON data, line 1: {"Message":...

STATEMENT: fetch 200 in "SQL_CUR4"

Executing the following script against the Qlogs database will replace all occurrences of the Unicode NULL character "\u0000" with the string "<UNICODE_NULL>"

SET statement_timeout = 0; SET lock_timeout = 0; SET client_encoding = 'UTF8'; SET standard_conforming_strings = on; SET check_function_bodies = false; SET client_min_messages = warning; CREATE OR REPLACE FUNCTION public.validate_payload(IN payload JSON) RETURNS TEXT AS $$ BEGIN RETURN payload->>'XXXXX'; EXCEPTION WHEN OTHERS THEN RETURN 'XXXXX'; END $$ LANGUAGE plpgsql; DO $$ BEGIN UPDATE public.log_entries SET payload = REGEXP_REPLACE(payload::TEXT, '\\u0000', '<UNICODE_NULL>', 'g')::JSON WHERE id IN ( SELECT d.id FROM ( SELECT id, validate_payload(payload) AS payload FROM public.log_entries ) AS d WHERE d.payload = 'XXXXX' ); UPDATE public.archive_entries SET payload = REGEXP_REPLACE(payload::TEXT, '\\u0000', '<UNICODE_NULL>', 'g')::JSON WHERE id IN ( SELECT d.id FROM ( SELECT id, validate_payload(payload) AS payload FROM public.archive_entries ) AS d WHERE d.payload = 'XXXXX' ); END $$ LANGUAGE plpgsql; DROP FUNCTION IF EXISTS public.validate_payload(IN payload JSON);

Configuration file reset

If for some reason the Qlik Logging Service configuration file QlikCentralizedLogging.config becomes corrupt or in some way unreadable, it can be deleted and a new file with default values will be created the next time the service is started. The same is true for individual settings, removing a setting from the configuration file will trigger its replacement with default values during the next service start sequence.

Multi-node configuration

In a multi-node environment, the usually runs on all nodes.

In a multi-node environment, the Qlik Logging Service usually runs on all nodes. With the default configuration, all nodes will be executing database management functions. This is not a problem, but a more efficient configuration would designate a single node as the one in charge of maintaining the database. For example, on a three (3) node deployment, the Qlik Logging Service running on each node could be configured as follows:

Central node and rim node 1

<!-- Centralized Logging Configuration -->

 

<!-- DB Size Management disabled -->

<add key="CentralizedLoggingDBSizeCheckPeriod" value="0" />

 

<!-- Archive Management disabled -->

<add key="CentralizedLoggingDBArchiveCheckPeriod" value="0" />

 

<!-- Purge Management disabled -->

<add key="CentralizedLoggingDBPurgeCheckPeriod" value="0" />

 

<!-- DB Stats Collection disabled -->

<add key="CentralizedLoggingDBStatsCheckPeriod" value="0" />

 

<!-- Platform Performance Collection enabled (1 minute) -->

<add key="CentralizedLoggingPlatformPerformanceCheckPeriod" value="60000" />

 

<!-- Process Performance Collection enabled (1 minute) -->

<add key="CentralizedLoggingProcessPerformanceCheckPeriod" value="60000" />

Rim node 2 (Database management)

<!-- Centralized Logging Configuration -->

 

<!-- DB Size Management enabled (5 minutes, 6GB max size) -->

<add key="CentralizedLoggingDBSizeCheckPeriod" value="300000" />

<add key="MaximumDatabaseSizeInGB" value="6" />

 

<!-- Archive Management enabled (60 minutes) -->

<add key="CentralizedLoggingDBArchiveCheckPeriod" value="3600000" />

 

<!-- Purge Management enabled (60 minutes) -->

<add key="CentralizedLoggingDBPurgeCheckPeriod" value="3600000" />

 

<!-- DB Stats Collection enabled (5 minutes) -->

<add key="CentralizedLoggingDBStatsCheckPeriod" value="300000" />

 

<!-- Platform Performance Collection enabled (1 minute) -->

<add key="CentralizedLoggingPlatformPerformanceCheckPeriod" value="60000" />

 

<!-- Process Performance Collection enabled (1 minute) -->

<add key="CentralizedLoggingProcessPerformanceCheckPeriod" value="60000" />