Best practices for QVD architecture when migrating to Qlik Cloud

Qlik Cloud supports large files, and native DataFiles storage can handle large QVD files. Architectural considerations differ, however, when using large files in Qlik Cloud compared to Qlik Sense Enterprise on Windows.

QVD load performance differs between Qlik Sense Enterprise on Windows and Qlik Cloud:

• In Qlik Sense Enterprise on Windows, Qlik Engine reads the QVD directly from the local disk storage. When the files are stored on the local server, there is also minimal latency when loading data.

• In Qlik Cloud, Qlik Engine makes a network call to read the file content from web file storage. This adds load time in two ways. First, reading from web file storage over the network adds transfer latency that local disk reads avoid. Second, the encryption required for compliance certifications, such as SOC 2/3 and HIPAA, adds processing overhead.

  Information noteFor more details on security and compliance, see Trust and Security at Qlik.

When storing QVD files in Qlik Cloud, include file read and write time in your reload strategy. Consider the following approaches when planning your QVD architecture in Qlik Cloud:

• Maintain optimal QVD sizes

• Optimize QVD load operations

For more information about working with QVD files, see:

• Working with QVD files

Maintain optimal QVD sizes

While Qlik Cloud can handle large files, keep your QVD files small enough to meet your performance requirements. There is no fixed target size, so validate performance as part of your data design and optimize further to reach your needs. The following approaches can help keep your QVD files within optimal size ranges for timely loading.

For more information about optimizing application performance with large data volumes, see:

• Large data volumes

Segment your data

Keep individual file sizes down by segmenting your data by natural dimensions such as:

• Time frame: Segment your data by time frames, such as years or months. For example, you could split a QVD containing all your data into two QVDs:

  • A QVD that contains data from the two most recent years.

  • A QVD that contains historical data older than two years.

• Category: Segment your data by logical categories, such as:

  • Countries or regions.

  • Product groups.

  • Departments.

  For example, you could split a QVD containing all your data by country or second-level administrative areas such as states or provinces.

• Aggregation level: Aggregate your data to a higher level to create a smaller dataset. For example:

  • Aggregate daily data into monthly summaries.

  • Aggregate individual customer data into regional summaries.

Reduce the cardinality of data

Reducing the cardinality of data in the QVD files improves compression and reduces QVD file size. Consider the following approaches:

• Rounding: Round high-cardinality numeric values when it makes analytical sense. This reduces the unique value count, leading to smaller datasets and faster symbol table processing.

• Splitting timestamps: Split timestamp fields into separate Date and Time fields. This significantly lowers cardinality and improves compression.

• Optimizing text fields: Text fields can have high cardinality, especially fields containing long descriptions. To optimize your QVD files, avoid including these text fields in multiple QVD files. Move them into dedicated QVD files to keep primary data QVD files lean. If a single text QVD is still not performant, segment it further.

Review QVD layer strategy

QVD files can grow over time and may no longer be optimized for their original use cases. Review your layer strategy and determine whether you can adopt a more granular structure. Using more, smaller files can limit the impact of file loading.

For example, splitting sales data into QVDs by year can speed up reload times and enable applications to read less data when they only require data points from the most recent years.

For more information about building and storing QVD layers, see:

• Loading new and updated records with incremental load

Optimize QVD load operations

When loading data from QVD files in Qlik Cloud, optimize your load operations. The following approaches can help reduce load times when reading QVD files.

Minimize storage reads

Avoid loading the same QVD files multiple times within a single load script. To reduce overhead, load the data once into a temporary table and repurpose it through RESIDENT loads.

For more information about data loading and resident tables, see:

• Data loading in Qlik Sense

Leverage optimized loads

For unoptimized scenarios such as loads with complex WHERE clauses or data transformations, consider loading the QVD in its optimized format into a temporary table first. You can then perform the necessary manipulations and filtering via a RESIDENT load into your final target table, shifting the processing burden from loading the QVD to the engine memory.

For more information about optimized and standard QVD load modes, see:

• QVD files

Alternate file storage solutions

If the governance overhead of storing your QVDs in Qlik Cloud causes read and write performance issues, consider testing alternate file storage solutions with configurable security level options.

For Qlik Cloud-supported file storage solutions, see Qlik Web Storage Provider Connectors. To minimize latency issues, ensure your alternate file storage solution is deployed as close as possible to your Qlik Cloud region's tenant.

For more information about managing data files, see:

• Managing data files

• Managing data files for tenant and analytics administrators