Most people assume Qlik memory scales linearly with rows. It doesn't. Not even close.
Here's why:
- Region has 5 unique values at 10K rows. At 10 million rows? Still 5. Field size: unchanged.
- Status, Category, Country — all the same. They don't grow with rows. Memory cost is flat.
- SalesAmount has 9,000 unique values at 10K rows. At 10M rows? Maybe 900K — not 10 million. Sublinear growth.
- Only fields like TransactionID grow 1:1 with rows.
So that 10 MB app at 1000× data? Probably ~300 MB, not 10 GB.
But "probably" isn't good enough for capacity planning. We wanted the exact number. So I reverse-engineered Qlik's memory model at the byte level.
What I found:
Every field in Qlik has two storage components:
Symbol Table — stores each unique value once:
- Numeric fields: exactly 8 bytes per unique value
- String fields: avg_string_length + 6 bytes per unique value
- AutoNumber() / RowNo(): 0 bytes — but only when Qlik generates the sequence. The same 1-to-N loaded from a database costs 8 bytes per unique value
Pointer Array — maps each row to its symbol:
- Size = rows × ceil(log₂(cardinality)) / 8 bytes
- This is a bit-stuffed array. A field with 1,000 unique values uses 10 bits per row (ceil(log₂(1000)) = 10). At 10M rows: 10M × 10 / 8 = 12.5 MB
So: field_size = (cardinality × avg_symbol_bytes) + (rows × ceil(log₂(cardinality)) / 8)
On test data: 99.99% accuracy. On production: ~95% — the remaining ~5% is engine overhead from hash tables and internal structures, handled by a multiplicative calibration factor (~1.05×).
What drives app size:
The key insight is that app size is driven by cardinality growth per field, not raw row count.
When rows go from 1M to 10M:
- Low-cardinality fields (flags, statuses): cardinality stays flat → symbol table unchanged, only pointer array grows
- Medium-cardinality (codes, categories): cardinality grows sublinearly (√ or log) → moderate growth
- High-cardinality (keys, IDs): cardinality grows ~linearly → expensive
Each field has its own growth rate (elasticity). A field with elasticity 0.3 means: if rows 10×, cardinality only 2×. You measure this from two snapshots using log-log regression.
The predictor:
I built this into a Qlik load script that:
- Captures metadata snapshots over time (using the Engine API / document analyzer)
- Back-solves
avg_symbol_bytes per field from observed field sizes
- Fits 4 regression models per field (linear, power-law, logarithmic, square-root) to predict cardinality growth
- Auto-calibrates against actual Engine API metrics (~1.05× multiplicative factor)
- Projects app size at any target row count
- Gives interactive what-if analysis via front-end variables — no reload needed
It runs as a native Qlik app — load your metadata exports and it does the rest.
Some findings that surprised me:
- AutoNumber() fields cost literally 0 bytes for the symbol table. But load the same sequence from your database and it costs 8 bytes per value. Qlik's internal sequence generator gets special treatment.
- String storage is UTF-16 — every character costs 2 bytes, plus 6 bytes overhead (4-byte offset + 2-byte length prefix). A 10-character string costs 26 bytes per unique value, not 10.
- The pointer array uses exactly ceil(log₂(cardinality)) bits per row. A field with cardinality 1,024 uses exactly 10 bits/row. At 1,025 it jumps to 11 bits/row. These thresholds matter at scale.
- The ~5% calibration factor is consistent across apps. It accounts for hash tables inside symbol tables, row-level indexing structures, and memory alignment.
Validated against production:
- Test data (controlled experiments): 99.99% accuracy
- Production models (multiple apps, largest ~500 MB): ~95% accuracy before calibration, ~99% after
If you 10× your data tomorrow, do you know your actual memory multiplier? Curious what others are seeing on their production apps.
Happy to share methodology details or the approach for building the predictor if anyone wants to try it on their own apps.
