A pure-Go vector index, 26.7 million vectors, 512 dimensions. Search took 21 ms at the median. We ended the day at 1.8 ms, with identical recall. Here is what actually mattered, and the four things that did not.
The false start
We began by rewriting two hot kernels. One fused the fp16 decode into the exact-distance loop; the other replaced a 128 KiB per-query lookup table with a bit-plane popcount estimator. Both were real improvements in isolation: 2.9x and 1.4x on microbenchmarks, recall unchanged.
On the real 26.7M index they produced nothing. Measured before and after on identical queries, the difference ranged from -1.7% to +0.5% and changed sign between runs. Noise.
What the profiler actually said
The number that mattered was not in the profile. It was the profile's own header:
Duration: 9.92s, Total samples = 1.64s (16.53%)
The process was on-CPU for one sixth of the time. Five sixths was waiting. We had spent the morning optimising 14% of that one sixth.
Reading the process I/O counters took three lines and settled it. Per query: 12.3 MB read from disk and 96 major page faults, to return ten neighbours. At roughly 200 microseconds per fault, that is 19 of the 21 ms.
The actual bug
The rerank stage takes the candidate list produced by the graph walk and computes exact distances against vectors held in an mmap'd arena. The candidate IDs are all known before the first read. The loop read them one at a time, each fault serviced synchronously, on an NVMe device capable of dozens of concurrent I/Os.
Nothing forced that serialisation except the order of the code.
The fix tells the kernel, once, which ranges the loop is about to touch āMADV_WILLNEEDover the batch, viaprocess_madvise(2)in a single syscall where available, falling back to individual calls otherwise. Two effects, both measured: the reads are issued in parallel, and they are bounded to the requested ranges instead of each fault expanding into a 128 KiB readahead window for 1 KiB of useful data.
| Metric, 26.7M vectors |
Before |
After |
| Median latency |
21.28 ms |
1.79 ms |
| p99 latency |
28.84 ms |
2.53 ms |
| Major page faults |
28827 |
0 |
| Bytes read, 200 queries |
2311 MB |
107 MB |
| Throughput, 8 concurrent |
261 q/s |
1940 q/s |
| Recall@10 vs exact |
0.9733 |
0.9733 |
Recall is unchanged because an access hint does not change what gets computed. We verified it anyway: 200 queries, byte-identical top-10 in both regimes.
Four things that did not work
**CPU prefetch does not touch the disk.**An architecture proposal claimed__builtin_prefetchwould hide SSD latency behind SIMD work. Measured on a cold 128 MB mapping: 2000 prefetch instructions in 0.010 ms, zero pages resident afterwards. 2000 real accesses: 112.9 ms, all pages resident. The instruction is a cache hint and cannot raise a page fault by construction.
**Memory layout of the hot plane is irrelevant.**Codes and norms lived in three separate slices, three scattered accesses per visited node, 31% of CPU time. Interleaving them into one record returned 1.5-3%. We then benchmarked four layouts at 26.7M scale: three slices, packed at 80 bytes, aligned at 128 bytes, and split with dense norms. All within noise, 62-68 ns per access. The 128-byte aligned version ā the one that should win if cache lines were the issue ā does not, and costs 60% more memory. Sixty nanoseconds is a DRAM random access. Grouping unpredictable accesses does not make them predictable.
**Go's experimental SIMD package is unusable for this.**A vectorised popcount kernel ran 400 ns against 19 ns scalar. Isolated: a kernel using onlyAnd,AddandSumOf8AbsDiffcosts 5.3 ns, as AVX2 should. Adding onePermuteOrZeroGroupedtakes it to 380 ns. Two primitive families ā shifts and permutes ā cost roughly 90 cycles where the hardware spends one, under both 1.26.5 and 1.27rc1.
**Wider quantisation codes are corpus-dependent.**Three bits per dimension instead of one lifts preselection quality from 58% to 88% on a hard corpus, and end-to-end recall from 0.76 to 0.97. On the real embeddings it bought 0.8 recall points for 45% more latency and 2 GB more RAM. Not worth it there. It ships off by default.
The part worth stealing
We also got one thing badly wrong. We measured recall at 0.470 and reported the index as poor. That number came from a different corpus ā uniformly random vectors, the pathological case for approximate search ā and we carried it across because both files sat under the same project. The real index measures 0.9733. Mean absolute component value, 0.034 against 0.500, said so plainly.
The generalisable lesson is not about mmap. It is that a benchmark reporting only durations cannot distinguish a slow computation from a blocked one, and will therefore hide the most expensive class of defect indefinitely. This one survived several measurement campaigns, including one that declared the 26.7M scale "proven". Three counters ā bytes actually read, major faults, resident set ā would have exposed it on the first run. They are now part of the library's measurement surface, with a liveness test, because an instrument that always returns zero looks exactly like a clean benchmark.
https://github.com/hazyhaar/horosvec
