AI Retrieval Core

Most AI PMs treat retrieval as a black box, they spec RAG systems without understanding the latency, cost, and accuracy tradeoffs happening underneath. I built this to fix my own blind spot.

An AI PM who understands retrieval latency makes better calls when the engineering team says 'this approach won't scale.' The most useful result here was a negative one: the obvious optimizations barely helped, and knowing why is what lets you push back.

~3.2 ms multithreaded search over 100K × 768d vectors. Found the workload is memory-bandwidth bound: 10 cores gave ~1.6x, and manual SIMD lost to compiler auto-vectorization.

A CPU vector search engine in C++: brute-force dot-product similarity over 100K × 768-dimension vectors, with scalar, manual SIMD, multithreaded, and per-thread heap variants benchmarked side by side.

RAG (Retrieval-Augmented Generation) is now a standard pattern in AI product development, but most AI PMs who spec RAG systems have never built the retrieval layer themselves. They don't know what 'fast retrieval' actually costs at the systems level, or what actually limits it. I built a CPU vector search engine in C++ that simulates a production-scale embedding workload, 100,000 vectors at 768 dimensions, and benchmarked it across scalar, SIMD, and multithreaded strategies. Not to become an infrastructure engineer, but to earn credibility in architecture conversations and know when to push back.

1. 1.
   Realistic Workload: Simulate BERT-scale embedding retrieval: 100,000 vectors at 768 dimensions, scored by dot-product similarity, so the numbers reflect a real semantic search corpus rather than a toy.
2. 2.
   Correct Baseline First: Implement single-threaded brute-force search as the correctness and latency baseline before trying any optimization.
3. 3.
   Compare Optimization Strategies: Benchmark hand-written ARM NEON SIMD, multithreaded scanning across CPU cores, and per-thread Top-K heap aggregation against the baseline.
4. 4.
   Find the Real Bottleneck: Use high-resolution latency measurement to identify whether the workload is limited by compute or by memory bandwidth, because that decides which optimizations are worth paying for.

• ◆
  Baseline Before Optimization: Single-thread scalar search came in around 5.1 ms per query. Every later strategy is measured against that number, not against intuition.
• ◆
  Test the Obvious Optimization: Wrote manual NEON SIMD expecting a speedup. It ran at about 9.6 ms, slower than the baseline, because the compiler had already auto-vectorized the scalar loop.
• ◆
  Scale Across Cores: Multithreaded brute force brought latency to about 3.2 ms on 10 Apple Silicon cores. Adding per-thread Top-K heaps landed at about 3.4 ms.
• ◆
  Explain the Ceiling: Ten cores delivered only about 1.6x. The scan saturates memory bandwidth long before it runs out of compute, so more threads stop helping.

Before building this, I knew vector search was 'fast'. After building it, I know that brute-force search over 100K BERT-scale vectors runs in single-digit milliseconds on a laptop CPU, and that throwing cores at it buys far less than expected because the bottleneck is memory bandwidth, not math. That changes the conversation. If an engineering team proposes more hardware for retrieval latency, the first question is whether the workload is memory-bound. Often the better levers are a smaller index, quantized vectors, or approximate search.

Python would have been faster to write. C++ was the right choice because it forces you to think about memory layout, cache behaviour, and threading in a way that high-level languages abstract away. The finding that manual SIMD lost to the compiler, and that ten cores gave 1.6x, only shows up when you are close enough to the hardware to see it. A Python wrapper around a library would have hidden both.