Benchmarking Vector search in six database engines

We loaded 990,000 OpenAI embeddings of 1,536 dimensions into six databases that index vectors themselves, built an HNSW index in each, and measured five things: time to a queryable index, storage and memory, recall against throughput, throughput under concurrent clients, and what happens when 10% of the corpus is rewritten. Same host, same container shape, same source pins, same query set.

The build numbers spread wider than we expected. Time to a queryable index ran from 1.9 minutes to 6.7 hours for the same million vectors. Index size ran from 3.9 GiB to 15.1 GiB, and the engine with the smallest index took the longest to build it. At recall 0.99 the field closes up: four of the six are within a factor of two of each other. Under 32 clients it opens again. pgvector returns 21.7× its single-client throughput. The three InnoDB-based engines return about 1×.

This is one run on one machine. Two numbers in the concurrency table have since been withdrawn by the harness that produced them, and the reason is below. Framework, configuration and raw records: github.com/Percona-Lab/vector-bench.

The machine, and how we ran it

<

The database runs in one container and the harness in another, on a private network. One container would charge the several hundred megabytes of NumPy holding the query set to the server’s cgroup and ruin the memory numbers. The harness never configures a server. The orchestrator does, so every engine gets the same treatment, and that treatment is written into the run manifest beside the results.

All six got the same budget: a 42.3 GB InnoDB buffer pool, 12.1 GB of maintenance_work_mem for pgvector’s build, 8 I/O threads and 8 build threads. The two engines with a graph cache got 30.2 GB of it. Both default to 16 MiB, and leaving that in place would judge a vendor on a value it plainly intends you to change. The orchestrator clamped I/O and build threads down from 64, because more than a handful contend rather than help and a large value stalled writes into a populated index outright. It raised /dev/shm from 8 GB to 13 GB so pgvector’s parallel build could hold a segment the size of maintenance_work_mem. Both adjustments are in the manifest.

Recall is ann-benchmarks’ distance-threshold definition, so these numbers stay comparable with published ANN results. Ground truth for filtered queries is recomputed by brute force over the rows that satisfy each predicate. A WHERE clause changes the correct answer set, and scoring a filtered query against unfiltered neighbours scores every engine against the wrong targets.

What ran

Percona Search for MongoDB is a Technical Preview and its documentation says not for production. Its numbers use the scalar-quantized index, which MongoDB recommends above a 3 GB vector index.

Loading the data

MHNSW and VIDX update the graph on every INSERT, so their load is the index build and there is nothing separate to time. That puts them at 41 to 76 rows a second against pgvector’s 613 and Valkey’s 30,170. pgvector’s 40.9 minutes is 26.9 minutes of COPY plus 14.0 minutes of parallel build across 8 workers.

They get the smallest index for it. 3.9 GiB, where the raw float32 vectors alone are 5.7 GiB. It is the only measurement in this run where MariaDB leads, and it matters when index-resident-per-node is the binding constraint. The table is a different story: MariaDB stores 14.3 GiB for those same vectors, Percona Search 5.8 GiB.

Build time is a re-indexing window. Changing embedding model is an afternoon on pgvector and a weekend on AliSQL.

Recall against throughput

Best throughput whose measured recall clears each floor. MariaDB and AliSQL repeat a number across two columns because their fastest configuration already returns recall 0.956 to 0.989. It wins both floors. That is not a missing measurement.

MariaDB 12.3 is the fastest engine in the run at recall 0.99, at 277 QPS. It gives six times the throughput of 11.8 at that floor for 50% more build time. That is a better graph rather than faster search, and it is the case for upgrading if accuracy above 0.99 is the target.

Raising the floor from 0.90 to 0.95 costs pgvector and Valkey a third of their throughput. It costs MariaDB and AliSQL nothing, because their fastest setting is already above both floors. Neither MHNSW nor VIDX exposes a knob that trades accuracy for speed at a fixed graph degree, so there is no cheaper operating point to move to. pgvector and Valkey expose ef_construction as well as a query-time knob.

One AliSQL point is unexplained. At M = 16, throughput rises from 2.6 to 4.4 QPS between ef_search 400 and 800. More search effort cannot be faster on the same index. AliSQL also has a latency cliff: p50 is 6.6 ms at ef_search 80, 50.8 ms at 120 and 384 ms beyond, reproduced across three runs, two corpus sizes and both graph degrees with no competing load on the host.

Concurrency

“In service” is throughput times median latency. By Little’s law that is the average number of queries inside the engine during the window. For the top three it tracks the client count, so the x-axis means what it says. For the two MariaDB builds it does not. At nominally 32 clients, 14.4 and 7.1 queries were in service.

Why are two cells marked? Our harness runs its concurrent clients as threads in one Python interpreter, each with its own connection. MariaDB Connector/C holds the interpreter lock across its socket wait, so some of those clients were waiting on the lock rather than on the database. A later smoke run made it obvious: at 4 clients on a small corpus, MariaDB returned 2,927 QPS at a 0.325 ms median, which is one query in flight.

The harness now computes “in service” for every window and refuses to divide throughput by the client count when it falls below half. On this run that refuses MariaDB 11.8 at 16 and 32 clients, MariaDB 12.3 at 2, 16 and 32, and AliSQL at 2. Six points of thirty-six. It refuses nothing from pgvector, Valkey or Percona Search. It also does not refuse a saturated server: a real queue raises the whole latency distribution including the median, so “in service” stays at the client count while throughput flattens.

The finding survives the withdrawal. Throughput did not rise and median latency rose twentyfold. MariaDB 11.8 returned 165 QPS at one client and 148 at thirty-two, p50 going 4.6 ms to 97 ms. AliSQL went 5.1 ms to 133 ms for the same flat throughput. Offering an engine more work and getting the same throughput at twenty times the latency is a queue, and that queue was on the server side. A client still waiting for the interpreter lock has not issued its query and adds nothing to these percentiles. What we cannot claim is that these three were measured at 32 concurrent queries. They were measured at 7 to 17.

The three flat engines are the three on InnoDB. They are also the three driven through MariaDB Connector/C, where the others use psycopg3, PyMongo and a Valkey client. Two variables, three engines, one pattern. The grouping cannot separate them and nothing here has been profiled. MHNSW and VIDX are separate codebases from different teams, which is what makes the storage engine the interesting half.

The same arithmetic shows one more thing. At a single client, pgvector, Valkey and Percona Search spent 99 to 101% of the window inside the query call. MariaDB 11.8, MariaDB 12.3 and AliSQL spent 74 to 75%. About 1.5 ms per query on that path is our harness, not the server, so their single-client latencies read worse than the engine is. Throughput is unaffected.

Filtered search

Every engine here answers “nearest neighbours of this vector where a scalar predicate holds”. They do it three different ways, and the ways diverge as the predicate tightens.

Recall is scored against ground truth recomputed over the qualifying rows only. pgvector’s two recall figures are marked because its queries returned fewer than k rows: 224 of 500 at 10% selectivity, all 500 at 1%.

pgvector post-filters and then stops. It walks the graph, collects ef_search candidates, and drops the ones that fail the predicate. At 1% selectivity nearly all of them fail, so the query returns one or two rows where ten were asked for. No error, no warning, full speed. Recall 0.1034 is a short answer rather than a bad one. Iterative scan exists in pgvector 0.8 for this case and was not measured here, so this is the cost of the default. The default is what an application gets if nobody knows to change it.

MariaDB and AliSQL post-filter too, but keep searching until enough rows survive. Recall holds above 0.97 and throughput goes under one query a second, with a p99 of ten to thirteen seconds.

Percona Search pre-filters, which is why it is the only engine that gets faster as the predicate tightens: 71.9 QPS at 10% selectivity, 117.5 at 1%. Its recall is the lowest of the five that answer completely, 0.858 to 0.867, and that is the scalar quantization rather than the filter. Unquantized it reaches 0.997 on this corpus, at the same index size. Valkey plans per query and is the only engine that is both accurate and usable at 1%.

Churn

Delete 99,000 rows, re-insert the same 99,000, measure the index again.

The three dashes are our bug, not an engine result. Each re-insert gets a wall-clock budget so a stalled write releases the machine instead of holding the run overnight. We set it to thirty minutes, sized against a Valkey defect where pipelined writes into an indexed prefix never returned. Thirty minutes is far too short for an engine doing the work slowly and correctly. MariaDB 12.3 got 51,480 rows of 99,000 in, MariaDB 11.8 got 41,580, AliSQL got 29,700, and the post-churn measurement never ran. The budget is two hours now. Recovering those three cells costs about 24 hours, most of it reloading the corpus.

The partial run still answers the question we wanted. Re-inserting into a populated index runs at 16 to 28 rows a second, against 41 to 76 for the initial load. Writing into an established graph costs two to three times what building it does, on all three engines. These re-inserts ran with no reader threads in the interpreter, which the records confirm, so the concurrency problem above does not touch them.

Percona Search loses 39%, the largest drop in the table, while recall goes 0.8432 to 0.8416. The index is not degrading, the throughput is. pgvector is flat because its repair path ran: VACUUM (ANALYZE) after the re-insert, the only maintenance hook any engine here exposes for this.

What we got wrong

• Concurrent clients were threads sharing one interpreter lock. That made every multi-client figure conditional on something the figure did not state. The harness now records, for every window, the time-average number of queries outstanding and a robust estimate of how many were in service, and refuses the per-client division when the second falls below half the client count. Six points in this run are refused. The fix is each client in its own OS process. It is implemented and measured, and it changes what every multi-client number means, so it carries a measurement-version bump: numbers taken after it will not be charted beside numbers taken before it.
• At one client, the MySQL-family path spends a quarter of the window outside the query call. The other three spend none. Latency comparisons across those two groups carry that difference. Throughput comparisons do not.
• The churn budget was sized for a hang, not for slow correct work. It cost three cells.
• pgvector’s filtered numbers are the default configuration. Iterative scan was not measured, and it is the feature that addresses the failure we recorded.
• Graph degree is two points, not a curve. M = 6 and M = 16 for MHNSW and VIDX so they would have real measurements below recall 0.90. The other three ran at M = 16 only. M = 6 was worse than M = 16 everywhere above recall 0.88.
• One configuration, one machine, one run. The four engines built from source all used -march=x86-64-v3. The two installed from packages compile nothing of ours. Mixing that flag between engines would turn the comparison into a comparison of compiler flags, so it is recorded per engine in the manifest.

The run manifest holds the host, the resolved budgets, the image IDs and the source commit of every engine, and the report generator refuses a run directory without one.

What this means, and where it stops

Choosing on this evidence: pgvector for concurrent traffic, where 4,798 QPS at a 10.8 ms p99 is an order of magnitude clear of the MySQL family, provided someone turns on iterative scan before shipping a filtered query. Valkey if the working set fits in memory and you need filters that are both accurate and fast; test its write path into an indexed prefix first. MariaDB 12.3 if the index has to be small and accurate and the traffic is a handful of concurrent clients. It is the fastest engine here at recall 0.99 and its index is a quarter the size of Percona Search’s. Percona Search if a tight predicate is the common case, since it is the only engine that speeds up as the filter narrows, and if a Technical Preview is acceptable. MariaDB 11.8 does nothing 12.3 does not do better. AliSQL builds slowest and has a latency cliff above ef_search 80.

Where it stops: one corpus at one dimensionality, one graph degree for half the field, one machine, one run, no profiling of the concurrency ceiling, and no second observation of anything. The numbers most likely to move under a re-run are the three missing churn cells and the six refused concurrency points. The least likely are build time, index size and recall, which agreed across earlier runs at two corpus sizes.

Try it

All runs can take about 40 hours, nearly all of it the incremental builds. Smaller profiles exist for checking the pipeline before committing a machine for two days: --profile dev is about a minute per engine on a synthetic corpus, --profile smoke about 45 minutes per pass and it exercises every chart. The run directory keeps the records, the manifest, per-phase logs and memory timeseries, so any number here traces back to the process that produced it. Issues and corrections are welcome.

Run full-20260825, profile tuned-complete, tuned resource pass, dbpedia-openai-1000k-angular, 990,000 × 1,536, M = 16, k = 10. Versions: MariaDB 11.8.8, MariaDB 12.3.2, AliSQL 8.0.44-2, pgvector 0.8.6, Valkey 9.1.1 with valkey-search, Percona Search for MongoDB 1.70.3-1 on PSMDB 8.3.8-2. One run, all of it. Missing measurements are marked missing, not estimated. Charts are the ones the framework generates.