Benchmarking Vector search in six database engines

We loaded 990,000 OpenAI embeddings of 1,536 dimensions into six databases that index vectors themselves, built an HNSW index in each, and measured five things: time to a queryable index, storage and memory, recall against throughput, throughput under concurrent clients, and what happens when 10% of the corpus is rewritten. Same host, same container shape, same query set.

The build numbers spread wider than we expected. Time to a queryable index ran from 1.9 minutes to 6.7 hours for the same million vectors. Index size ran from 3.9 GiB to 15.1 GiB, and the engine with the smallest index took the longest to build it. At recall 0.99 the field closes up: MariaDB 12.3, Valkey and pgvector are within a factor of 1.3 of each other. Under 32 clients pgvector returns 21.7× its single-client throughput. We do not report a comparable figure for the three InnoDB-based engines: their multi-client numbers measured a limit in our client, not in the server (see Concurrency).

This is one run on one machine. Multi-client numbers for MariaDB 11.8, MariaDB 12.3 and AliSQL are not reported, and the reason is below. Framework and configuration: github.com/Percona-Lab/vector-bench. Raw records, logs and manifest of this run: release run-full-20260825.

The machine, and how we ran it

The database runs in one container and the harness in another, on a private network. One container would charge the several hundred megabytes of NumPy holding the query set to the server’s cgroup and ruin the memory numbers. The harness never configures a server. The orchestrator does, from the same code for every engine, and each engine’s settings are in its server logs beside the results.

Memory went to each engine’s own main cache, so the sizes differ by engine: a 42.3 GB buffer pool for the three InnoDB engines, 72.5 GB of shared_buffers plus 12.1 GB of maintenance_work_mem for pgvector, 108.7 GB of maxmemory for Valkey. Percona Search ran on MongoDB’s default WiredTiger cache, which came to 57–64 GB; the framework did not set it. The three engines with a graph cache, MariaDB 11.8, MariaDB 12.3 and AliSQL, got 30.2 GB of it. All three default to 16 MiB, and leaving that in place would judge a vendor on a value it plainly intends you to change. Index builds got 8 threads; the orchestrator clamped them down from 64, because more than a handful contend rather than help and a large value stalled writes into a populated index outright. It raised /dev/shm from 8 GB to 13 GB so pgvector’s parallel build could hold a segment the size of maintenance_work_mem.

In the recall-against-throughput results, recall is ann-benchmarks’ distance-threshold definition, so those numbers stay comparable with published ANN results. The filtered and churn sections score recall as the overlap with the true top-10 IDs. Ground truth for filtered queries is recomputed by brute force over the rows that satisfy each predicate. A WHERE clause changes the correct answer set, and scoring a filtered query against unfiltered neighbours scores every engine against the wrong targets.

What ran

Percona Search for MongoDB is a Technical Preview and its documentation says not for production. Its numbers use the scalar-quantized index, which MongoDB recommends above a 3 GB vector index.

Loading the data

MHNSW and VIDX update the graph on every INSERT, so their load is the index build and there is nothing separate to time. That puts them at 41 to 76 rows a second against pgvector’s 613 and Valkey’s 30,170. pgvector’s 40.9 minutes is 26.9 minutes of COPY plus 14.0 minutes of parallel build across 8 workers.

They get the smallest index for it: 3.9 GiB, where the raw float32 vectors alone are 5.7 GiB. The index is smaller than the vectors because MHNSW and VIDX keep 16-bit quantized copies of the vectors in the graph, while pgvector and Valkey index full float32. The full-precision vectors stay in the table: MariaDB stores 14.3 GiB there, Percona Search 5.8 GiB. Counting index and table together, AliSQL uses 14.4 GiB, pgvector 15.2, MariaDB 11.8 and Valkey 18.2 each, and Percona Search 20.9. Percona Search’s index measured 15.1 GiB after the load and 7.2 GiB at the start of the churn phase; we do not know which reflects its steady state.

Build time is a re-indexing window. Changing embedding model is an afternoon on pgvector and a weekend on AliSQL.

Recall against throughput

Best throughput whose measured recall clears each floor. MariaDB and AliSQL repeat a number across two columns because their lowest ef_search setting already returns recall 0.956 to 0.989. It wins both floors. For every MySQL-family engine the ef_search 10 point measured slower than ef_search 20, a warm-up effect the report flags, so their ≥ 0.90 column may be understated. Percona Search shows dashes because the scalar-quantized index used everywhere else in this post peaks at recall 0.892; unquantized, it reaches 138, 124 and 88 QPS at the three floors. The pgvector and Valkey figures come from builds with ef_construction 400; the load table and the other sections use 200 for pgvector and the default for Valkey.

MariaDB 12.3 has the highest throughput at recall 0.99, 277 QPS against Valkey’s 264, a gap one run cannot separate from noise. It is also the one engine built with -march=native rather than x86-64-v3. It gives six times the throughput of 11.8 at that floor for 50% more build time, which is the case for upgrading if accuracy above 0.99 is the target.

Raising the floor from 0.90 to 0.95 costs pgvector and Valkey a third of their throughput. It costs MariaDB and AliSQL nothing, because their lowest ef_search setting is already above both floors. All six have a query-time knob; what MHNSW and VIDX lack is a build-time one. pgvector and Valkey expose ef_construction, and they do not.

One AliSQL point is unexplained. At M = 16, throughput rises from 2.6 to 4.4 QPS between ef_search 400 and 800. More search effort cannot be faster on the same index. AliSQL also has a latency jump at M = 16: p50 is 6.6 ms at ef_search 80, 50.8 ms at 120 and 191 to 384 ms beyond. At M = 6 the jump comes later, between ef_search 200 and 400. The report marks this range as unreliable, so treat these AliSQL points as unmeasured.

Concurrency

Dashes are not missing data: the multi-client numbers for these three engines reflect a limit in our benchmark client, explained below, so they are not reported.

“In service” is throughput times median latency. It is a rough estimate of how many queries were in flight, not Little’s law. Little’s law uses mean latency. With a heavy tail the median is far below the mean, so this figure undercounts. For pgvector, Valkey and Percona Search it is close to the client count. For the three MySQL-family engines it is well below it.

Why are these points not reported? Our harness runs its concurrent clients as threads in one Python interpreter, each with its own connection. Their throughput stayed flat from 1 to 32 clients, and they are the three driven through the MariaDB Python connector. The others use psycopg3, PyMongo and a Valkey client.

A later smoke run of ours on the same host shows where that ceiling comes from. On a small corpus, going from 1 to 4 clients, queries in service per client stayed at 0.97 to 0.98 for pgvector, Valkey and Percona Search. For MariaDB 11.8, MariaDB 12.3 and AliSQL it fell from 0.95 to 0.24. That is about one query in flight however many clients ask. A server-side bottleneck would not produce the same ceiling in three separate engines with three separate codebases. A shared client library would.

We have not identified the exact mechanism. The connector releases the interpreter lock while it waits for a query, so the lock is not simply held across the socket wait. Our latency timer also starts in Python before the driver call, so time spent waiting for the interpreter lock is inside the measured latency, and a rise in median latency does not show a queue on the server. We did not profile the server, so we make no claim about how these engines scale under concurrent clients.

The harness now runs each client in its own OS process. This removes the shared interpreter as a cause. It has been tested on synthetic drivers but not yet on these engines. We will publish a re-measurement of the three engines with it rather than estimate the result.

At a single client, every engine spent the whole window inside the timed query call: throughput times mean latency is 1.00 for all six, so there is no measurable harness overhead outside the call. The MySQL-family engines have a heavier tail at one client, a mean of 5.0 to 6.7 ms against a median of 3.7 to 5.1 ms, and their single-client points vary by 22 to 24% between repeats because the first repeat runs cold.

Two limits apply to the multi-client numbers that are reported. Every engine ran at ef_search 100, which gives each engine a different recall, so throughput here is not compared at equal accuracy. And the harness container has 16 CPUs for up to 32 clients: Valkey and Percona Search flatten between 16 and 32 clients, which may be the client running out of CPU rather than the server.

Filtered search

Every engine here answers “nearest neighbours of this vector where a scalar predicate holds”. They do it three different ways, and the ways diverge as the predicate tightens.

Recall is scored against ground truth recomputed over the qualifying rows only. pgvector’s two recall figures are marked because its queries returned fewer than k rows: 224 of 500 at 10% selectivity, all 500 at 1%.

pgvector post-filters and then stops. It walks the graph, collects ef_search candidates, and drops the ones that fail the predicate. At 1% selectivity nearly all of them fail, so the query returns one or two rows where ten were asked for. No error, no warning, full speed. Recall 0.1034 is a short answer rather than a bad one. Iterative scan exists in pgvector 0.8 for this case and was not measured here, so this is the cost of the default. The default is what an application gets if nobody knows to change it.

MariaDB and AliSQL post-filter too, but keep searching until enough rows survive. Recall holds at 0.96 to 0.99 and throughput goes under one query a second, with a p99 of ten to thirteen seconds.

Percona Search pre-filters, which is why it is the only engine that gets faster as the predicate tightens: 71.9 QPS at 10% selectivity, 117.5 at 1%. Its recall is the lowest of the five that answer completely, 0.858 to 0.867, and that is the scalar quantization rather than the filter. Unquantized it reaches 0.997 on this corpus, at a similar index size. Valkey plans per query and is the only engine that is both accurate and usable at 1%.

Churn

Delete 99,000 rows, re-insert the same 99,000, measure the index again.

The three dashes are our bug, not an engine result. Each re-insert gets a wall-clock budget so a stalled write releases the machine instead of holding the run overnight. We set it to thirty minutes, sized against a Valkey defect where pipelined writes into an indexed prefix never returned. Thirty minutes is far too short for an engine doing the work slowly and correctly. MariaDB 12.3 got 51,480 rows of 99,000 in, MariaDB 11.8 got 41,580, AliSQL got 29,700, and the post-churn measurement never ran. The budget is two hours now. Recovering those three cells costs about 24 hours, most of it reloading the corpus.

The partial run still answers part of the question. Re-inserting into a populated index ran at 16 to 28 rows a second, against 41 to 76 for the initial load: 1.8 times slower for MariaDB 12.3, 2.5 for AliSQL and 3.3 for MariaDB 11.8. The two rates are not measured the same way: the load used 8 threads, the re-insert one sequential writer, into a graph that already held 891,000 rows.

Percona Search loses 38.9%, the largest drop in the table, while recall goes 0.8432 to 0.8416. The index is not losing accuracy, the throughput is. Percona Search builds its index asynchronously, so its 20,223 rows/s is the rate at which mongod accepted the writes, not the rate at which the index absorbed them, and the drop may include indexing that was still catching up. Valkey re-inserted at 137 rows/s against 30,170 at load, and we have not explained that gap. pgvector’s throughput did not change.

What we got wrong

• Concurrent clients were threads sharing one Python interpreter, and the MySQL-family numbers measured that. The three engines driven through the MariaDB Python connector stayed at about one query in flight however many clients ran. Their multi-client points are not reported. The fix is each client in its own OS process. It is implemented and tested on synthetic drivers. It has not yet been run against these engines, and no results from it are in this post. It carries a measurement-version bump, so numbers taken after it will not be charted beside numbers taken before it.
• Memory budgets were per engine, and the manifest does not keep all of them. Each engine’s main cache was sized separately. The manifest’s resolved resources are rewritten by whichever engine ran last, so the per-engine values above come from the server logs. The manifest also lists only the last phases of the MySQL-family recall sweep; the M = 16 results in the recall table come from earlier phases of the same run.
• The churn budget was sized for a hang, not for slow correct work. It cost three cells.
• pgvector’s filtered numbers are the default configuration. Iterative scan was not measured, and it is the feature that addresses the failure we recorded.
• Graph degree is two points, not a curve. M = 6 and M = 16 for MHNSW and VIDX so they would have real measurements below recall 0.90. The other three ran at M = 16 only. M = 6 was worse than M = 16 everywhere above recall 0.88.
• One configuration, one machine, one run, and build flags that were not uniform. MariaDB 11.8, AliSQL and pgvector were built with -march=x86-64-v3; MariaDB 12.3 was built with -march=native. MariaDB and AliSQL choose their AVX-512 distance kernels at runtime either way, while pgvector relies mostly on the compiler for SIMD, so x86-64-v3 limits it to AVX2. The two engines installed from packages compile nothing of ours. The flag is recorded per engine in the manifest.

The run manifest holds the host, the image IDs and the source commit of every engine, and the report generator refuses a run directory without one.

What this means, and where it stops

Choosing on this evidence: pgvector if you want a fast index build and throughput that holds through churn; turn on iterative scan before shipping a filtered query, because its default returns short answers at tight selectivity. Valkey if the working set fits in memory and you need filters that are both accurate and fast; test its write path into an indexed prefix first. MariaDB 12.3 or AliSQL if storage matters: their 16-bit index is the smallest here, and MariaDB 12.3 has the highest single-client throughput at recall 0.99. Percona Search if a tight predicate is the common case, since it is the only engine that speeds up as the filter narrows, and if a Technical Preview is acceptable. MariaDB 11.8 loads faster than 12.3 but is six times slower at recall 0.99. AliSQL builds slowest and has an unexplained latency jump above ef_search 80.

This post makes no recommendation based on concurrent throughput. For pgvector, Valkey and Percona Search the multi-client numbers stand. For the three MySQL-family engines we do not have a valid measurement yet.

Where it stops: one corpus at one dimensionality, one graph degree for half the field, one machine, one run, no valid concurrency measurement for the three MySQL-family engines, and no second observation of anything. The numbers most likely to move under a re-run are those concurrency numbers, the three missing churn cells and the AliSQL points above ef_search 80. The least likely are build time, index size and recall, which agreed across earlier runs at two corpus sizes.

Try it

All runs can take about 40 hours, nearly all of it the incremental builds. Smaller profiles exist for checking the pipeline before committing a machine for two days: --profile dev is about a minute per engine on a synthetic corpus, --profile smoke about 45 minutes per pass and it exercises every chart. The run directory keeps the records, the manifest, per-phase logs and memory timeseries, so any number here traces back to the process that produced it. The run directory behind this post is published as release run-full-20260825. Issues and corrections are welcome.

Run full-20260825, profile tuned-complete, tuned resource pass, dbpedia-openai-1000k-angular, 990,000 × 1,536, M = 16, k = 10. Versions: MariaDB 11.8.8, MariaDB 12.3.2, AliSQL 8.0.44-2, pgvector 0.8.6, Valkey 9.1.1 with valkey-search, Percona Search for MongoDB 1.70.3-1 on PSMDB 8.3.8-2. One run, all of it. Missing measurements are marked missing, not estimated. Charts are the ones the framework generates.