Benchmarks / metadata latency
Sub-millisecond metadata, even one hop away.
100 drives on 3 serve-http servers, with TiDB metadata, RustFS blocks and a Redis route registry. One client per drive, each on its own connection. Every client either talks to its drive’s owner, or to another server that forwards the request over HTTP/3.
Run 44015339854 · 17 November 2026 · 4 vCPU Intel Xeon 6973P-C, 15 GiB (GitHub ubuntu-latest)
Explore
Every operation, every percentile.
Pick a scenario and a percentile. Blue bars were served by the drive’s owner, violet bars were forwarded from another server, and amber bars took a millisecond or longer.
Durable TiDB, reads only, route refresh on. The default: routes in use are confirmed in the background before they expire.
local forwarded 1 ms or more · hover or focus a row for every percentile
01 / reads
Read traffic stays under 1 ms at p99, even forwarded.
Local reads are answered from the owner’s memory in about a tenth of a millisecond. Forwarding adds roughly 0.3 ms for the HTTP/3 hop, and background route refresh keeps the slowest 1% under the target. When the same drives also run durable writes, forwarded read p99 rose to 1.51–2.50 ms on this runner, as TiDB competes for its four vCPUs.
02 / writes
Writeback mutations: under 1 ms too.
With writeback, an owner applies mkdir, rename and delete in memory and publishes them afterwards. Forwarded mutations land under the target as well.
A durable drive acknowledges a mutation only after TiDB commits it, so it cannot reach 1 ms: local mkdir measured p50 3.25 ms and delete p50 2.97 ms, most of it the TiDB commit. Their p99 tail varied from 13 ms to hundreds of milliseconds between runners. The likely cause is the runners’ disk sync latency under TiKV; that is not yet confirmed.
03 / repeatability
Same code, three different runner CPUs.
GitHub assigns a different CPU to each run, so every summary records it. With route refresh on, the default, forwarded reads met the target on all three. With it off, one runner missed.
| Runner CPU | Durable, refresh on | Durable, refresh off | SQLite |
|---|---|---|---|
| AMD EPYC 9V45 | 0.65 | 1.26 | 0.63 |
| AMD EPYC 9V74 | 0.82 | 0.92 | 0.74 |
| Intel Xeon 6973P-C | 0.72 | 0.87 | 0.67 |
04 / method
How it was measured.
Everything runs on one GitHub-hosted runner, so these numbers include the servers, TiDB and the load client competing for four vCPUs.
- Cluster
- Three
serve-httpowners, each owning a third of the drives, forwarding to each other over HTTP/3 with mutual TLS. - Storage
- TiDB v8.5.7 (one PD, one TiKV with production defaults, one TiDB), RustFS for blocks, Redis for routes. A SQLite run swaps TiDB for local SQLite.
- Load
- One client per drive on its own keep-alive HTTP/1.1 connection, 10 requests a second each, staggered so clients do not fire together. 20 seconds measured per mode after a warm-up.
- Routes
- “Local” clients call their drive’s owner. “Forwarded” clients call the next server, which forwards every request.
- Operations
- stat and list are JSON metadata calls, head is
HEADon a file, ping is/readyz; mkdir, rename and delete change the namespace.
Lessons from getting these numbers: Docker Desktop on a Mac added about 0.5 ms to forwarded p99, and a TiKV configured for small test machines (one scheduler thread, 64 write latches) turned durable writes’ p99 into 400 ms. Both were environment, not mount-rs, so the benchmark runs on Linux runners with TiKV defaults.
All results from run 44015339854
| Route | Op | Requests | p50 ms | p90 ms | p99 ms | p99.9 ms | Under 1 ms |
|---|---|---|---|---|---|---|---|
| forwarded | ping | 5,000 | 0.13 | 0.18 | 0.33 | 0.96 | 99.9% |
| forwarded | stat | 5,000 | 0.41 | 0.53 | 0.72 | 1.15 | 99.8% |
| forwarded | head | 5,000 | 0.42 | 0.53 | 0.73 | 1.52 | 99.8% |
| forwarded | list | 5,000 | 0.43 | 0.55 | 0.76 | 1.20 | 99.8% |
| Route | Op | Requests | p50 ms | p90 ms | p99 ms | p99.9 ms | Under 1 ms |
|---|---|---|---|---|---|---|---|
| forwarded | ping | 5,000 | 0.13 | 0.18 | 0.31 | 0.57 | 100.0% |
| forwarded | stat | 5,000 | 0.44 | 0.65 | 0.87 | 2.79 | 99.7% |
| forwarded | head | 5,000 | 0.43 | 0.58 | 0.87 | 1.52 | 99.5% |
| forwarded | list | 5,000 | 0.44 | 0.62 | 0.86 | 1.36 | 99.7% |
| Route | Op | Requests | p50 ms | p90 ms | p99 ms | p99.9 ms | Under 1 ms |
|---|---|---|---|---|---|---|---|
| local | stat | 3,399 | 0.13 | 0.18 | 0.26 | 0.59 | 100.0% |
| local | head | 3,300 | 0.13 | 0.19 | 0.26 | 0.46 | 100.0% |
| local | list | 3,300 | 0.15 | 0.21 | 0.31 | 0.64 | 100.0% |
| local | mkdir | 3,300 | 0.16 | 0.22 | 0.33 | 0.63 | 100.0% |
| local | rename | 3,301 | 0.16 | 0.21 | 0.31 | 0.51 | 100.0% |
| local | delete | 3,400 | 0.15 | 0.20 | 0.31 | 1.07 | 99.9% |
| forwarded | stat | 3,399 | 0.40 | 0.50 | 0.70 | 1.21 | 99.7% |
| forwarded | head | 3,300 | 0.41 | 0.50 | 0.69 | 0.92 | 100.0% |
| forwarded | list | 3,300 | 0.42 | 0.52 | 0.72 | 1.01 | 99.9% |
| forwarded | mkdir | 3,300 | 0.44 | 0.54 | 0.75 | 0.99 | 99.9% |
| forwarded | rename | 3,301 | 0.45 | 0.56 | 0.81 | 1.37 | 99.7% |
| forwarded | delete | 3,400 | 0.41 | 0.51 | 0.77 | 1.38 | 99.6% |
| Route | Op | Requests | p50 ms | p90 ms | p99 ms | p99.9 ms | Under 1 ms |
|---|---|---|---|---|---|---|---|
| local | ping | 9,999 | 0.12 | 0.17 | 0.25 | 0.93 | 99.9% |
| local | stat | 10,000 | 0.13 | 0.18 | 0.27 | 0.65 | 100.0% |
| forwarded | ping | 10,000 | 0.12 | 0.17 | 0.28 | 0.60 | 100.0% |
| forwarded | stat | 10,000 | 0.41 | 0.49 | 0.67 | 1.14 | 99.8% |
| Route | Op | Requests | p50 ms | p90 ms | p99 ms | p99.9 ms | Under 1 ms |
|---|---|---|---|---|---|---|---|
| local | ping | 2,801 | 0.13 | 0.20 | 0.53 | 1.16 | 99.9% |
| local | stat | 2,900 | 0.14 | 0.21 | 0.54 | 0.97 | 99.9% |
| local | head | 2,900 | 0.14 | 0.21 | 0.37 | 0.69 | 100.0% |
| local | list | 2,900 | 0.17 | 0.24 | 0.46 | 1.20 | 99.8% |
| local | mkdir | 2,899 | 3.25 | 5.62 | 47.51 | 74.85 | 0.0% |
| local | rename | 2,800 | 3.20 | 5.02 | 30.65 | 69.59 | 0.0% |
| local | delete | 2,800 | 2.97 | 4.63 | 8.58 | 63.57 | 0.0% |
| forwarded | ping | 2,801 | 0.13 | 0.26 | 1.59 | 3.55 | 98.5% |
| forwarded | stat | 2,900 | 0.45 | 0.72 | 2.50 | 6.51 | 96.3% |
| forwarded | head | 2,900 | 0.46 | 0.73 | 1.67 | 3.17 | 96.8% |
| forwarded | list | 2,900 | 0.50 | 0.85 | 1.51 | 2.91 | 94.0% |
| forwarded | mkdir | 2,899 | 3.69 | 6.83 | 179.48 | 230.13 | 0.0% |
| forwarded | rename | 2,800 | 3.84 | 6.99 | 54.48 | 191.31 | 0.0% |
| forwarded | delete | 2,800 | 3.46 | 6.75 | 328.45 | 383.67 | 0.0% |
Run it yourself
The benchmark is in the repository.
scripts/metadata-latency-bench.py starts the cluster and drives the load client.