Benchmarks / metadata latency

Sub-millisecond metadata, even one hop away.

100 drives on 3 serve-http servers, with TiDB metadata, RustFS blocks and a Redis route registry. One client per drive, each on its own connection. Every client either talks to its drive’s owner, or to another server that forwards the request over HTTP/3.

0.72msp99 forwarded stat, durable TiDB drive
0.81msp99 forwarded rename, writeback drive
0.26msp99 local stat on the owner
0errors159,999 measured requests

Run 44015339854 · 17 November 2026 · 4 vCPU Intel Xeon 6973P-C, 15 GiB (GitHub ubuntu-latest)

Explore

Every operation, every percentile.

Pick a scenario and a percentile. Blue bars were served by the drive’s owner, violet bars were forwarded from another server, and amber bars took a millisecond or longer.

Scenario
Percentile

Durable TiDB, reads only, route refresh on. The default: routes in use are confirmed in the background before they expire.

0.00 ms0.25 ms0.50 ms0.75 ms1.00 mspingforwarded0.33 msstatforwarded0.72 msheadforwarded0.73 mslistforwarded0.76 ms1 ms target

local forwarded 1 ms or more · hover or focus a row for every percentile

01 / reads

Read traffic stays under 1 ms at p99, even forwarded.

Local reads are answered from the owner’s memory in about a tenth of a millisecond. Forwarding adds roughly 0.3 ms for the HTTP/3 hop, and background route refresh keeps the slowest 1% under the target. When the same drives also run durable writes, forwarded read p99 rose to 1.51–2.50 ms on this runner, as TiDB competes for its four vCPUs.

Metadata read latency, p50 and p99Writeback stat, local: p50 0.132 ms, p99 0.26 ms; SQLite stat, local: p50 0.128 ms, p99 0.268 ms; Durable stat, forwarded: p50 0.414 ms, p99 0.722 ms; Durable list, forwarded: p50 0.433 ms, p99 0.761 ms; Durable stat, forwarded: p50 0.441 ms, p99 0.869 ms; SQLite stat, forwarded: p50 0.408 ms, p99 0.674 ms0 ms0.25 ms0.5 ms0.75 ms1 ms1.25 ms1.5 msWriteback stat, localclient connected to the ownerp99 0.26p50 0.13SQLite stat, localclient connected to the ownerp99 0.27p50 0.13Durable stat, forwardedroute refresh on (default)p99 0.72p50 0.41Durable list, forwardedroute refresh on (default)p99 0.76p50 0.43Durable stat, forwardedroute refresh offp99 0.87p50 0.44SQLite stat, forwardedone HTTP/3 hopp99 0.67p50 0.411 ms target
p99 p50 · request to response, measured by the client

02 / writes

Writeback mutations: under 1 ms too.

With writeback, an owner applies mkdir, rename and delete in memory and publishes them afterwards. Forwarded mutations land under the target as well.

Writeback mutation latency, p50 and p99Writeback mkdir, local: p50 0.157 ms, p99 0.329 ms; Writeback rename, local: p50 0.16 ms, p99 0.312 ms; Writeback mkdir, forwarded: p50 0.442 ms, p99 0.754 ms; Writeback rename, forwarded: p50 0.446 ms, p99 0.805 ms; Writeback delete, forwarded: p50 0.409 ms, p99 0.771 ms0 ms0.25 ms0.5 ms0.75 ms1 ms1.25 ms1.5 msWriteback mkdir, localacknowledged from memoryp99 0.33p50 0.16Writeback rename, localacknowledged from memoryp99 0.31p50 0.16Writeback mkdir, forwardedone HTTP/3 hopp99 0.75p50 0.44Writeback rename, forwardedone HTTP/3 hopp99 0.81p50 0.45Writeback delete, forwardedone HTTP/3 hopp99 0.77p50 0.411 ms target
p99 p50 · request to response, measured by the client
Durable mutations are bounded by the database commit.

A durable drive acknowledges a mutation only after TiDB commits it, so it cannot reach 1 ms: local mkdir measured p50 3.25 ms and delete p50 2.97 ms, most of it the TiDB commit. Their p99 tail varied from 13 ms to hundreds of milliseconds between runners. The likely cause is the runners’ disk sync latency under TiKV; that is not yet confirmed.

03 / repeatability

Same code, three different runner CPUs.

GitHub assigns a different CPU to each run, so every summary records it. With route refresh on, the default, forwarded reads met the target on all three. With it off, one runner missed.

Forwarded stat p99, milliseconds
Runner CPUDurable, refresh onDurable, refresh offSQLite
AMD EPYC 9V450.651.260.63
AMD EPYC 9V740.820.920.74
Intel Xeon 6973P-C0.720.870.67

04 / method

How it was measured.

Everything runs on one GitHub-hosted runner, so these numbers include the servers, TiDB and the load client competing for four vCPUs.

Cluster
Three serve-http owners, each owning a third of the drives, forwarding to each other over HTTP/3 with mutual TLS.
Storage
TiDB v8.5.7 (one PD, one TiKV with production defaults, one TiDB), RustFS for blocks, Redis for routes. A SQLite run swaps TiDB for local SQLite.
Load
One client per drive on its own keep-alive HTTP/1.1 connection, 10 requests a second each, staggered so clients do not fire together. 20 seconds measured per mode after a warm-up.
Routes
“Local” clients call their drive’s owner. “Forwarded” clients call the next server, which forwards every request.
Operations
stat and list are JSON metadata calls, head is HEAD on a file, ping is /readyz; mkdir, rename and delete change the namespace.

Lessons from getting these numbers: Docker Desktop on a Mac added about 0.5 ms to forwarded p99, and a TiKV configured for small test machines (one scheduler thread, 64 write latches) turned durable writes’ p99 into 400 ms. Both were environment, not mount-rs, so the benchmark runs on Linux runners with TiKV defaults.

All results from run 44015339854
Durable TiDB, reads only, route refresh onThe default: routes in use are confirmed in the background before they expire.
RouteOpRequestsp50 msp90 msp99 msp99.9 msUnder 1 ms
forwardedping5,0000.130.180.330.9699.9%
forwardedstat5,0000.410.530.721.1599.8%
forwardedhead5,0000.420.530.731.5299.8%
forwardedlist5,0000.430.550.761.2099.8%
Durable TiDB, reads only, route refresh offThe ingress looks a route up in Redis whenever its one-second cache entry expires.
RouteOpRequestsp50 msp90 msp99 msp99.9 msUnder 1 ms
forwardedping5,0000.130.180.310.57100.0%
forwardedstat5,0000.440.650.872.7999.7%
forwardedhead5,0000.430.580.871.5299.5%
forwardedlist5,0000.440.620.861.3699.7%
Writeback (exclusive ownership)Mutations are acknowledged from memory and published to TiDB afterwards.
RouteOpRequestsp50 msp90 msp99 msp99.9 msUnder 1 ms
localstat3,3990.130.180.260.59100.0%
localhead3,3000.130.190.260.46100.0%
locallist3,3000.150.210.310.64100.0%
localmkdir3,3000.160.220.330.63100.0%
localrename3,3010.160.210.310.51100.0%
localdelete3,4000.150.200.311.0799.9%
forwardedstat3,3990.400.500.701.2199.7%
forwardedhead3,3000.410.500.690.92100.0%
forwardedlist3,3000.420.520.721.0199.9%
forwardedmkdir3,3000.440.540.750.9999.9%
forwardedrename3,3010.450.560.811.3799.7%
forwardeddelete3,4000.410.510.771.3899.6%
SQLite metadataLocal SQLite metadata on each owner, no TiDB round trips.
RouteOpRequestsp50 msp90 msp99 msp99.9 msUnder 1 ms
localping9,9990.120.170.250.9399.9%
localstat10,0000.130.180.270.65100.0%
forwardedping10,0000.120.170.280.60100.0%
forwardedstat10,0000.410.490.671.1499.8%
Durable TiDB, reads and writes mixedEvery mutation is committed to TiDB before the response. Route refresh off.
RouteOpRequestsp50 msp90 msp99 msp99.9 msUnder 1 ms
localping2,8010.130.200.531.1699.9%
localstat2,9000.140.210.540.9799.9%
localhead2,9000.140.210.370.69100.0%
locallist2,9000.170.240.461.2099.8%
localmkdir2,8993.255.6247.5174.850.0%
localrename2,8003.205.0230.6569.590.0%
localdelete2,8002.974.638.5863.570.0%
forwardedping2,8010.130.261.593.5598.5%
forwardedstat2,9000.450.722.506.5196.3%
forwardedhead2,9000.460.731.673.1796.8%
forwardedlist2,9000.500.851.512.9194.0%
forwardedmkdir2,8993.696.83179.48230.130.0%
forwardedrename2,8003.846.9954.48191.310.0%
forwardeddelete2,8003.466.75328.45383.670.0%