Router Service
High-Performance Request Router validated at 10K+ RPS
Designed and built an enterprise-grade request router and load balancer that survives sustained 10K+ RPS with sub-5ms median latency, circuit-breaking, retries, and live WebSocket telemetry.
role: Sole engineer — owned design, implementation, performance tuning, and stress validation end to end.
Outcome
The Problem
API gateways like NGINX, Envoy, and Spring Cloud Gateway are battle-tested but treat the JVM-based runtime as a black box. I wanted a router I could actually reason about — small enough to read in an afternoon, instrumented enough to debug a p99 spike in minutes, and fast enough to hold up under real load.
The goal: prove that a focused Java 21 service could match the latency profile of a Go-based proxy on commodity hardware while remaining easy to extend with custom routing rules.
Constraints
Real engineering is defined by what you can't change. Before writing code I locked in the budget the system had to live inside.
- ›Latency: p99 < 20 ms added by the router itself, p50 < 5 ms
- ›Throughput: must sustain 10K RPS on a single 4-vCPU container without GC pauses dropping requests
- ›Resilience: no upstream failure should propagate to the caller — circuit-break inside 3 failures
- ›Observability: every request must surface in Prometheus + a live WebSocket feed for the ops dashboard
- ›Footprint: < 256 MB resident memory after warm-up; multi-stage Docker image < 200 MB
Approach
I split the router into three planes that each could be tuned in isolation — a routing core (regex-matched rules with three load-balancing strategies), a resilience layer (circuit breaker, retry with exponential backoff, token-bucket rate limiter), and an observability plane (Actuator + custom Prometheus collectors + a WebSocket fan-out for live traffic).
Spring Boot 3.5 on Java 21 unlocked virtual threads for the upstream HTTP client — the single biggest unlock for sustaining concurrency without thread-pool tuning gymnastics. I leaned hard on records, sealed interfaces, and pattern matching to keep the routing core small (< 800 LoC) and verifiable by reading.
- ›Routing core: regex rule matcher → strategy interface (RoundRobin / Random / LeastConnections)
- ›Resilience: per-route circuit breaker (Resilience4j semantics, hand-rolled to avoid pulling another DI graph) + retries with jittered backoff
- ›Rate limiting: token bucket per client IP, lock-free atomic accounting
- ›Telemetry: Micrometer → Prometheus, plus a Sink that fan-outs every request to a WebSocket topic for the live ops dashboard
Tradeoffs
Engineering is the act of choosing what to give up. These are the deliberate tradeoffs I made and why.
Linear, debuggable code with the throughput of async — without coloring every method.
I wanted full control over the state machine for instrumentation, and to avoid a 3MB transitive graph.
Live tail is what ops actually opens during an incident; metrics are for trend analysis.
Alpine kept the image under 200 MB and every base shipped CVEs at the time — JRE was the meaningful surface.
What I'd take with me
- 01Virtual threads aren't free — `synchronized` blocks pin them and silently murder throughput. Audit your hot paths before you assume the runtime fixes everything.
- 02Live ops feedback (WebSocket tail) reveals more in 30 seconds than dashboards do in 30 minutes. Build it for yourself; you'll keep it.
- 03Hand-rolling a circuit breaker is 200 lines and forces you to understand failure semantics you'd otherwise paper over with a config flag.
- 04If you can't read your own service in an hour, neither can the on-call engineer at 3am.
Happy to discuss tradeoffs, run the code live, or whiteboard the next iteration.