Case study, 2025. Kyndryl. Details are generalized.
Taking an API gateway’s tail latency apart
A Java gateway sat in front of five services, carrying REST and Kafka-backed traffic. When one dependency slowed down, every response through the gateway slowed with it. The work was making one slow service stop being everyone’s problem.
- What
- Tail-latency work on a Java API gateway in front of five services.
- My part
- Worked on the gateway with the Kyndryl engineering team as a co‑op.
- When
- 2025
- Stack
- Java, Spring, Resilience4j, Kafka
- Results
- p99 from 850 ms to 280 ms, measured on internal dashboards.
- Timeout errors down 85%.
Context
The gateway was on the request path for several downstream services. Tail latency mattered more than the average: a single slow dependency could hold a response open, and under load the requests waiting on it backed up behind it until its latency became everyone’s latency.
This is an employer project, so the write-up is generalized. There is no public repository, and the only figures here are the two that were measured and confirmed: p99 and timeout errors.
Approach
The levers were timeouts, circuit breakers and the queries behind the slowest calls, rather than a rewrite. Each dependency got its own timeout budget, sized from what that call actually needed, instead of one global timeout that hid which dependency was spending the time. Resilience4j circuit breakers sat around the dependencies, so a service that kept failing or crawling was skipped quickly instead of being waited on. And the slowest query paths were tuned so that fewer calls came near their budget in the first place.
Decisions
| Option | For | Against | Chosen |
|---|---|---|---|
| One global timeout | Simple to configure | Hides which dependency spends the budget | No |
| A budget per dependency | Makes the slow path explicit | Needs measurement per call | Yes |
| Retries alone | Hides transient failures | Amplifies tail latency under load | No |
| Fail fast with circuit breakers | Protects callers and the struggling service | Needs a degraded response path | Yes |
Results
p99 went from 850 ms to 280 ms, a reduction of about two thirds, and timeout errors fell by 85%. Both were measured on internal dashboards. No p50 or p95 figures are published.
What I’d change
I’d write the latency budget, the SLO and the rollback criteria into a design doc before changing any code, so the work had a pass line rather than a dashboard to watch. I’d also add retry budgets, so a future change that adds retries can’t quietly bring the tail back.