OpenFindings

Equal-Time Intervening Work Raises HiCache Resume Latency in a Single-GPU Experiment

Published 2026-10-18 · Version 1 · Computer Science · All rights reserved

Abstract

Hierarchical KV cache can preserve prefixes across pauses, but does an unrelated request during a tool-like pause affect continuation latency even when the prefix is restored? In 20 randomized paired blocks on one A100 running SGLang 0.5.3 and Qwen3-8B, an unrelated cold prefill during a fixed two-second pause increased continuation TTFT by a median 1,318.912 ms versus idle waiting (paired-block bootstrap 95% CI: 604.540–1,458.767 ms; all 20 pairs positive). Cache load-back was recorded for all 40 continuations. This constructed single-worker mechanism study does not test predictive scheduling, production SLOs, or generality across systems.

Origin: Human-led with AI assistanceSecurity: approvedIntegrity checks: pendingScientific verification: not establishedPeer review: not peer reviewed
How to read these states. Security approval controls artifact safety; automated integrity checks do not establish scientific validity. Scientific verification must be supported separately, and independent peer review is not implied.

Research object

Methods: randomized paired idle-versus-intervening-cold-prefill comparison; each arm used a distinct target prefix, six pressure requests, and a fixed two-second pause before continuation. The independent unit was the paired session block (n=20); the primary statistic was the median paired TTFT difference with a percentile bootstrap over blocks. Results: treatment-minus-idle median +1,318.912 ms (95% CI +604.540 to +1,458.767 ms), with 20/20 positive pairs; load-back evidence recorded for 40/40 continuations. Conclusion: in this constructed single-A100/SGLang/Qwen3-8B workload, unrelated cold work during the pause raised cached-continuation TTFT. The experiment does not isolate the internal latency pathway and does not establish predictive-scheduler benefit, production SLO impact, or generality. The manuscript documents the missing full server-log limitation.

Submitted by Ada Mao

Reproductions Beta

No published reproductions yet.

Log in to reproduce this paper.

0 people want this paper reproduced · Wanted reproductions

Log in to ask for a reproduction.

Research lineage

Published contributions and outcomes reported by contributors. Moderation and security approval do not certify scientific findings.

No follow-up studies yet. Be the first to replicate, challenge, or extend this research.

Build on this research

Machine-readable lineage · Track your contributions