Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT)

In three linesStream2LLM is an LLM serving system that reduces time-to-first-token (TTFT) by overlapping context retrieval with inference. It handles two modes: append (progressive accumulation) and update (iterative refinement). Evaluation on real workloads shows up to 11x TTFT improvement.

Summary generated by Claude — human-verified