2042912 - LlamaRunner buffers the first decoding token, distorting time-to-first-token metrics

Status: RESOLVED DUPLICATE of bug 2005365

(Core :: Machine Learning: On Device, defect, P3)

Description

[Joe Bowser]

Description:
Bug 2005365 added inference metrics emission from LlamaCppPipeline.mjs,
including timeToFirstToken. The JS side computes this as the time
between runStart and the arrival of the first non-prompt chunk.

The native side at LlamaRunner.cpp:104-148 buffers output tokens
until minOutputBufferSize is reached (or the phase completes / changes)
before pushing a chunk to JS. As a result, on the prompt→decoding
phase transition, the prompt buffer flushes but the first decoding
token is appended to a fresh response and held until the buffer
fills with minOutputBufferSize tokens (default 20).

Effect: timeToFirstToken actually measures "time to first batch
of minOutputBufferSize decoding tokens". On small models or short
responses (e.g. the TinyStories-656K test fixture, which emits all
14 output tokens in a single buffered chunk), nearly all wall-clock
time lands in TTFT and decodingTime collapses to microseconds,
which also inflates tokensPerSecond to nonsensical values.

Fix: force a flush on the first decoding token of a run, regardless
of buffer fill. One extra IPC message per inference; negligible
overhead. Sketch:

// capture a mutable `firstDecodingFlushed = false`
bool isFirstDecodingToken =
    chunk.mPhase == /* decoding */ && !firstDecodingFlushed;
    
if (response.mTokens.Length() >= bufSize ||
    response.mIsPhaseCompleted ||
    isFirstDecodingToken) {
  if (self->MaybePushMessage(mozilla::Some(response))) {
    if (isFirstDecodingToken) firstDecodingFlushed = true;
    response = mozilla::dom::LlamaChatResponse();
  } 
} 

Identified by Valentin Pollet in review of D302656:
https://phabricator.services.mozilla.com/D302656#inline
(the "Time To First minOutputBufferSize tokens" comment on line 343)

See Also: Bug 2005365, D302656.

Comment 1

[Joe Bowser]

Attachment: Bug 2042912 - Flush the first decoding token eagerly in LlamaRunner so JS can measure TTFT accurately. r?valentinp

LlamaRunner buffers output tokens until minOutputBufferSize is
reached before pushing a chunk to JS, which made the timeToFirstToken
metric emitted by LlamaCppPipeline (Bug 2005365) actually measure
time-to-first-batch rather than time-to-first-token.

Force a flush on the first generation token of each run, via
PushMessage so it lands regardless of whether a consumer is
currently pending. Subsequent generation tokens continue to follow
the normal buffer-fill / phase-completion flush policy, so IPC
volume is essentially unchanged (one extra message per inference).