radiance

–ms

decode step

–tok/s

decode

–tok/s

prefill

–

requests in flight

–

kv cache in vram

–%

acceptance

Requests

nothing in flight

Batch

decode prefill waiting preempted free

Admission queue

–

Memory

–%

of prompt tokens from cache

–

conversations can resume

–tok

context in vram

–

moved to host ram

Will resume most recently active first