Scheduler¶
Continuous-batching engine: admits, prefills, decodes, and completes multiple generation requests against one model with a shared KV cache pool.
anllm.engine.scheduler.BatchedEngine
¶
Methods:¶
poll(req_id)
¶
Atomically drain new tokens AND read current status/reason.
Consumers (the HTTP layer) must use this instead of separate drain/status calls: those interleave with pump steps and can strand the final tokens behind an already-visible finished status.
run_until(req_id)
¶
Drive steps from the calling thread until this request finishes.