Generator

Streaming and one-shot token generation loops over a loaded model.

anllm.engine.generator.generate(model, prompt_tokens, *, max_new_tokens=256, sampler=None, eos_token_ids=(), cache=None)

anllm.engine.generator.stream_generate(model, prompt_tokens, *, max_new_tokens=256, sampler=None, eos_token_ids=(), cache=None)