Features
Everything llama.cpp can do, exposed as a Flutter API
llamafu wraps the llama.cpp runtime and surfaces it through typed Dart. Here is the full capability set, grouped by what you are trying to do. Everything runs on-device.
Core inference
Everything you need to generate text from a GGUF model on-device.
Streaming text generation
Generate token-by-token with a Dart stream, so your UI can render output as it arrives.
Chat with history
Multi-turn chat completions with role-tagged messages and a managed conversation context.
Embeddings
Produce embedding vectors for semantic search, RAG, and clustering — all locally.
Tokenize / detokenize
Direct access to the model’s tokenizer for counting, truncation, and prompt budgeting.
Advanced capabilities
Structured, multimodal, and tool-augmented generation — on the device.
Vision / multimodal
Run vision-language models such as LLaVA and Qwen2-VL to reason over images on-device.
Tool & function calling
Let the model request tool calls, then feed the results back for agentic mobile workflows.
Structured JSON output
Constrain generation to a JSON schema so you get parseable objects, not free-form prose.
GBNF grammar constraints
Enforce a formal grammar during decoding to guarantee syntactically valid output.
Customization & control
Tune the runtime and the model to the device and the task.
LoRA adapters
Load and hot-swap LoRA adapters at runtime to specialise a base model without re-shipping it.
Sampling controls
Full control over temperature, top-k, top-p, and repetition penalties per request.
Context & threads
Configure context size and CPU thread count to match each device class and memory budget.
GPU acceleration
Offload compute to the GPU where the device and backend support it — CPU stays the reliable baseline.
Ready to wire it in?
The quickstart takes you from pub add to your first token.