How it works
One thin path: Dart → FFI → llama.cpp
llamafu is deliberately a thin wrapper. Your Flutter code calls a typed Dart API; that crosses
dart:ffi into a small native
C++ layer, which drives the llama.cpp inference engine directly on the device.
Widgets and app logic. Calls llamafu like any other Dart package.
High-level, typed, async. init / complete / stream / chat / generateJson / embeddings.
dart:ffi bridges Dart calls into the native C bridge — no platform channels.
RAII lifetime management, memory safety, and input validation around the engine.
GGUF loading, tokenization, and transformer inference with hand-tuned kernels.
How a GGUF model loads and runs
The GGUF file is memory-mapped and parsed. Weights, tokenizer, and metadata come from one self-describing file — no separate config.
Your prompt is tokenized and evaluated to build the context. Sampling settings (temperature, top-k, top-p, penalties) are applied per request.
Tokens are generated one at a time and streamed back across FFI into Dart, so your UI renders output as it is produced.
Design principles
- →Thin, not clever. llamafu adds a typed API and memory safety over llama.cpp — it does not fork or reimplement the engine.
- →One runtime, many models. GGUF means you swap the model file, not the runtime. New architectures ride the llama.cpp ecosystem.
- →Safe by construction. The C++ layer owns native lifetimes with RAII and validates inputs before they reach the engine.
- →CPU-first, GPU where possible. Inference runs reliably on CPU; GPU acceleration is used when the device and backend support it.