FAQ
Frequently asked questions
The short answers. For the full API, see the documentation.
What is llamafu? +
llamafu is a Flutter FFI plugin built on llama.cpp for running GGUF large language models fully on-device, on Android (API 21+) and iOS (12.0+). You add the package, point it at a .gguf file, and call a typed Dart API — no server involved.
Which platforms does it support? +
Android (API 21+, NDK 21+) and iOS (12.0+, Xcode 14+). It requires Dart 3.1+ and Flutter 3.10+. It is a mobile plugin, not a desktop or web SDK.
Which models can I run? +
Any model in the GGUF format used by llama.cpp — LLaMA 3, Mistral, Phi-3, Qwen2, Gemma 2, plus code models and vision models like LLaVA and Qwen2-VL. Thousands of GGUF files are available on Hugging Face.
Does it need an internet connection? +
No. Once the GGUF model is on the device, inference runs fully offline. Nothing is sent to a server, so the feature keeps working with no connectivity.
How big are the models? +
It depends on the model and quantization. For mobile, small quantized models (roughly 1–4B parameters at Q4_K_M) are the sweet spot — often a couple of gigabytes or less. Q4_K_M balances quality and size; Q4_0 is smaller and faster; Q8_0 is higher quality if you have memory headroom.
Is llamafu free? +
Yes. llamafu is open-source under the MIT license, and on-device inference has no per-token cost — the compute happens on the user’s device.
Does it use the GPU? +
llamafu uses GPU acceleration where the device and backend support it, with CPU as the reliable baseline. You also control thread count and context size to tune performance per device class.
How is it different from using raw llama.cpp? +
llamafu wraps llama.cpp behind a thin C++ layer and a typed Dart API accessed via dart:ffi, so Flutter developers get streaming, chat, embeddings, vision, tools, and structured output without writing FFI bindings or managing native memory by hand. See the llamafu vs raw llama.cpp comparison for detail.