llamafu / cognisoc

Quickstart

From pub add to your first token

Four steps to run a large language model entirely on the device — no server, no API key. For the full reference, see the documentation.

Prerequisites: Flutter 3.10+, Dart 3.1+. Android needs NDK 21+ (min SDK API 21). iOS needs Xcode 14+ (deployment target 12.0+).
1

Add the plugin

Add llamafu to your Flutter project from pub.dev:

flutter pub add llamafu
2

Get a GGUF model

llamafu loads models in the GGUF format. Grab one from Hugging Face — a small quantized model like phi-3-mini-q4_k_m.gguf is a good first pick for mobile. Bundle it as an asset or download it on first launch to the device filesystem. See the quantization guide to choose a level.

3

Initialise the runtime

Point llamafu at the model file and configure threads and context size for the device:

import 'package:llamafu/llamafu.dart';

final llm = await Llamafu.init(
  modelPath: modelFile.path,      // a .gguf on the device
  threads: 4,
  contextSize: 2048,
);
4

Generate text

Stream tokens as they are produced and render them in your UI, then release native memory:

await for (final token in llm.stream(
  prompt: 'Write a haiku about the ocean:',
  maxTokens: 128,
  temperature: 0.7,
)) {
  setState(() => output += token);
}

llm.close(); // release native resources

Where to next