On-device Gemma in Flutter: flutter_gemma and LiteRT-LM

Cloud AI is not the only path. On-device models keep data local, work offline, and cut per-request cost. Flutter’s story here is flutter_gemma plus Google’s LiteRT-LM inference runtime.

Why LiteRT-LM matters

It abstracts hardware differences and aims at GPU/NPU acceleration across Android, iOS, Web, Windows, Linux, and macOS — the same six targets Flutter ships.

Product patterns that work

1. Vision assist — camera frames → short structured descriptions (Gemma Vision style).
2. Task coaching — local multi-step planning without a round trip.
3. Privacy-sensitive dictation/notes — text never leaves the device.

Engineering checklist

• Model download UX (multi-hundred MB) with resume.
• Thermal/battery budgets on mid-tier phones.
• Fallback path when NPU/driver is missing.
• Clear user messaging that processing is local.

Pitfalls

• Do not block the UI isolate on token generation.
• Quantization tradeoffs: measure quality on your tasks, not benchmarks alone.

---

Originally published on FlutterCook. Read the latest version there — that copy is the one kept up to date.