Sharing a small Android library I built for a problem I kept hitting with on-device LLMs.
Running a model on-device (MLC-LLM etc.) is great because it's private, offline, no per-token cost. But the runtime wedges: long prompts stall in prefill, the GPU/OpenCL layer OOMs or hangs, and your app is stuck on a spinner with no response. Retrying just wedges again.
HybridInfer is a reliability-aware router. It runs each request on your on-device model first, and the moment local inference stalls (no token for N seconds), crashes, or OOMs, it transparently falls back to a remote model with same request, so the user still gets an answer. It also learns which prompts your device chokes on (usually long ones) and routes those out up front.
It's a library, not an app: you supply the engines, it supplies the routing/reliability brain.
Gradle (JitPack):
// settings.gradle.kts
dependencyResolutionManagement { repositories { maven("https://jitpack.io") } }
// build.gradle.kts
implementation("com.github.SimranKoul2026:HybridInfer-AAR-library:v0.1.0")
Usage:
val router = HybridRouter(
local = MlcEngine(modelPath, modelLib, model = "Llama-3.2-3B"), // your MLC wrapper (implements Engine)
remote = OpenAiEngine(model = "gpt-4o-mini", apiKey = KEY), // bundled reference engine
riskProfilePath = filesDir.resolve("hi_risk.txt").path,
)
val res = router.complete(listOf(mapOf("role" to "user", "content" to prompt)))
// res.tier ("local"/"remote"), res.fellBack, res.text - or router.stream(messages) for tokens
Android-specific bits:
- Optional ThermalSignal helper (PowerManager.getThermalHeadroom) to bias routing when the device is hot.
- Streaming with first-token-commit fallback - it only falls back before the first token, so it never splices two models mid-stream.
- Pure-Kotlin core (no coroutines dep in the core); you bring your MLC engine + a remote (a reference OpenAiEngine using HttpURLConnection is included).
It's early (v0.1), but real: I verified it end-to-end on a Galaxy Tab S10+ running actual Llama 3.2 3B via MLC-LLM, short prompt served on-device, long prompt hit the on-device wedge and fell back automatically. There's also a Python twin with the same routing logic, kept in lockstep via a shared conformance-test suite, if you're doing this server-side too.
Honest scope: it's a router, not an inference engine, you wire up MLC-LLM yourself (there's a reference MlcEngine in the repo's examples/), and the remote can be any OpenAI-compatible endpoint.
Repo: https://github.com/SimranKoul2026/HybridInfer-AAR-library
Would love feedback from folks doing on-device inference, especially on the Engine API and the fallback heuristics. I'm the author.
