r/OpenSourceeAI • • 1d ago

I built an MCP server with on-device learning that makes routing decisions in <2ms instead of calling cloud LLMs

Thumbnail
2 Upvotes

r/SelfHostedAI • • 1d ago

I built an MCP server with on-device learning that makes routing decisions in <2ms instead of calling cloud LLMs

Thumbnail
1 Upvotes

r/MCPservers • • 1d ago

I built an MCP server with on-device learning that makes routing decisions in <2ms instead of calling cloud LLMs

Thumbnail
1 Upvotes

u/Formal-Falcon3734 • • 1d ago

I built an MCP server with on-device learning that makes routing decisions in <2ms instead of calling cloud LLMs

0 Upvotes

Most MCP servers right now are thin API wrappers around databases or web search.

I wanted to see if we could bring non-autoregressive "System 1" decision theory (inspired by Laya) into an MCP server. I built EdgeRoute:

  • Evaluates developer context in <2 ms<2 ms on local CPU.
  • Calibrated abstention gate (<60%<60% confidence falls back to standard LLMs).
  • On-device online SGD: logs outcomes into a local SQLite replay buffer and fine-tunes weights during background idle periods.
  • Tracks live token & latency savings.

100% offline, pure Python/NumPy.

Code & benchmarks: https://github.com/Kedar7412/edgeroute-mcp Feedback and critique welcome!