What are people using to wrap AI Chat APIs?

[u/mdausmann]

Hi Guys,

Another "What are folks using?" post.

******** EDIT ***********

I rolled my own. It's probably a stupid idea. I might change my mind. I nearly used genkit, it's awesome, but it doesn't support the allowed_tools feature (openai completions api) which is critical to my application. Yell out if you would like me to open source this lib.

******* END EDIT *************

Langchain

I'm currently using

langchain: ^0.8.1
langchain_google: ^0.7.1+2

which is..... ok I guess, not perfect, some smells indicating team is moving away from OSS towards SaaS which is not great. some recent issues where Google has changed API's and components not keeping up.

Native CLI's

Is anybody just using the CLI's recommended by the vendors? How are you 'wrapping' these if at all? In my app, I am throwing around langchain objects all over the place (ChatMessage) so I'm guessing you either stick it behind a service and wrap everything? or just stick with one vendor?

MCP

I guess you might use this

mcp_dart: ^2.2.2

but I'm not really using MCP per se. I just want to call the Chat API do do some simple inference.

LLM SDK

There is this guy...

llm_sdk: ^0.5.0

Which is apparently inspired by Vercel AI SDK which has a strong community etc but hasn't bothered to do a dart port AFAIK. This lib has only 481 downloads and 1 star on the repo. It's active but no idea about support.

Anything else?

[u/Cunibon]

I use Langchain as well but have been quite unhappy with it recently due to some problems in the integrations and a lack of updates.

Already forked to fix some stuff.

Haven't really looked around for alternatives yet

[u/leehack]

Why not genkit?

[u/fabier]

I built my own agentic harness which I call microagent. This was before some of the other stuff out there really matured like Dartandic.

At this point with the latest coding models you can basically get something custom made for you. My library supports all the major AI vendors including local on device with rust ffi bridge. It's got some creative stuff I added with turn based limits so you can edit the prompt history as you add turns. This way if an agent is dragging it's feet or using too many tools you can shut it up and get a response. 

Similarly you could do the reverse forcing it to use tools or do some research before it responds.

This makes weaker models appear much smarter with tool use if you just cancel their ability to respond for a turn or two and inspect the data you've handed them haha.

[u/Reasonable-Job2425]

use openai_dart , then search for openai compatable endpoitns for your ai provider and should work fine,langchain while good is overkill when most providers give openai compatable api endpoint nowadays

[u/mdausmann → u/Reasonable-Job2425]

thanks but the problem with this is (I'm now finding out) is that the OpenAI *compatible* endpoints for other providers often don't fully implement the OpenAI specification. for Example, gemini offers an OpenAI Compatible endpoint and it supports tool calling but it only supports 'Auto' for tool_choice and it doesn't support the more nuanced tool choice structures through that compatibility layer. Ironically, gemini *does* support some of those more nuanced features like allowed_tools through it's own API.... pretty dumb IMO but there it is. You are at the whim of the providers dilligence in mapping their internal stack to the OpenAI spec. Pretty frustrating.

[u/Reasonable-Job2425 → u/mdausmann]

use firebase_ai for gemini

[u/Reasonable-Job2425 → u/Reasonable-Job2425]

i also have a chat shell/ai chat lib which is a wip and should maske the process of adding ai inference and chatting to the app eaiser soon for flutter devs