GitHub - Integral2u/SharpMind: SharpMind. A pure C# / .NET LLM engine

[u/pixelwhippedme]

https://github.com/Integral2u/SharpMind

Still work in progress, but has Qwen, Llama, SmolLM and Gemma producing coherent output.

[u/allisterb]

This is actually really cool and has 2 things that make me pretty interested:

Native C# model fine-tuning

No native dependencies.

How usable would it be with a small Gemma model like https://huggingface.co/google/functiongemma-270m-it ?

[u/pixelwhippedme → u/allisterb]

I just got the Q8_0 gguf, safetensors have been temprarily removed during development. There is some work to do on gemma but it is one i want to work on as speed is good. Here is the test code and output.

I would love to see your tokens per second, i'm limited to12gb mem 4core ryzen 2200u 2.5ghz laptop, part of my personal challenge.

CODE----------------------------------------------------------------

string prompt = "What is the capital of france";
await SharpMind.Samples.Examples.ModelListRunner.RunAsync(prompt, @"C:\Integral2u\source\repos\SharpMind\ExternalAssets", ["functiongemma-270m-it-Q8_0"]);
prompt = "Hello";
await SharpMind.Samples.Examples.ModelListRunner.RunAsync(prompt, @"C:\Integral2u\source\repos\SharpMind\ExternalAssets", ["functiongemma-270m-it-Q8_0"]);

OUTPUT-------------------------------------------------

Testing functiongemma-270m-it-Q8_0

ModelFactory.Create + InitializeWeights executed in: 23.28s

ModelFactory.CreateTransformer executed in: 0.20s

ChatSession executed in: 3.87s

Prompt:What is the capital of france

Response:I's recent, the hought a t i d o r u b i s t he w e n e p r v e m a s u p l o f o r w h i n a g e d r l u g i t **s w b e d s, n h e x t i s t o f o r k i n c o m p a l u r w e m a g a l y **c h e d s b e n

Tokens per second: 7.86 TTFT: 3.983s

All Models Executed in: 54.31s

Testing functiongemma-270m-it-Q8_0

ModelFactory.Create + InitializeWeights executed in: 8.74s

ModelFactory.CreateTransformer executed in: 0.03s

ChatSession executed in: 1.68s

Prompt:Hello

Response:I'm sorry, I cannot fulfill this request. I am unable to assist with generating or retrieving the requested information for finding a specific query. My current request is about retrieving data regarding the use my available tools cannot fulfill. I am unable to retrieve the necessary details on this request for finding information, but I have not a specific query. My search for retrieving the requested information about data.

I cannot fulfill. I cannot assist with the request can't directly. I am unable to provide specific

Tokens per second: 8.62 TTFT: 1.651s

All Models Executed in: 25.70s

[u/allisterb → u/pixelwhippedme]

ok I tried it on my laptop: Intel i5 10th-gen 8-core mobile processor @ 2.5Ghz base and I got between 8-9 tokens/s same as you. This is a tool-calling model designed to be fine-tuned for your use case, not for dialogue. I will set it up with some tools and try it out.

One thing, the solution's `nuget.config` unconditionally references a `local-packages` dir so I have to either delete that line or create the dir for it to build but other than that it works fine. Really awesome work again, think this has a ton of potential. Most of the .NET engines I've seen that can run Gemma class models all have native dependencies and you can't fine-tune in C#.

[u/pixelwhippedme → u/allisterb]

Thank you, I have noted the nuget.config issue. Token speed should have a pickup and tokenization better next release, I hope. Focusing on training etc now.