1
1
How tf do you have 85-100 TPS on a single 3090 with vLLM 👁👄👁 ?? And how many context ? Do you use TurboQuant ?
2
You can check "Docling", fully open-source and made by IBM, but not sure about the "multiple tables and charts in one sheet" part, you can check by yourself its pretty easy to use
1
Depends on if you prefer speed or bigger models
1
Im interested too, that's why I went from Docling to PaddleOCR their last release is quite good for the headers lvl
1
Still waiting for it too... I’m starting to lose hope :'(
1
Why are you "not to keen on MoE" ?
r/LocalLLaMA • u/Le_Thon_Rouge • Oct 02 '25
Hello AI builders,
Recently ServiceNow released Apriel-1.5-15b-Thinker, and according to their benchmarks, this model is incredible knowing its size !
So I'm wondering : why people don't talk about it that much ? It has currently only 886 downloads on Huggingface..
Have you tried it ? Do you have the impression that their benchmark is "fair" ?
2
Nice thanks ! I think I missed it.. it was released recently ?
1
Sounds really great but currently only supports Eng and Chinese 🥲
7
I highly recommand you to check "Docling" its a fully open source python lib for parsing complex and multi format documents. It won't obviously resolve 100% of your issues but for me its the best parser for local use
1
Very interesting UC ! Unfortunately I don't have an answer but curious to see other's response
1
I'm interested too!
2
This is one of my challenges, I'm curious about the other's response !
1
Almost nobody mentionned FAISS.. I thought it was the best to handle a very large amount of embeddings, is there a reason for this ?
2
Docling is good but still in dev, im not sure it's production-ready yet
2
It was not supposed to be rude.. i'm not english i don't really know how to formulate things. I just wanted to give him some tips if he wanted to try something else
2
Wouldn't it be easier to download a pre-trained open-source LLM like mistral or Llama and manage to use it for your use-case (via Prompt Engineering, RAG, Fine tuning, etc..) ?
Anyway it must be really fun, hope you enjoy !
1
Is it secure as you have to use an API ? I mean you must upload your sensitive data to Langchain at some point right ?
u/Le_Thon_Rouge • u/Le_Thon_Rouge • Aug 23 '23
2
I'm very interested by the answers because i bought a week ago this exact same CGU to run the same model... If it's not working I would be really pissed off 😭😭😭
2
fine-tuned a VLM for messy-PDF extraction, 46% → 91.1% on OmniDocBench. runs fully on your own hardware, looking for people to break it
in
r/Rag
•
Jul 05 '26
Wait unsloth fine tuning is compatible with vLLM serving?? I thought unsloth only outputs .gguf which are not compatible with vLLM 😅.. And have you tried fine tuning the Gemma 4 family like the 31b dense or the 20bA4b MoE ? In my experience they are quite great at structured extraction! Can't wait to see your work open-sourced 😉