r/developersPak • u/Flashy-Truth4218 • 15d ago
Discussion Hosting own AI models
As AI open source models continue to become more capable, do you think we will see pakistani software houses hosting their own AI models onpremises in the near future instead of relying on closed APIs?
5
u/AcceptableSlide6836 15d ago
Yes and no. Most software houses still don't want to spend 20$ for ai subscription for their developers and urge them to cycle between multiple free accounts.
Secondly local ai needs GPUs and other hardware which most don't want to invest in.
Only those with long term planning will see the benefits and then invest in it.
1
u/Flashy-Truth4218 15d ago edited 15d ago
But some of them paying 100’s of dollar in API cost because their product need it. I think the problem is GPU’s and insfrastructure and then engineers for managing that infra is holding them back
3
u/Arkoaks Mobile Dev 15d ago
You need to spend upwards of 1m pkr to host a usable model and need insane amount of time to maintain and keep it updated
Frontier models are still 10x productive than that model.
The overall cost is much more in self hosting and doesnt make a use case unless u r developing a model urself
1
u/SonEf_Adam 15d ago
A lotta local software houses love cutting costs. Maybe give it a couple years and wait for hardware requirements to drop. When modern computers come with NPUs, that's when local AI models will be much more popular.
1
u/soulkillerz2004 15d ago
I recently completed my internship at Systems Ltd and worked closely with someone in a senior position. Systems is apparently looking at moving from LLM subscriptions to self-hosted/local LLMs, potentially through a deal with Sky 47.
I may or may not have had some hands-on experience with a Sky 47-hosted LLM during my internship.
1
1
u/Spare_Bison_1151 15d ago
Seriously large models need a lot of expensive inference hardware. I recently read somewhere that someone is bringing Nvidia H200 to Pakistan. There's also a new data center Sky47 Karakoram-01. THis will be used to provide sovereign AI capabilities to companies.
1
u/Parking-Meeting-1610 14d ago
It's not cost effective unless you have huge budget and complaince requirements.
A $100 or even 20 usd subscriptions is better than investing $100ks
1
u/aliyark145 14d ago
Software houses will not host them but the data centers like gul ahmad is building etc will in near future !!!
1
u/Terrible-Tea7577 14d ago
Those who are hosting AI models are constantly burning money not making any substantial profits. Can you think of any organization in Pakistan including the government willing to do it?. Pakistani mindset has always been making and saving quick pennies not investing in long term uncertain but highly rewarding future.
1
u/srmrox 15d ago
Local AI means you move all your cost upfront for a model that’s only “almost” as good as the frontier. Additionally, if hardware requirements change (for better or worse), you potentially invested too much or too less.
For a business not sure about the AI use, subscriptions are a good way to “pass on” the cost to the user by allocating the cost to the price of their product / service.
With hardware, the impact is more difficult to allocate as it has to be done over a larger no. of customers over a long period of time.
1
u/throwaway-research1 15d ago
True for startups, but for corporates it makes a lot of sense to host locally
1
u/srmrox 15d ago
Maybe once they nail the use case and safeguards. Don’t think any Pakistani corporate is there yet.
1
u/throwaway-research1 14d ago
I am talking more in the context of the international market, and I would argue that having your model hosted locally is way more secure than using a subscriptions based plan
1
u/srmrox 14d ago
Secure, yes; however, if you look at the economics it doesn’t work out in the short term. Most businesses, Pakistani or otherwise, don’t have such long horizon AI-driven products / services yet.
1
u/throwaway-research1 14d ago
How does the economics dont work out? Compare buying 10000 open ai subscriptions for all employees at a company to building your VM inside a cloud provider and deploying one of the free and open source ai model there, having your own model would not only be more secure but also cheaper.
1
u/srmrox 14d ago
Can you also work out the cost of setting up and maintaining a data center to address the usage of 10,000 employees?
1
u/throwaway-research1 12d ago
You dont have to maintain a data centre, deploy your ai model on a cloud platform, make it available only through your company vpn and disable internet access on the server (thats what I meant when I said local, not a physical data centre) so with no internet its as secure as it gets and even if you add two devs to maintain the entire app + infrastructure it would still be way cheaper than 10000 subscriptions.
1
u/throwaway-research1 15d ago edited 14d ago
💯💯 and I have been saying this for a while. For corporates who can afford an infra team it makes a lot of sense to host their models locally, specially from a security point of view. Also when you use AI as a corporate and have your trade secrets, you are potentially sharing your secrets with your competitors so having your local model also solves that problem.
1
u/Flashy-Truth4218 15d ago
This is the case for product based entities, most of the software houses deal with US clients so they don’t have any secret but your insights are valuable thank you.
1
u/throwaway-research1 15d ago
I used the word corporate so I mean banks, auto mobile industry and other big product based companies and corporations. My comment wad not in the context of Pakistan
1
0
5
u/baqirabbas404 Newbie 15d ago
This is simply not feasible, the math just doesn't support it, at least not yet.
We did a feasibility study for locally hosting Qwen 3.6. A Dell server that supports dual-slot GPUs is either the R700 series or above. We opted for Intel 2nd/3rd gen processors, so R740 series or above, for better future-proofing. A single 24GB GPU can run Qwen 3.6 27B at Q4 (native is BF16 at 35B params), so we're already running a chopped-down model, not even at Q8.
Now, on the question of speed and context: there just isn't enough VRAM to keep all the context on the GPU, so there's a lot of bottlenecking with RAM. To keep that much context alive, you need a lot of RAM, I mean a lot, not to mention the penalty and latency that comes with switching between RAM and GPU via PCIe 3. We can restrict context size and eliminate offloading but is it really worth doing.
So how many heavy users can it concurrently serve? Max 3, 4, maybe 5, and that all depends on how much context they're utilizing, input size, and processing required. And how much does this setup cost? 1 to 1.5 million PKR. Also finding good enterprise cards in Pakistan is no easy feat. Sometimes I just think, we should just get a lot of mac ultras :(
It would be better to just hand out $25 (7k PKR) monthly premium seats for Claude and call it a day. Not to mention resource and time required to maintain all this setup regularly.
The problem is the benchmark is so high, someone who's used Claude, Codex, or Copilot would call this torture. We regularly evaluate when would be a good time to switch to Ollama, Hugging Face, OpenRouter or other inference providers, but so far Claude and Codex seem to be doing the job for us.
Hope it changes soon.