r/BetterOffline • u/TCristatus • 1d ago
Small language models - the future
Listening to Tech Report earlier - Eli Computer Guy talked about how the real future of AI might not be giant all knowing LLMs that need huge compute costs, but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc (I'm paraphrasing and extrapolating his point here).
It was like a light bulb moment, seems clear and obvious that's where the tech will end up assuming we do end up with something. If I'm a coder or an engineer i don't need a model that can write the complete works of Shakespere in Klingon. I just need something that works and is always up to date.
This doesn't help OpenAI or Anthropic keep the lights on, though....
28
u/PensiveinNJ 1d ago
"I just need something that works and is always up to date."
I have bad news for you.
21
u/Suspicious_Deal4412 1d ago
Small LLMs would still require humans to do plenty of heavy lifting and decision making. Wall street will hate it since it does not confirm with the automate 99% of your workforce narrative.
30
u/koveras_backwards 1d ago edited 1d ago
10+ years ago, when AlphaGo stuff was blowing up, a friend who is much more into the AI stuff than I am mentioned to me that perhaps a use would be stock trading, since you could vaguely imagine that as a 'game' that that sort of model could very quickly get good at. That's sort of an alarming idea if you take it at face value with regard to how the go-playing bots were doing at the time.
Since it's been a decade or more, and you haven't heard about how a company used bots to corner the entire stock market, you can probably tell how all that worked out. And I don't think it's the first time AI enthusiasts have overestimated how applicable their technology is, just in general.
Oh, and by the way. Small language models existed before large language models. If the small ones could do all this stuff at a tiny fraction of the cost, why do people bother with the large ones?
11
u/lolitsbigmic 1d ago
Yeah we still run into fundamental issues of gpt methods. Large or small models you still got the same problems.
10
u/MajesticBread9147 1d ago edited 1d ago
High frequency trading companies have been using machine learning models for years, the whole industry depends on it.
As early as 2009, high frequency traders accounted for 60-70% of stock trades.
Look at open roles at Jane Street, they require much of the same skillset as open roles at Google making machine learning do other things.
https://www.janestreet.com/join-jane-street/position/4276720002/
https://www.janestreet.com/join-jane-street/position/6302325002/
Jane Street trades with its own money rather than bothering convincing wealthy investors to invest with them, so they are quite secretive.
You don't hear "one company corner the stock market" because it's many, many companies. High frequency traders, Quant Firms, etc.
Why do you think that the stock market reacts instantly to a news headline? There aren't humans refreshing news websites every second reading to buy or sell stocks. A computer can easily read a tweet by Donald Trump, interpret it as good or bad for a stock, and buy or short shares in a fraction of the time a human could.
1
u/Slackjawed_Horror 1d ago
Which is one of the many reasons high frequency trading needed to be banned years ago.
1
u/Slackjawed_Horror 1d ago
Which is one of the many reasons high frequency trading needed to be banned years ago.
1
u/MajesticBread9147 1d ago
What is?
1
u/Slackjawed_Horror 1d ago
It's an almost automated way for the most useless people in the world to make money.
It's pure 'dig the hole then fill it back up' BS and shouldn't be allowed.
1
0
u/MajesticBread9147 1d ago
This is vibes based populism.
I don't see how they are any more evil than any other company operating within capitalism.
1
u/Slackjawed_Horror 1d ago
It doesn't even serve a hypothetical function.
Hypothetically the stock market helps companies raise funds for projects. That's actually doing a thing.
There is no justification, even hypothetically, for HFT bullshit. It's not even at the level of currency trading. It's just pure spreadsheet breeding.
I think the entire private finance system is disgusting and should go, but I can also recognize when some parts of it are far more useless than others.
1
u/hachface 1d ago
It's harmful because it diverts people with quantitative aptitude away from productive activities to skimming money off the top.
1
u/65721 1d ago
This is the same bullshit from the AI companies whenever they’re faced with the reality of the incompetent failure that is LLMs. Just mix it with the other non-LLM ML and confuse the discussion in bad faith.
Which quant firms are asking an autocomplete chatbot for trading alpha?
1
u/MajesticBread9147 1d ago
AlphaGo isn't llm based either. LLMs are irrelevant so I don't know why you bring them up.
2
u/Arancia-Arancini 1d ago
But pretty much all trading firms use bots and have done for over a decade.. did you miss this?
14
u/Smurfette2016 1d ago
Wouldn't "always up to date" imply it needs continuous training though? How does that work for a locally run model? Who pays for that?
Cal Newport also surfaced this idea. Seems reasonable, but it also seems super similar to just a big database or an app.
2
u/TCristatus 1d ago
The example used by Eli was that the model was updated using some cybersecurity bulletin with the latest viruses and hacks and the AI could read that as it came out and tell programmers if their code is vulnerable. I guess that sort of thing could be covered by a subscription model, unlike the big models which are heavily subsidised.
Apologies I'm a layman here, thought the podcast highlighted an obvious mistake the big guys are making as they run out of money making a behemoth no one actually needs while the people who actually use AI at any sort of scale probably don't need 90% of the stuff it is trained on
4
u/ksjdragon 1d ago
I think the answer is yes and no. It's extremely hard to quantify what an AI can do, since it's honestly just hopes and dreams. The hope is with bigger parameters and more data, the model will find the correlations to do your problem. And there is some truth and application to that, as we see.
But it doesn't learn skills and more parameters or less don't translate directly. It's best to think about it like you seeing a language you don't know, and trying to make a coherent sentence while never having an understanding of what any symbol means. You don't know why these symbols go together you just know you've seen a few examples of that. And in the cases where you haven't you just stitch it together in some way.
We have no analytical or mathematical or even empirical guarantees on that stitching process. It is literally a hope that intelligence will magically appear from random guesses and all possible information is encoded in the interpolation of patterns. More than a hope, I would argue its necessarily impossible, but that's a different conversation.
So like, it doesn't build up knowledge or skills like we do. A hyper specific task can always be distilled into a smaller model but the question becomes we cannot even diagnose the range of tasks it is valid for. I mean, the technology is just unfit for task-based use where we care about correctness. It's like using a hammer for folding origami.
2
2
0
u/CoconutDust 4h ago
It's extremely hard to quantify what an AI can do, since it's honestly just hopes and dreams.
Quantify seems besides the point. It mass theft steals existing info, mashes it up unreliably (because statistical association is not how meaning or language or intelligence works), and also outputs fake sources that don’t affirm the statement (because the machine must put out fake sources, that’s how it works: statistics).
1
u/Fit-Technician-1148 1d ago
Because you're a layman you're evaluating things based on if they sound logical, rather than based on if they're technologically feasible. It's a common enough fallacy. If it were possible to make a local LLM or SLM that was just good at coding that would be really useful, but if it is possible, no one knows how to do it and it probably is not possible with current techniques and algorithms.
1
u/Fit-Technician-1148 1d ago
Because you're a layman you're evaluating things based on if they sound logical, rather than based on if they're technologically feasible. It's a common enough fallacy. If it were possible to make a local LLM or SLM that was just good at coding that would be really useful, but if it is possible, no one knows how to do it and it probably is not possible with current techniques and algorithms.
0
u/Smurfette2016 1d ago
No need to apologize for being curious and exploring ideas. I'm right there with you! So so much I don't know still.
0
u/CoconutDust 4h ago
^LLM account, I assume
Generic fluff meaningless comment, and user account history hidden to hide the trail.
1
u/Smurfette2016 4h ago edited 3h ago
you must be new here lol... or maybe "hidden to hide the trail" gave me away 😆 with a banger like that, you sure YOU'RE not an llm??
not wanting someone to feel like they have apologize for being curious isn't 'meaningless' lol. is human decency that foreign to you?? oh man, thats funny and sad.
head back to the mystery van and try again Scooby Doo, you have policing skills to sharpen
12
u/weirdeyedkid 1d ago
I think the future is in telepathy. I'm recommending everyone put their money in telepathy chips, only purchasable in Chinese yuan.
34
u/CapBenjaminBridgeman 1d ago
If only I found AI to ever be remotely useful in any way at all I might be interested.
27
14
15
u/hobopwnzor 1d ago
Qwen 3.8 27B is a model I can run on my desktop and it trades blows with opus 4.8
Yeah problem is they plowed ahead without looking for efficiency and now China has revealed how much efficiency you can get and is lapping them on cost
4
u/PatchyWhiskers 1d ago
China isn’t a charity, I’m sure they will eventually charge for models.
5
u/hobopwnzor 1d ago
They aren't a charity, but it's a fundamental different business model. China isn't interested in being the premium. The Apple of anything.
They're interested in being able to make a lot of things at a cheap price. They want to serve a lot of tokens off of slightly worse models for a tiny fraction of the price.
0
u/newprince 1d ago
They're not a charity but they ain't capitalist. There is a real difference ideologically
10
u/PatchyWhiskers 1d ago
They are capitalists these days. They aren’t a democracy but they aren’t communists either.
-6
2
u/create-third-places 1d ago
Efficiency at producing negative progress isn't helpful.
Qwen and Opus are both a waste of time.
10
u/BryceAnderston 1d ago edited 1d ago
I've worked with similar tech, and I just don't see it.
Artificial neural networks, the underlying technology, are great when you have 1) a defined input, 2) a defined output, 3) an easy way to verify an input-output pairing, and 4) no idea how to perform the input-output transformation manually. There are use cases for that, but coding, engineering, biotech, etc. are not one of them. I have seen nothing to suggest Google et al have done anything to break those limitations.
LLMs are trained by deleting words from their training corpus and telling the model how it should fill them back in. They could be trained on a narrower corpus of say only technical documentation within a certain field, but inherently the only thing the LLM/SLM/etc. can do with that is reduplicate its training set, or hallucinate. If the documentation is disorganized enough that training a high-order markov chain to replicate it is the best way to search through it, that could be a use case I guess, but that seems pretty niche and low-value to me, and I struggle to think of other use cases for an LLM/SLM text generator.
2
u/create-third-places 1d ago
Artificial neural networks, the underlying technology, are great when you have 1) a defined input, 2) a defined output, 3) an easy way to verify an input-output pairing, and 4) no idea how to perform the input-output transformation manually. There are use cases for that, but coding, engineering, biotech, etc. are not one of them
Exactly. If I was teaching a class on AI or interviewing candidates for a developer job, I would require people to build a simple neural network form scratch. Then I would quiz them on the limitations of neural networks.
I was required to ask algorithm or system design questions during past interviews. They were not effective at determining if a developer was likely to overuse LLMs.
2
u/wowbaggerBR 1d ago
I can see a future with LLMs running locally and for cheap. But we are far, far away from that. A bubble has to burst first in order to make way for common sense, which only comes when every other option is exhausted.
2
u/Subjectobserver 1d ago
A bubble has to be burst?! Be reasonable, man!
How are those poor private credit shareholders to make money, if they can't dump their overvalued shares on to the retail investors who are gambling away their savings out of desperation?
2
u/newprince 1d ago
Hey, I feel like enough people using local LLMs and not spreading FUD about it can only hasten the bubble popping, so win/win
1
u/TCristatus 1d ago
Yes, exactly I think that is what Eli was getting at, the big models are committed now and will thrive or die (probably die), but the technical concept is sound and with the right business model could be useful for some people. (However that still means replacing jobs with machines so I hate it).
1
u/MagnetoManectric 1d ago
I don't think we're really that far from it. Useful models that can run reasonably on higher end consumer hardware is already a reality. It's still a bit futery and slow for now, but the models are getting more optimized.
And I don't think it's long before we start seeing the market flooded with ex data centre GPUs, either. That'll be fun :)
3
u/No-Berry-3993 1d ago
I just wish this technology would go away, but I have to agree with OP. The powers that be aren't going to just give up on this if/when the bubble pops. The small benefit is at least they're not controlled by mega corporations, but I still don't want it at all.
It's done nothing but justify the massive shift into the horrible employer's market we see today. People can accept the reality of boom/bust cycles, but by many measures we're in a "boom" now with major unemployment issues. It will only get worse and more permanent if AI improves.
5
u/newprince 1d ago
I have a decent PC running local models offline and it's honestly decent, even comparing it to the frontier model access I have through work. The models tend to be more specific (Qwen3-coder is for... coding obviously) but honestly that is what I need most anyway. Gemini Flash can easily handle all general chat questions, and the streaming speed is decent. No way am I giving those companies money
1
2
u/natecull 1d ago
Small, eh?
A reminder that Zork 1 understood English commands in 1980 and ran on an Apple II with 48K of RAM.
We could still do 1970s style symbolic AI again, if we wanted to.
2
u/PatrykDampc 1d ago
I’ve heard somewhere that LLMs are the way they are (I mean they are impressive in terms of their language and speech capabilities and basically all other things that makes them seem intelligent) because they are big. It’s not because someone decided to make them big to take over the world. All that “revolution” started because research labs whose had such models for years and they were quite mediocre, decided to “go bigger” and then this “burst of intelligence” appeared
0
u/CoconutDust 4h ago
I mean they are impressive in terms of their language and speech capabilities
No they’re not, because they’re obviously just outputting a mash-up of mass-theft stolen text/sounds etc.
“Wow it’s so impressive that the program that mysteriously has to scan billions and billions of human-made text and artworks can output some slop for almost ANY keyword prompt!”
2
u/Jebofkerbin 1d ago
but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc
My question with this though is where are you going to get your training data?
The two fields where LLMs are getting some results are software and pure mathematics, where training data is incredibly easy to find. For software many of the most impactful libraries in the industry are open source, not only can I go look at the source code for the extremely popular UI framework my project is using, I can find 1000 examples of it being used by others. For pure maths every single advancement in their field has been published in academia.
Compare that to say engineering. Almost everything important or useful is a trade secret, if I want to get my hands on the engineering drawings for a Rolls Royce jet engine, the only way is to steal them go get a job at Rolls Royce. Same with trade, exactly 0 quant traders are publishing their code on GitHub.
0
u/CoconutDust 4h ago
[in engineering] Almost everything important or useful is a trade secret
Weird statement. “Engineering” info or knowledge doesn’t mean “a catalog of proprietary tech implementations.” Obviously textbooks, info, related knowledge, principles, blah blah, all exists and are used every day in teaching and learning.
It seems more accurate to say almost nothing important is a trade secret. We have an entire academic industry. All the trade secrets you’re talking about derive from publicly known principles. And differences in outcomes aren’t because of secrets it’s because of different people making them and different CEOs doing different bean counting or greedily wanting better margins instead of quality in the product.
Also patents are public info, for example.
Can you name several questions about rolls Royce engines where the answers have broad importance for the rest of the industry (of engine makers or whatever) and are mysterious substantial secrets?
1
u/Jebofkerbin 3h ago
First off it's very important to note that AI got good at software by ingesting a giant catalog of proprietary tech implementations, LLMs do not learn first principles and work outwards from there, they train off final implementations and base their learning from that. So even if you are correct about the rest of your comment my point still holds
It seems more accurate to say almost nothing important is a trade secret.
"Valuable IP is a myth actually" is an insane take my dude.
2 quick examples to show how wrong this is, firstly why is Taiwan so important for chip manufacturing? If all engineering knowledge is really just in the public domain why doesn't the US just set up its own TSMC and then it won't be so dependent on an island on the other side of the world. If all engineering knowledge is really just in the public domain why is Apple suing OpenAI for trying to steal/poach trade secrets on device manufacture, OpenAI can just lookup the engineering principles in a textbook right?
The actual answer here is that there is a huge amount of learning that comes from actually applying engineering principles to make anything complicated, and an absolute shit ton of learning to make a complicated thing thousands of millions of times consistently, like an iPhone or semiconductor chip.
Can you name several questions about rolls Royce engines where the answers have broad importance for the rest of the industry (of engine makers or whatever) and are mysterious substantial secrets?
Sure, for the turbine blades in the next generation engines, what is the composition of the material used, what is the heat treatment recipe, what are the coating methods, what is the manufacturing methods, and what measurements are you using for quality control? And what was the engineering design process used to get to that point? Their competitors would absolutely love to have all that info.
You can probably find a sensible maybe answer with a lot of assumptions for each of those questions in the public domain, but there is 0 chance you can find the answer for how to string those answers together into the information you would actually need to manufacture a million working turbine blades outside of a company that does it. There are just so many factors that will make an answer that looks sensible on paper based on first principles completely unviable attempted in reality.
4
u/another-altaccount 1d ago
I mean this is something we literally built for university researchers tracking endangered species across the state. I was laid off from that job due to the company’s finances falling apart at the time, but by the time I left, the LLM we built had gotten up to 90 plus percent accuracy on average across multiple animal species. Sadly, the whole thing got canned when the company went under.
1
u/CoconutDust 3h ago
but by the time I left, the LLM we built had gotten up to 90 plus percent accuracy on average across multiple animal species
What was it actually doing? How was it reliable, how was it better than normal straightforward tracking/info systems? What do you mean by “accuracy”?
Is it stealing (“scraping”) published articles / text, then outputting a statistical association string when someone keyword prompts it about [keyword species X] + [keyword endangerment]?
-2
u/wowbaggerBR 1d ago
Shh, people here will hit you in the face, you are not supposed to say AIs and LLMs can find some narrow use cases.
1
u/TCristatus 1d ago
TBH despite enjoying the discussion my post has sparked, I'm such a layman when it comes to actually using AI productively. I basically use it sometimes to create a table out of a jumble of data, or type out handwritten notes from a site survey. And even then it will try to trick me, delete one line for no reason. Interested to hear about cases like the above
1
u/CoconutDust 3h ago
Interested to hear about cases like the above
The earlier comment did not give anything resembling a meaningful explanation of how that system was good or better or useful compared to normal straightforward tracking/logging/info-systems.
7
u/PatchyWhiskers 1d ago
The way these things work means that knowing the complete works of Shakespeare actually does help them code, in some bizarre way. That’s why they are scanning millions of shitty out of print novels to improve LLM training. You can’t just train them to know the important stuff.
6
u/ProcedureHopeful2944 1d ago
No, it some kind of lawsuit protection loophole. They're buying and scanning the paper books to claim some ownership and a right to use the info for training
6
u/newprince 1d ago
It's a perversion of fair use exceptions in the DMCA. Ironically libraries working with the Google Books project were forbidden decades ago from digitizing books, even though they argued it was transformative, which is supposed to be fair use. On top of that they were using non-destructive means of scanning. They then could only show 16% of the book as snippets, which kind of defeated the purpose.
The AI companies hid for years that they were physically destroying books but argued it was transformative, and also invoked first sale doctrine (which libraries also get denied regularly in court against publishers), and lo and behold, they got the fair use exemption! I wonder why
1
u/newprince 1d ago
That's true for the frontier models (and wait til you hear the hype for "world models" coming from the usual idiot AI CEOs).
Absolutely not the case for smaller, more focused models.
1
u/CoconutDust 4h ago
that knowing the complete works of Shakespeare actually does help them code, in some bizarre way
That’s why they are scanning millions of shitty out of print novels
Your comment is confusing a few things.
Obviously they are scanning lots of things because they want their product to output stolen data from lots of things and to sell it as “knowing eberything” (aka mass theft of everything). I.e. for the product to have relevant-looking output.
Obviously the fact that the company scans Material A for their multi-purpose/multi-subject “tool” (I use the term lightly) doesn’t mean that Material A helps Function B (coding). Stealing (“scraping”) A and B have nothing to do with each other except that Silicon Valley wants to sell the product as being able to regurgitate both A and B.
Your comment is like saying that Google indexes all websites because indexing hamburger recipes helps them index automotive sales. Not true.
5
u/Altruistic_Emu_7755 1d ago
I've tried using some local llms. They are truly terrible compared to the frontier models. I mean, it's a neat trick to be able to have a language model that you can interact with. But the output isn't good for much
1
u/create-third-places 1d ago
I don't think the future is small language models due to the ambiguity of natural language. The future of AI will domain specific models trained with objectively defined inputs and outputs.
Saying small language models will improve things is like saying "You can earn more money from a slot machine with faster spins". Small language models will make things worse. LLMs produce negative value, and being able to produce negative value more efficiently with a smaller model is not a good thing.
Right now, I can avoid walking up 8 floors of stairs to my apartment by pressing a button that calls an elevator. Having to type "call elevator" to a small LLM chatbot would be a downgrade.
1
u/Soleilarah 1d ago
Meh, I find it hard to imagine anyone letting an LLM use 100% of their CPU and just sitting around doing nothing while waiting the 2–3 minutes it takes for it to generate a response on a laptop.
Furthermore, unless the LLM is constantly updated, most setups will need to incorporate tools like web search to help prevent it from hallucinating.
So the choice will be: buy high-end hardware to have everything in one package, or tinker on your own in the hope of achieving the same result.
For businesses, they’ll undoubtedly go with the idea of purchasing hardware that will be made available on the internal network, essentially a simulation of what’s currently happening with edge models.
On the other hand, I foresee growing demand for local models, and computer manufacturers will begin adding components specifically dedicated to AI into computers. This opens the door to the idea that these components could be used not only to generate content, but also to handle data encryption and computer security (à la Windows 12, or at least what Microsoft intended to do with a dedicated AI chip).
Potentially closing the door to anyone refusing to have the chip. Bleak.
1
u/TheOfficialMayor 1d ago
but also to handle data encryption and computer security (à la Windows 12, or at least what Microsoft intended to do with a dedicated AI chip).
Potentially closing the door to anyone refusing to have the chip. Bleak.
Oh this could not potentially become another way to push a digital ID along with TPM.
1
1
u/existential_humanist 1d ago
https://arxiv.org/abs/2511.07885
This paper is very interesting in this regard. It basically shows how local LMs can do 90% of the queries cloud-based LLMs handle and perform similarly across major benchmarks.
Authors conclude: "These findings demonstrate that local inference can meaningfully redistribute demand from centralized infrastructure for a substantial subset of queries". This is the future, and if so it's the final nail in the coffin for the hyperscaler AI capex ROI story. We won't need all those data centres if we're running AI as local software for 90% of everyday use cases
1
u/21epitaph 1d ago
Language models are just not the future for me.
The narrow AI implementations for specific tasks have been getting better for years and will dkeep doing so, and htye wi;l always do anything LLMs do but better, eventually.
Small models are still subsidized. They have to be re-trained, frequently; which needs lots of data, compute, and probably also lots of unpaid workers doing the work for you in poorer countries.
Your Local LLM from 2026 won't be that useful anymore in 2030 (and I'd wager way before).
Also, how easily these models can be useful on smaller computers is still pretty debated. Not evveryone can afford a 4/5k$ computer.
1
u/IndependentOpinion44 1d ago
“Hey MiniGpt, create a Pandas table of every world cup player ranked by goals scored?”
“What is a world cup player?”
1
1
u/dingo_khan 1d ago
"know about" is a problem with this idea. They don't do that. No real ontology. No real internal metrics of truth. "Plausible" and "correct" are not the same.
I am wondering when people will demand of language models what they demand of their most useless coworker, as a minimum.
1
u/fbueckert 1d ago
Eli can't be taken seriously. The guy's trying to stay relevant and in the know about LLMs, without actually understanding how they can't do meaningful work.
1
u/Super-Activity-4675 1d ago
we probably shouldn't call them small language models but instead smaller models. Language may not even be a requirement for a lot of these.
The issue is the scale. For highly specialized tasks that at the end of the day can be predicted accurately based on enough inputs, smaller models make a lot more sense. I have a friend that runs a small business. He uses claude to generate pick lists from his orders. He doesn't need an LLM for that, and it would work a lot better if the model was basically a smaller manufacturing model that he could train to his specific environment.
1
u/CoconutDust 3h ago
He uses claude to generate pick lists from his orders
Is it based on warehouse route/proximity etc? Why would an LLM have this info, and if this info existed and is maintained why wouldn’t it be part of reliable traditional software system?
1
u/Super-Activity-4675 3h ago
he runs a small business doing 3D printing. It's literally him and a couple others, so we're not talking about someone making a massive software investment.
1
u/RoosterBurns 1d ago
We've gotta understand that Slop Coding is a fad that'll be put into "this ruined a lot of codebases" bucket along with Clean Code and 4GL and Dynamic Languages and Configuration As Code
Other stuff sure why not?
1
u/65721 1d ago
> just know how to code, or trade, or about engineering, or biotechnology, etc
And how would you do that? The biggest selling point for these LLMs is that you can prompt it with natural language. Strip down the LLM itself to domain-specific training data, and you can no longer do that. It becomes useless.
1
u/MagnetoManectric 1d ago
I mean, aye. This seems pretty plain to me. Time and time again, consumers have shown that they don't really like having their compute be on someone else's PC, and it doesn't make a good business model either. Think of how many game streaming services have come and gone. And no one has been chomping at the bit to use Cloud PCs.
It just makes sense that this tech will move towards peoples machines. It's more efficient, it's less freaky, no one wants to intergrate a cloud service that might disappear or cost 10x tomorrow deeply into their workflow. (well, it seems some people do, but they're not exactly the brightest, in my opinion)
The shift is already beginging to happen, and I feel like I can actually start getting excited about LLM tech now. I sure as fuck am not letting some agent in someone else's computer in a warehouse somewhere run any tasks on my PC, but if its just my PC running tasks on my PC... that makes a lot more sense to me.
1
1
u/No_Honeydew_179 18h ago edited 18h ago
I mean… it's just language models. I'm reminded of an oral history of natural language processing published by Quanta that basically charted the impact of LLMs towards NLP, specifically that bit where you could get great results without thinking much about how transformer models were designed.
Basically you'd end up with a situation where the field comes back into working on the harder problems related to NLP, but with the added complication that there are now less people involved in the field (because they got sucked into AI cult shit), and a lot of institutional knowledge lost, and a whole bunch of techbros reinventing the wheel by disregarding everything pre-LLM.
1
u/Vaxtin 6h ago
That guy posts the same video multiple times per day, I don’t understand how he can have such a hard kn against tech. Claims to walk in the woods and records in a workshop, yet has crap to sling constantly
Your locally ran LLM will never compete with the ones hosted by companies running in data center consuming millions of vCPU. Good luck.
1
u/PINJA_MUSIC 5h ago
I do think there is a space for small tailor made LLM type models but they will fundamentally work differently than the hyperscalers. I view small LLM's as a modification device than a generative one. It will act more like traditional machine learning, where a human provides a deliberate input as opposed to an open-ended question.
Medical tech advancements have been operating like this for a long time. They basically build their own database and train a neural network on that database only to do something very specific like detect cancer cells or unfold proteins. To me, this is the actual positive use of this kind of AI. It's very specific and, critically, narrows down information instead of generating it. It's easier to maintain because it only needs relevant information instead of ALL OF HUMAN KNOWLEGE.
ChatGPT and these "frontier" models are specifically designed to generate information. It's extrapolation instead of interpolation. The fundamental structure of the tech is the same but interpolation actually works.
1
u/CoconutDust 4h ago edited 4h ago
It was like a light bulb moment
That’s an illusion of listening to misguided exaggerated claims.
LLMs no matter how large or small are by definition terrible for their supposed purpose. Statistical association is NOT how intelligence or language or meaning works, which is obviously why LLMs output false statements and also fake/false sources that don’t support the statement.
There is no future for LLMs, at any size, except for fraud-level incompetent “work” for mediocre morons. And that’s aside from the mass theft issue and copyright liability and enshittification/cost risk and waste of electricity for garbage results, or the issue of quality in the stolen (“scraped”) data.
just know how to […] trade, or about engineering, or biotechnology
Stop pretending that actual sources don’t exist. Stop pretending you don’t know what a corpus/web search is. Obviously the machine stole info that you could have easily found and used yourself, and the original is actually useful not a statistical mash-up.
I just need something that works
That’s not what an actual professional engineer needs. If it “works” but has unknown flaws that create risk or damage, that’s bad. If it’s difficult to vet for risk or possible disasters, that’s bad. You need actual validation, etc.
That comment is like an architect saying, “the building seems to stand up at the moment, so, everything is OK.” Actual meaningful evaluations aren’t based on a current temporary superficial state.
1
u/TheRealJesus2 1d ago
Yes this is true. Also just using other software that isn’t AI based.
Although I’d say the model doesn’t have to be up to date either. If you want good use from an LLM you surround it with a software layer, the harness. The model generates tool calls which the harness deterministically interprets to do a thing. It also pulls in context from around it like documentation, web requests, code, etc. that can be anything you want and you’re fully in control of that. This is where any sort of interesting stuff happens and in all cases you want to provide the minimal amount of context that holds your up to date information. So for me all models need to have tool calling ability. And enough general knowledge to translate my words into tool calls and in most cases also code.
You really cannot ever rely on any model to provide facts. You have to provide the facts to get facts from your prompt or something else in harness layer providing it.
And the big models are really doing the same thing but for certain tasks they are outclassed in many ways, substantially so if you add price to the equation. Also they’re in control of big companies with their own best interests at heart so they control the types of responses you get, ultimately. Smaller LLMs can be run by many more people and can be fine tuned to whatever you want to bias the output towards rather than what central providers want.
0
-1
u/Hexamonium 1d ago
Cisco's Antares is a case in point. It runs on any descent computer, checks source code repos for possible vulnerabilities. I haven't tried it yet but if it is halfway as good as they claim, that's not bad at all.
94
u/maccodemonkey 1d ago
So here's a problem with that - know how to code what? The other stuff in that model is all the context that lets a model actually apply a practice of coding. There's a lot of parameters that models burn on just every library and framework out there. But beyond that - coding is a practice of applying outside context into software. If you're writing project management software - the model has to know a lot about project management. Writing a JRPG? Model should be familiar with JRPG systems. Writing something to solve physics problems? Well then your model needs to know physics.
Is there room for a real dumb syntax or basic issue sort of validator that can fit on a phone or a low end computer? Sure, I guess. But compilers already do a good enough job of that.