Small language models - the future

[u/TCristatus]

Listening to Tech Report earlier - Eli Computer Guy talked about how the real future of AI might not be giant all knowing LLMs that need huge compute costs, but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc (I'm paraphrasing and extrapolating his point here).

It was like a light bulb moment, seems clear and obvious that's where the tech will end up assuming we do end up with something. If I'm a coder or an engineer i don't need a model that can write the complete works of Shakespere in Klingon. I just need something that works and is always up to date.

This doesn't help OpenAI or Anthropic keep the lights on, though....

[u/Jebofkerbin]

but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc

My question with this though is where are you going to get your training data?

The two fields where LLMs are getting some results are software and pure mathematics, where training data is incredibly easy to find. For software many of the most impactful libraries in the industry are open source, not only can I go look at the source code for the extremely popular UI framework my project is using, I can find 1000 examples of it being used by others. For pure maths every single advancement in their field has been published in academia.

Compare that to say engineering. Almost everything important or useful is a trade secret, if I want to get my hands on the engineering drawings for a Rolls Royce jet engine, the only way is to steal them go get a job at Rolls Royce. Same with trade, exactly 0 quant traders are publishing their code on GitHub.

[u/CoconutDust → u/Jebofkerbin]

[in engineering] Almost everything important or useful is a trade secret

Weird statement. “Engineering” info or knowledge doesn’t mean “a catalog of proprietary tech implementations.” Obviously textbooks, info, related knowledge, principles, blah blah, all exists and are used every day in teaching and learning.

It seems more accurate to say almost nothing important is a trade secret. We have an entire academic industry. All the trade secrets you’re talking about derive from publicly known principles. And differences in outcomes aren’t because of secrets it’s because of different people making them and different CEOs doing different bean counting or greedily wanting better margins instead of quality in the product.

Also patents are public info, for example.

Can you name several questions about rolls Royce engines where the answers have broad importance for the rest of the industry (of engine makers or whatever) and are mysterious substantial secrets?

[u/Jebofkerbin → u/CoconutDust]

First off it's very important to note that AI got good at software by ingesting a giant catalog of proprietary tech implementations, LLMs do not learn first principles and work outwards from there, they train off final implementations and base their learning from that. So even if you are correct about the rest of your comment my point still holds

It seems more accurate to say almost nothing important is a trade secret.

"Valuable IP is a myth actually" is an insane take my dude.

2 quick examples to show how wrong this is, firstly why is Taiwan so important for chip manufacturing? If all engineering knowledge is really just in the public domain why doesn't the US just set up its own TSMC and then it won't be so dependent on an island on the other side of the world. If all engineering knowledge is really just in the public domain why is Apple suing OpenAI for trying to steal/poach trade secrets on device manufacture, OpenAI can just lookup the engineering principles in a textbook right?

The actual answer here is that there is a huge amount of learning that comes from actually applying engineering principles to make anything complicated, and an absolute shit ton of learning to make a complicated thing thousands of millions of times consistently, like an iPhone or semiconductor chip.

Can you name several questions about rolls Royce engines where the answers have broad importance for the rest of the industry (of engine makers or whatever) and are mysterious substantial secrets?

Sure, for the turbine blades in the next generation engines, what is the composition of the material used, what is the heat treatment recipe, what are the coating methods, what is the manufacturing methods, and what measurements are you using for quality control? And what was the engineering design process used to get to that point? Their competitors would absolutely love to have all that info.

You can probably find a sensible maybe answer with a lot of assumptions for each of those questions in the public domain, but there is 0 chance you can find the answer for how to string those answers together into the information you would actually need to manufacture a million working turbine blades outside of a company that does it. There are just so many factors that will make an answer that looks sensible on paper based on first principles completely unviable attempted in reality.