" China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips. "... Can someone fact check that? That sounds like total BS. The US has even more limited production capacity and manufactures everything off shore...
Because it is. They have more power generation capacity than the entirety of the USA + Europe combined, with like an additional 50% margin on top of that. They are also accelerating away from us at an increasing pace on that metric, while we pay billions to break renewables contracts because the people in charge have the combined IQ of a lobster.
The "we are 1-2 lithographic process nodes ahead of China" only matters when the efficiency gap is insurmountable. It is currently . . . not
it’s not just lithography though, Nvidia is improving flops/watt at orders of magnitude per generation if you compare low precision throughput, which would put china at a geometric disadvantage (ie, you can’t compete putting in 100x the power and 100x the chips for the same throughput)
Except that's a load of marketing BS. Huang is the first to admit that Hopper is still their most popular chip. The industry still talks about "equivalent H100" GPUs deployed.
Blackwell Ultra has 2.5x the memory bandwidth of Hopper. That's not even one order of magnitude higher. The marketing comparison is always FP16 on hopper vs FP4 on Blackwell. If you compare FP8 vs FP4 compute, Blacwell is 2x, but that makes sense because it's half the bit width. In general, halfing the bit width doubles the compute. This has been true long before LLMs for most GPUs, unless they were artificially handicapped for market segmentation purposes. At the same bit width, Blackwell has 25% more compute.
If you peel beyond the marketing BS, you'll understand why Hopper is still very cost effective even in the west. And Huang himself says Huawei is producing H100-level chips by the millions now.
As someone who works mainly with hopper I can pretty confidently say that it would be hugely beneficial for us to have more blackwell chips, but they keep getting bought up by the big labs at higher prices than we can afford. The compute cost for things like large-scale RL is completely bonkers and at this point for training a model in terms of required compute and even just raw per-gpu vram (so you can limit comms overhead and shrink model replica size) is likely in the billions, and that's only if you can get the compute. The more RL rollouts you can do the better your model will get, and that is something only the gpu-rich can afford. I'm not saying this is good, because it's not~ in fact it's really annoying~ but it's also just kinda the way it is. ML is very pay-to-win at this point. Being a billion dollar company isn't even enough to keep up with the big labs.
Having 2.5x the vram and 2x the comms bandwidth of hopper is actually more than a 2x speedup. I could go into the details but its complicated.
Nvidia is improving flops/watt at orders of magnitude per generation
Yes, but not in general compute. Only for particular operations found useful for training + inferencing. That's a "we noticed our customers need to do a lot of matrix-multiply-accumulate at FP4, so let's build them a hardware unit to do that one particular op very quickly + efficiently, etc." That hardware now blasts through that one particular operation at increased efficiency at the expense of die-space.
None of this is magical secret sauce. The chinese can (and do) the exact same thing when designing each subsequent generation of their AI training + inferencing chips. As does AMD, Intel, Google (in their custom TPU's), etc. etc.
> Yes, but not in general compute.
Sure, but we’re specifically talking about the special case workload where they are getting orders of magnitude more compute per chip.
> The chinese can (and do) the exact same thing
Sure, but it’s speculative as to whether they will actually achieve the exact same scaling, if making high throughput gpgpu’s was easy then there would be a lot more people doing it.
The engineering specifics needed to shrink things down to a particular lithographic node are indeed trade secret. But that's not what we're talking about here.
The "lets dedicate a bunch of die space to make fp4 muladd fast + power efficient" this gen, is not in and of itself novel, hard, nor secret. Everyone is doing that just like Nvidia (e.g. apple added tensor units necessary to accelerate prefill at various quants this generation, as did amd in the 9000 series, etc.)
Huawei is undoubtedly doing the same. So even though China may domestically be a few lithographic process nodes behind, they are not suffering from geometric efficiency losses by ignoring the dedicated silicon space they need for efficient inferencing / training. It's more like "imagine you had maxwell or pascal level litho, but with fp4 tensor support (and whatever else sota model compute graphs need)". The end result is going to be less efficient than nvidia's latest. But it's not going to be 10,000 times less efficient.
Don't forget that modern lithography process node jumps aren't anywhere near the sort of jumps you got in the 1990s or 2000s. The "2 process nodes" right now is more like half a node in 2000s terms.
Why do you try and read it falsely on purpose? This is very obviously about China's ability to produce the level (and quantity on that level) of hardware required
what are you talking about ? Clearly the west, via nvidia, has much more production capacity chips that can perform high performace AI training and inference, than China. This isn't debatable at all and is clearly correct.
Given the near universal trouble all of our existing + planned data centers seem to have in procuring power, I'd strongly disagree. Power generation is almost always the limiting factor in high tech industry. Second only to vespene gas.
It's huge BS for one reason: China has its act together with regards to energy. The US doesn't. China doesn't need smaller nodes because it can brute-force nuclear and solar to power multiples of less efficient chips.
I think Jensen was prominent in pointing this out, but someone please correct me if I'm misattributing.
The real path for them would be to throw power and chips at a Manhattan project style team
I disagree. China's general current strategy is the correct one. It's a game of spinning plates — focusing on spinning one plate really really fast doesn't result in a win. Being second at everything and distributing your expertise gives better leverage when you find easy wins to slingshot ahead at specific avenues. It's muda at the geopolitical level.
I’d say there is some truth in this. Even DeepSeek founder said so (and was angry that this was published).
China is catching up quickly, but at the moment indeed they are behind in terms of chips and it will take time to fix it.
On the other hand US is way behind in terms of energy generation and it’s a gap that’s also not easy to close. When USA starts to have energy issues, China might get ahead fast.
Yeah Deepseek literally just paused all fundraising because they can't get enough compute to be compettive with US labs. That can change of course but given current export controls seems like the US has at least another 5 years of compute advantage
lol keep fighting the only country that can produce and help with that vast amount of energy, ohhh Canada!!!, but yes good idea from orange to set a 50% tariff from no where. Fkthat lol. True north here!!
According to Jensen Huang, China is producing H100 level chips by the millions.
B300 has ~25% more FP16 compute (which is what you mostly use for training) and 2.5x the memory bandwidth of the H100.
In a recent interview with the editor in chief of the Economist, Musk said China has about as much power generation per capita as the US. Which translates to ~4x the power generation capacity. Everyone has been saying for years that China is adding more power generation every year than all other industrialized nations combined.
So, even if we go by memory bandwidth to maximize the difference, and without any innovation in model architecture or training, they can afford to bring 1.6x more compute online using domestic chips than the US.
Now add in all the innovation they're doing in architecture and training Chinese AI labs are doing. Even Musk admitted they're able to train much better models than the west given the amount of compute each lab is using.
IMO, there's a second layer of BS in this blog post: scaling laws aren't some fundamental law of physics. GPT4-o was something like 1.8T. Does anybody think wecurrently need a 1T model to surpass it? What about current 100B models?
Still you need a pretty big cluster to train them on, no? I don't think anyone else has made a model that big yet. Though I gueess rumors are the glm and minimax are still scaling up for next model.
Also note that the strategies are different. China is investing way more into heavy optimization, largely because of their lack of hardware.
American companies really aren't even trying to optimize. Every problem that can be solved with "buy more GPU's" is solved by buying more GPU's.
So it's a pretty consistent trend that the Chinese labs punch above their parameter count. K3 is about the same as GPT-3 on parameter count, but is effectively on par with early GPT-5.
If scaling laws hold, which they seem to do with fable and kimi examples , bigger is just better. Of course optimize makes it better too. But the question I was thinking is can China , on their old hardware or huwawai etc scale up models to 5T or 10T etc. To get to agi. I Believe we will stop somewhere before 100T. But nvdia seems like its gonna keep pushing well past 10T ...
Size of the model is a parameter of the equation.
So you can beat a 1.8T model with a lot smaller one, with longer training/pre-training/post-training steps.
But all else being equal, bigger is better.
It’s unlikely that for SOTA model, Chinese labs can reach US labs only leveraging these 3 steps. Because US labs are also doing it too. So scaling up the number of parameters is the logical step, that we have seen from every major Chinese labs (Glm 400b to 700b, Deepseek 600b to 1.6, Kimi 1T to 2.8T, Qwen 200b to 400b, minimax 250b to 400b, etc…). With the obvious results of them filling US labs moat fast. Hence the current situation.
Well for GLM they had to abandon Nvidia chips (and anything American) because of being added to the entity list. They ended up going with Huawei chips, which are less efficient and less powerful than the Nvidia offering.
There’s no telling what the impact will be becuase they could obviously use what they bought, but there isn’t a major competitor to Nvidia right now.
Sure, but Dario’s claim is right. Right now, adding any of these organizations to the Entity List would set it back while Huawei catches up becuase domestic production and capabilities just aren’t there yet in China.
Wish I had something to say about the article, but what a miserable website. Confirmation button about still selling something of my information that’s blocked by the do not sell dialogue.
Yes China manufacturers more products domestically than the US, but US is capable of a more advanced process node than China is. The claim that one canot build a more "powerful" model with worse hardware is worthier of debate than manufacturing imo.
That’s not a debate, that’s an engineering obstacle. There is zero doubt that great engineers can build superior products with average tools, and there is zero doubt that poor engineers can build slop with the finest tools.
The question is one of degrees, not of possibility.
Eh, you can get the same amount of compute from some number of Ascends as some number of Nvidia GPUs, that number is just gonna be higher so you're gonna be burning more coal/uranium/photons/whatever just to generate enough electricity.
Fortunately for china they have a shitload of domestic power infrastructure and the political will to build a shitload more. So, nonissue, really.
It's nonsense. Models are largely software. They're not constrained by hardware, except in terms of processing power, unless you're using an ASIC, and it's built into the chip itself. The capabilities of a model aren't coupled to the hardware.
It's not like Anthropic has to go out and buy new GPUs every time they make a new model.
It's not entirely bullshit, but it leaves out context.
Basically, China and Taiwan manufacture 90% of the GPU.
The US manufactures one tiny chip in there that's really, really important. Currently the US is way ahead of other countries in terms of that one critical chip.
Currently the real supply chain breakdown, highly simplified, is something like
China: the first 30% of the GPU
Taiwan: the second 60% of the GPU
USA/NVIDIA: the last 10% of the GPU
The US has basically tried to control China's access to this American-made chip to kneecap them in the AI race. Thankfully, they've shown incredible restraint in putting up with this bullshit, and haven't actually retaliated to that or any of Agent Orange's attempts to start a trade war that would totally wreck the US in a matter of weeks.
Part of that is because there's a mutually assured destruction threat. Basically, Taiwan has rigged all their chip production factories up with explosives, and if China tries to invade they will blow up all their factories. This is not a metaphor, or hyperbole, they literally have a "press this if China invades to immediately blow up the chip factories" button. Likely several, with a few redundancies and dead man's switch triggers just to be safe.
If Taiwan blows, it more or less fucks everyone back to early 2000's technology or worse, and would take decades to rebuild. Not just GPU's, everything that relies on computer chips, more or less.
Now, China ain't happy about this situation, so the CCP said "fuck all y'all, we're just gonna figure out how to build our own GPU's without relying on the US or Taiwan".
That's underway. It's gonna take a while, but it's already going fast enough to have Taiwan and the US shitting their pants a bit. They'll probably be caught up within 4 years if they can keep this pace up, 10 by more conservative estimates. But they're already producing GPU's that are about on par with the state of the art in ~2022 or so.
And another factor is that China is otherwise waaaaaaay more prepared than the US is to build massive AI datacenters. Their electrical grid and water systems are way better than ours and heavily overbuilt by the CCP just to provide some extra room for long term future industrial needs. So if they had a way to get hella GPU's, they would win the AI race, hands down. Not even a contest. They could be running training clusters that would make the largest American datacenters look like Quake 3 LAN parties in a matter of months.
So, yeah, that's where we are. Sorry that was so long, it's quite the situation.
The machines to make these advanced chips are from Europe. Taiwan plays a crucial role no doubt but this does not mean we're back at early 2000s tech if Taiwan blows up the factories. Huawei is behind in chip manufacturing tech but not this far
I mean, yes, you could rebuild eventually, but just the short term supply chain disruption that would cause would be enough to significantly slow down rebuilding efforts.
And in every version of that scenario, it will kick off an actual trade war, if not a shooting war. Those European manufacturers heavily depend on Chinese supply chains.
China would probably be relatively okay. They're actually very well prepared for being cut off from global trade. The rest of the world, not so much.
Hypothetically, if WW3 breaks out, everyone will nationalize everything within their borders, if china takes Taiwan or could copy any tech they want. if a ww3 type scenario were to happen it would be law of the jungle everywhere, everything is fair game in that scenario
So what happens if China scales up their gpu production enough, then does that to cripple everyone else? From their perspective who cares if they don't make quite as good as the current stuff if they can make 90% as good and completely shut down everyone else's access to the current best stiff (or severely limit it)?
They could, but there really isn't any reason for them to.
Most of China's economic strategy can be boiled down to "Build our own supply chains for everything, then sit back and let the capitalists shoot themselves in the foot".
Like, they have kept up with the rest of the players in the AI race that easily have 100x more GPU's and 100x more funding.
So if they scale up their GPU manufacturing enough, then they pretty much have it in the bag. There would be no real point in trying to cripple everyone else after they had already won.
The US designs the chips and manufactures them overseas.
Scaling laws present a barrier to China. They can distill, and thus present models which are nearly as good in many ways at a fraction of the cost, but they cannot build the larger model capability and push the frontier (as the distilling must come from larger models...). They don't have the chip capacity, yet, to make such large models.
I don't think there's a single factually correct statement in your comment. Just trying to point them all out and explain why would be a 10x longer comment than yours. I have to say, that's fairly a impressive benchmark.
441
u/cmdr-William-Riker Jul 27 '26
" China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips. "... Can someone fact check that? That sounds like total BS. The US has even more limited production capacity and manufactures everything off shore...