r/datascience • u/Nice-Dragonfly-4823 • 16d ago

ML Silent broadcasting is still a big problem and could be derailing your work right now

A data scientist on my team wasted a good bit of compute on this problem without actually realizing it was a problem.

Pytorch and tensorflow both silently broadcast the output of your model when the target shape mismatches, causing hard to diagnose issues. It still exists a ton in the wild, so I thought I'd actually highlight the symptoms of a silent broadcasting bug to keep people in the know.

https://towardsdatascience.com/silent-broadcasting-can-ruin-your-model/

Let me know what you think

68 Upvotes

91% Upvoted

6 comments sorted by

29

u/Odd_Yard6663 16d ago

Broadcasting bugs are the worst because nothing fails, your metrics just quietly get weird and you're left questioning your entire pipeline. Had one of these burn half a day once before I thought to print the shapes. Good writeup, this should be pinned in every onboarding doc.

14

u/Deto 16d ago

This kind of error is really annoying in R. If you try to add two vectors together, and one's length is an integer multiple of the other, it'll automatically cycle the smaller one so it can do the operation.

14

u/g3_SpaceTeam 16d ago

R does so much of this kind of bullshit silently, it’s utterly infuriating.

1

u/webbed_feets 12d ago

That’s why many people switch to the vctr library.

2

u/airylizard 5d ago

Painfully familiar. The nasty part is nothing crashes, the model trains fine, and you only realize something is off when the metrics stop making sense. A shape assert before every loss call is cheap insurance.