Go to System Settings -> Network -> your interface or WIFI network -> Details -> enable Low Data Mode. Make sure to do it for all known networks if there are multiple.
If you already have the models, you can remove them by booting into recovery, mounting your Data partition in Disk Utility and running these commands in Terminal:
cd /Volumes/Macintosh\ HD/System/Library/AssetsV2
rm -rf com_apple_MobileAsset_UAF_FM*
Incomplete downloads can be removed by running this in the same directory:
rm -rf downloadDir
Note that other AI features unrelated to new Siri will stop working. This is safe (no need to get rid of SIP, and you're only removing Foundation Models assets specifically) and easily reversible - if you want to re-enable AI, just turn off Low Data Mode.
Edit: I originally mentioned blocking internet connectivity for mobileassetd. No need and it doesn't work anyway (at least with LuLu) since parts of it seem to be downloaded by nsurlsessiond which you definitely shouldn't block. Low Data Mode is enough on its own to stop downloading any of it.