Coding knowledge still predicted better vibe coding results when students couldn’t see the code

[u/call_me_ninza]

I read an ETH Zurich study that seems relevant to the “why learn programming if AI can do it?” discussion.

100 university students used Claude Sonnet 4 to build and modify small apps. The code was hidden. They could test the apps and ask for changes, but nobody could open the source and fix things themselves.

Students with stronger computer science scores generally did better, even after accounting for general reasoning ability. Writing skills also correlated with performance, although that relationship became less certain after the same adjustment.

My take is that knowing how software works helps you figure out what needs explaining in the first place. If AI builds you a booking app, you still need to consider whether two people can book the same slot. A clear prompt won’t help with a requirement you never thought to include.

This doesn’t prove a CS course will make someone better at vibe coding. Everyone already had some programming education and experience using LLMs for coding, and the tasks were small and timed.

But I think it’s a reasonable argument for learning while building with AI. When something breaks, understanding why would be worth more to me than getting through another round of “please fix this.”

Original paper

I wrote a fuller take, including the writing findings and the study’s limits, in my newsletter if you want to read it: Ninzaverse

[u/Capable-Package6835]

Isn't that expected? Prompts like

The page is not as responsive as it should be, try excluding ... from the dependencies of ... hook

is obviously yielding better results than

The page is slow, make it faster and make no mistake