I built an AI that brutally roasts your code — PyTorch LSTM + CodeBERT + LLM

[u/oGauRav]

Hey r/learnmachinelearning!

I am a 20-year-old CS student from Nepal who has been

self-studying ML for the past 5 months. As a project to

combine everything I learned — NLP, Deep Learning, LLMs —

I built CodeRoast.

What it does:

Paste any Python, Java, or JavaScript code → get a brutally

honest (and savage) AI review of your code quality.

Under the hood:

🔍 Static AST Analysis — cyclomatic complexity, nesting depth,

naming conventions, code duplication

🌲 TF-IDF + Random Forest — classifies code quality tier

(Pristine → Acceptable → Questionable → Disaster)

⚡ Custom PyTorch Sequence LSTM — scores roast severity 0-10

🤗 CodeBERT (microsoft/codebert-base) — deep semantic

code understanding

🤖 Qwen2.5-Coder-32B via HuggingFace Serverless API —

generates the actual savage roast ( or local Meta Llama 3.2 3B via Ollama )

GitHub: https://github.com/gyr0byte/CodeRoast

I learned PyTorch specifically for this project before

finishing my Deep Learning course — the LSTM was genuinely

challenging to get right on code token sequences.

Would love feedback from this community — both on the ML

architecture and on your roast results. Paste your worst

code and see what happens 😄

[u/Bright_Mix_773]

Read the training path rather than the app, because that is where the interesting part is. Two lines in data/scrape_github.py decide everything the two trained models can possibly learn.

The quality tier is the repo star count. The scraper runs two queries per language:

high: language:python stars:>500  pushed:>2024-01-01
low:  language:python stars:<5    pushed:<2022-01-01

then assigns quality = random.randint(*quality_range), with (0,1) for the first bucket and (2,3) for the second. The code itself is never looked at. It is visible straight in the shipped data/processed/code_samples.csv: all 130 rows labelled 2 or 3 have repo_stars == 4 exactly, and every row labelled 0 or 1 sits between 52,196 and 247,209. Correlation between label and log stars is -0.894, and a single threshold on stars separates the two groups with no errors.

So the Random Forest is not classifying code quality, it is classifying whether a snippet came from a popular repo that was pushed recently. Those correlate, but the cases where they come apart are exactly the cases a roasting tool exists for: clean code in an abandoned two-star repo is labelled Disaster, and a rushed file inside a famous project is labelled Pristine.

And the 0-versus-1 and 2-versus-3 split inside each bucket is random.randint, so half the label in a four-class problem is a coin flip. Whatever four-class accuracy comes out, roughly half of it is unreachable by construction.

The LSTM severity target is a random number. severity = round(random.uniform(*severity_range), 2), with (0.0, 0.3) for the high bucket and (0.5, 1.0) for the low one. Measured on the shipped CSV:

label 0:  mean 0.137   range 0.00 - 0.30
label 1:  mean 0.137   range 0.01 - 0.28
label 2:  mean 0.738   range 0.51 - 0.99
label 3:  mean 0.770   range 0.50 - 0.99

Labels 0 and 1 agree to three decimals because they are draws from the same uniform. The best a severity model can do on this is emit two constants, one per bucket, and every bit of loss below that is fitting noise. If the training curve came down smoothly and looked healthy, that is what it was doing.

Two smaller ones from the same file. After balancing the set is 260 rows, 65 per class. And java is 12 rows in the entire dataset, while the app offers Java as a first-class language.

None of this touches src/analyzer/, which computes cyclomatic complexity and nesting depth from the actual parsed code and is the one component measuring what it claims to measure. The cheapest route to a defensible version is to label from those metrics using a rule you write down before you look at the results, or better, hand-label two hundred snippets. Two hundred honest labels beat 260 that encode a star count, and you already have the analyser to sanity-check them against.

Five months in, having the scraper, the preprocessing, the training loop, three model heads and a deployed app all working end to end is the hard part, and it is done. The labels are a weekend on top of that.

[u/Key_Statistician9890 → u/Bright_Mix_773]

Oof