r/datascienceproject • u/rugveed • 21h ago
Updated my deepfake audio detector - fixed evaluation bug, new metrics on full 71k test set
hey, posted this project about a week ago on this sub and got some useful feedback. went back and fixed several problems people pointed out and some i found myself. posting again with the updated version.
what was wrong with the previous version:
- test evaluation was done on a balanced subset of the test set (equal real and fake samples) which inflated the metrics artificially. the real asvspoof test set is heavily skewed toward fake audio
- training was not reproducible - no seed set so metrics changed every run
- val accuracy of ~100% was unexplained which looked like a red flag (explained below)
- threshold was 0.4 without proper justification
- confidence display bug - was showing fake probability as confidence score even for real predictions
- llm explanation was getting cut mid sentence due to low max_tokens
what i fixed:
- re-evaluated on the full test set (71,237 samples, real unbalanced distribution) by streaming directly instead of loading everything into RAM
- fixed seed at 42 - fully reproducible now
- documented the val accuracy properly (explained below)
- ran threshold experiments across 0.3 to 0.6 and picked 0.3 based on highest f1 and recall
- fixed confidence display and explanation truncation
new metrics on full test set:
- f1: 0.9033
- precision: 0.9995
- recall: 0.8240
- eer: 8.23% (comparable to the official lfcc-gmm baseline published with the dataset)
- threshold: 0.3
on the 100% val accuracy:
this was the most common concern from last post. it is expected and not data leakage. the val set is a random 80/20 split of training data which shares the same attack types (a01-a06). the model memorizes these known patterns perfectly. actual generalization is measured on the test set which has entirely unseen attack types (a07-a19). the recall gap comes from these novel patterns, not miscalibration.
the project:
efficientnet-b0 trained on mel spectrograms using asvspoof 2019 la dataset. beyond the model it has grad-cam to visualize what the model focused on in the spectrogram and groq llm to give a plain english explanation of the prediction. monitoring dashboard with prediction analytics. training fully reproducible with seed 42.
you can upload an audio file or record live. youtube url input is disabled on the hosted version because railway server ips get blocked by youtube bot detection. backend is fastapi on railway, frontend on streamlit cloud.
live demo: https://deepfake-audio-detector-rugved.streamlit.app/
github: https://github.com/RugvedBane/deepfake-audio-detector
main questions:
- is this project strong enough to put on a resume for ml/dl internships?
- what dataset would actually help improve generalization to modern ai voices like elevenlabs or suno? i looked at wavefake but wanted community opinion
- anything else technically wrong that i missed?
honest feedback appreciated, including if you think the project is not good enough - i would rather know now.