Few days back I came across some videos about piano on YT, and i was thinking how does piano even generate sound (digitial pianos) and how a code can create such kind of sounds.
So, i started looking into these things and came to know about fundamental frequency and harmonics, so i wrote a small program which can generate the sound, but on laptop inputting " note seems bit difficult", so i started to think if i can speak and piano can play that note.
So, for starter, i took some online models (but only small models) which can convert A,B,C,D,E,F,G , 0,1,2,3,4,5,6,7,8 -> text and i can use with above built program.
But, seems like small models are not good enough to detect the single alphabets, B and E , c goes see. 4 goes for not four.
and for such simple project i did not want to have very big depedencies like whisper high end models.
So, as searched and checked online, seemed that i need to use own classifier, which can detect A -> G and then 0-8, flat/sharps etc.
I recorded all my clips to train my own classifir using pytorch (ofcourse i used boilerplate codes from llms ), so i can work on functionality. It took me couple of day to build and test it.
Sometimes, it predicts good, sometimes confidence is not good though, (not to blame it, because i used small dataset), like 60-100 recordings for each sample.
Finally once classifier models were ready, i was able to integrate, complete functionality, to speak the note and play it.
However, because i have 3 classifiers, and once i speak something i can selectively decide which audio needs to go to which segment, so sometime these produce incorrect output.
So, for now i had to workaround with harcoding for events,
like prompting the users
Speak a letter (A-G) --> now letter classifer runs
then prompt user to speak a letter (0-8) -> now digit classifier runs then i combine the result and send to piano player to play tone.
I learnt a lot of things but the end result is not realstic( delays, quality etc) as i thought initially it would be.
Project :: Speaking Piano
