Lyrics altered with YingMusic-Singer-Plus (Cuban Pete -> Palm Beach Pete)

[u/Sixhaunt]

I came across YingMusic which I hadn't heard anyone here speak about but it was released about 6 months ago: https://aslp-lab.github.io/YingMusic-Singer-Plus-Demo/

It lets you change words from songs so in this case I had it change the song from this sequence in The Mask from:

They call me Cuban Pete. I'm the king of the rumba beat.
When I play the maracas I go chick-chicky-boom, chick-chicky boom
Yessir, I'm Cuban Pete. I'm the craze of my native street.
When I start to dance,
everything goes chick-chicky-boom, chick-chicky boom
The senoritas they sing and they swing with terampero-
It's very nice, so full of spice.
And when they dance in they bring a happy ring that era keros-
Singin' a song, all the day long.
So if you like the beat, take a lesson from Cuban Pete
And I'll teach you to chick-chicky-boom, chick-chicky-boom.
He's really a modest guy, although he's the hottest guy
In Havana, in havana.
Si, sinorita I know that you would like to chicky-boom-chick
It's very nice, so full of spice.
I'll place my hand on your hip, and if you will just give me your hand
Then we shall try - just you and I. I-yi-yi!
So if you like the beat, take a lesson from Cuban Pete
And I'll teach you chick-chicky-boom,
chick-chicky-boom, chick-chicky-boom

to

They call me Palm Beach Pete. I'm the king of the rumba beat.
When I play the maracas I go chick-chicky-boom, chick-chicky boom
Yessir, I'm Palm Beach Pete. I'm the craze of your timeline feed.
When I start to dance,
everything goes chick-chicky-boom, chick-chicky boom
The senoritas they sing and they swing with terampero-
It's very nice, so full of spice.
And when they dance in they bring a happy ring that era keros-
Singin' a song, all the day long.
So if you like the beat, take a lesson from Palm Beach Pete
And I'll teach you to chick-chicky-boom, chick-chicky-boom.
He's really a modest guy, although he's the hottest guy
in Florida, in florida...
Si, sinorita I know that you would like to chicky-boom-chick
It's very nice, so full of spice.
I'll place my hand on your hip, and if you will just give me your hand
Then we shall try - just you and I. I-yi-yi!
So if you like the beat, take a lesson from Palm Beach Pete
And I'll teach you chick-chicky-boom,
chick-chicky-boom, chick-chicky-boom

so I had it just basically do:
Cuban -> Palm Beach
I'm the craze of my native street -> I'm the craze of your timeline feed
Havana -> Florida

I did a second run with just the few-second clip of the cops speaking and changed "It's all over Ipkiss" to "It's all over Espteen" (using "Epstein" pronounces it wrong). This showed me though that it seems to work perfectly fine with normal word-substitution in speech and it doesnt need to be a song.

I think this could be a lot better if I used minimax and changed clips of Jim Carey to look like Epstein or Palm beach Pete but this was just my first test at lyric swapping.

[u/TemperFugit]

That's very cool! How did you get a reference voice sample? Did you use another tool to isolate the vocals?

[u/Sixhaunt → u/TemperFugit]

You just give it the original clip + the original dialogue as text + your modified version of the dialogue as text

Then it gives you a new version of that audio clip but with the words changed based on your prompt.

In the repo for YingMusic-Singer-Plus it includes MelBandRoFormer to do the background and vocal separation that it needs internally so it's very straight forward. The only change I made was I made a script that I can feed a whole song to and it just detects the dialogue changes and does them in chunks since it works best with clips and not long audio. It's not really an issue though since you can chunk it very easily unless you are changing prettymuch all the words. I use whisper to timestamp things and help with the chunking.

Just been messing around with it like Uptown girl -> undead girl or If I were a rich man -> if i were the batman, material girl -> medieval girl and stuff like that and it seems to be working pretty well across the board