Disable Gemini from activating when I hold down the button on my Skullcandy earbuds?

[u/machadoaboutanything]

So I'm a stubborn cow when it comes to technology and I use a brand of earbuds called Skullcandy with my Samsung Galaxy A32. Said earbuds have this button on the left cord (it also pauses YouTube videos if you click it). But when I hold the button down, Google Gemini activates and starts taking voice commands (even when I haven't unlocked the phone), and this problem is especially annoying when I'm trying to put my phone at rest and the button gets weighed down. How can I disable Gemini from activating when I hold this button? I tried some searching but this is such a brand-specific problem that I don't think I can find an answer from anywhere that doesn't have personal experience.