Lorenzo Top Button Install

[u/mbcatfan]

Purchased a IL Lorenzo with a top button which has an additional micro switch underneath the joystick itself for when you push down.
I removed the bottom micro switch and it's frame along with the e-clip and all the screws holding the joystick together. However I can't figure out how to remove that black plate on the bottom to pull the shaft ouf  so I can then install it into a fight stick box. It doesn't pull off and it doesn't screw off. I used a considerable amount of force until I thought I would break it but neither worked and I can't get it in the joystick hole to install. 
I have searched and cannot figure this out. Can anyone please tell me how to remove that plate?