Sounds like you have a loose connection in the box or to the switch. Loose connections can lead to fires or power outages.
If the switch is also loose on the box you may need to install a box extender. The maximum recess for a drywall finish is 1/4", flush if the surface is combustible.
If the switch is also loose on the box you may need to install a box extender. The maximum recess for a drywall finish is 1/4", flush if the surface is combustible.