First before I forget, let me caution you to be careful with simple things like towel rack and toilet paper holders. I cannot remember all the times I have encountered a mystery leak putting a pinhole in copper plumbing installing such things.
Sadly, if your anchor is not biting into the drywall because it is too loose for the hole you made, your situation will get worse over time as it works around and abrades the drywall.
It sounds like you now have a hole too large for the plastic anchors shown. A metal expansion anchor might work. As suggested a butterfly toggle is a possibility unless your bracket calls for two screws. You will not have room for two. A butterfly has wings that fold as you slide it through the hole and that unfold when clear on the other side. You then draw up the bolt to cinch the fastener. Overkill for mounting a toilet paper dispenser but not much else will work on a large hole. There are all sorts of hybrid variations:
To be honest though, if you have problems with the toggles you may have to cut out a small piece of drywall, secure some blocking for a patch and for the mounting bracket. Tape, mud, prime and paint.
You can patch the hole with dry mud but it wil not be strong enough for plastic screw anchors and screws.
I am sure it is of minimal comfort but this happens more often than you might think so do not be too hard on yourself. I have used thousands of drywall anchors of different kinds hanging art for galleries over the years and every once in awhile the situation got away from me in a hurry. All you can do is curse briefly and move on!