I went through the same thing....
Use a shop vac to 'suck' up as much as you can....if you take a real small hose....say 1/4" poly tubing like you use for the water supply for the ice maker...put one end up in the hose of the vac...other down the hole...it will get most of it....
Then take the same hose and use the exhaust port of your shop vac.....if you cup your hand over the exhaust..push the hose through your fingers...quite a bit of air comes out....
Warning....you can get a good puff of dust out of that hole....it's messy.
You know you have it cleaned out when you drop the rod in there and you get a nice 'clink' sound. If it's a 'thud', you still have dust down there.