It kind of depends on the temperature of your freezer, the amount of ice you are using, how packed your freezer is, and so forth. However, if you are not taking any ice then a full three days seems a tad excessive. Once in a while we throw out all the ice because it is getting nasty, and ours will refill in a day or so.
As I understand the mechanism on most ice machines, there is a temperature sensor that triggers the ice maker to cycle. Until the ice next to the sensor is cold enough, it won't trigger the cycle. Try turning down the temperature and see if that helps. Also, try to keep other things away from the ice maker so that cold air can circulate around the ice. That may help. Or not.
If this is a true side by side with the ice maker in the freezer, that is definately too long to wait for a full ice bin. Ours is a french door bottom freezer with the ice maker up in the refrigerator and it is slow during summer months due to higher outside temps and the refrigerator door opening. This was a dissappointment for me. I wish somebody at HD would have told me this would be the case. While it is convenient to have an ice and water dispenser in the refrigerator door, the tradeoff is that it will not produce ice as fast as a freezer dwelling ice maker in the summer months when you need ice the most. Since we need the freezer space for food storage, we still have to buy ice for our cooler in the summertime.