I'm not sure of the area you are getting the low voltage measurement from. You say you are measuring 240 volts at the breaker but at the fuse box one side measures 121 volts and the other one 55 volts and it varies when you wiggle the wire. Is that "fuse box" a fused disconnect at the AC location? If so, it sound like the wire may be loose or the terminal may be burned and corroded. If so, turn the breaker off and break the disconnect connections down for inspection and possible cleaning/repairing as need be.
If instead you are wiggling a wire at the main panel, post back that clarification and be careful in there since the energy level is high enough to do severe harm if you slip up.