According to one calculator I found online (being too lazy to do the math myself), if your CAT-3 is 26AWG, then you've dropped one volt at the end of the line when pulling 80mA. This might be enough to make a difference in whether the camera works or not.
You should measure what is the voltage at the end of the run across the conductors when the camera is connected and the system is on. Also, since use the remaining 4 wires as well to carry power, thereby decreasing resistance. (i.e. three wires in parallel for positive, three wires in parallel for negative)
Have you tested with a short length of cable that the camera and rest of system works?