One of the O-rings in a cartridge has most likely broken and is now letting water through while the valve is in the "OFF" position. It may be the hot side or cold side, but cartridges are pretty cheap or even free (depending on manufacturer), so replace both of them.