Originally Posted by fabrk8r
I think whoever suggested a water heater check valve is correct.
This is an example...not an exact match, but very similar. How old is your water heater?
Based on the notion that the part in question is similar to this (i.e. part of a check valve) here's my theory that fits with what the OP stated happened...
The water heater had a check valve, but it was faulty, stuck in a partially open mode, therefore hot water did flow, but at a reduced pressure. Eventually, the check valve completely failed and the spring that should be holding the valve against the gate on the cold water side instead fired the valve backward into the cold water lines, likely when there was a drop in the pressure differential accross the check valve when cold water started running. This piece of the valve got caught up in the cold water flow and began lowering the water pressure to a faucet when it tried to transision to a 1/2" line. This "peg" has a density close to that of water, so it was easy for it to move around when ever the water wasn't on pushing it in a particular direction, hence the reason the problem moved from faucet to faucet.