Sounds like you didn't nail the unit even with the sheetrock when you installed it. It might be easier to remove the moulding, reinstall the unit even with the sheetrock then reinstall your moulding. This would require you to cut away the sheetrock at least the width of your shelf thickness.
Other then that, you did well.
Actually, I didn't nail the unit in at all. It's just sitting on a shelf inside the closet. I wanted the shelf unit wider than the hole in the wall to try and mask some of the cables a little (I put 2 outlets in the shelf, on the rear of the right side).
For now, I bought a plastic j-channel to cover up the exposed drywall edge, and I will put a piece of quarter round on the top edge. Thanks for the help.