New construction issues!
Contractor is bs'ing you because there is no plausible reason, except criminal building practices, but those are rare (legally speaking). There probably is insulation (cheap enough), although draft prevention may be lacking.
If you really have hvac register in the closet, have you confirmed that it is tied to any supply or return, trunk, pipe or tube? Is the connection secure and taped, and is the grill or the control valve closed?
Also, if you have solid closet doors, heat is prevented from warming the closets. At 65 degrees, the room is not warm enough anyway. My front entrance side coat closet is cold also, it faces 2 exterior walls and a basement that is under-heated.