Well, good luck finding any problem areas without primer on the walls. Since drywall is gray and joint compound is white, it can be really, really difficult to see any imperfections. It's best to apply the primer and THEN look for problem areas, etc. Repair those and then sand, wipe off dust, and apply primer to those spots you fixed. What I like to do after the primer dries is mark areas that need fixed by circling those spots with a pencil or attaching some blue painter's tape next to the spots so you remember where they are.