I have done that once with a 4-1/2" grinder. Makes a total dusty mess, but cuts through soft mortar very quickly. I found that if the mortar was decayed, the grinding wheel went right through it. Other places, the grinder had a hard time with the mortar, so I considered that mortar sound, and did not replace it.