It’ll be dirtier, but I’d shave it off with a concrete grinding wheel on a grinder. Cinder block is kinda brittle. I’d hate to have a chunk snap off below the concrete level. Shaving it should be less stressful on the cinder block. I’d also see about filling in the holes.