In my experience looking at Louisiana homes with brick veneer (during hurricane structural evaluations), I noted that most of the time the brick was held on by thin, easily removed streel brick ties. If that is the case with your house, you can generally easily remove the brick with prybars. If I were you, I would take the brick down gently, since you can usually get some salvage value for old brick, but if you really don't care, a demolition hammer will certainly get the brick off. Just watch out you don't damage the sheathing, unless you don't want to save it either.