Before spending any time or money fighting or remedying the situation, I would talk with one of my local planning and/or zoning officials. I am not an attorney, and I don't play one on tv (nor on the internet), but it seems to me that if there is anything tangible on which an attorney could build a case, it will be ordinances that your municipality has in place, and these are the people who you are already paying to enforce them.