Whether or not the law is on your side, the reality is: you're the new guy and you're disrupting a long established neighborhood in a small town. Nobody is going to take kindly to that -- including the judge, the zoning board, the neighbors, and the town residents.

Even if you win, you're going to lose. Not to mention the small fortune you'll pay in legal fees trying to fight this.