Since you've already spoken with him and he's not doing anything to remedy the issue, and the issue is preventing you from enjoying your own land, then yes, contact the town/county. He's the one violating the law. You're doing nothing wrong by notifying authorities.