I am not arguing that zoning is never used irresponsibly. It surely is.

Does your lot have town water and sewage? Gas? If there is town sewage. I don't know about the infrastructure, but I imagine an accessory might not impact the infrastructure in you area. If everyone on your street did, it might require a major overhaul. That is a decision that needs to be made at the local level.

My section of town has wells. A newer subdivision installed 10 wells some years back and everyone else's wells went dry. Now, our zoning requires subdivisions to have on shared well with an impact study done prior. The state rules would allow developers to skip that.

I hear what you are saying, but zoning laws are often there for a reason and it isn't usually to keep people out.