Madison resident here. Like people have said, similar towns. Moved here about 2 1/2 years ago, but I grew up a few towns to the east. Both are amazing. The house we wanted most was in Guilford. I would have been ecstatic to move there. Great schools, solid sports, lots of outdoor activities. Guilford has more shopping and restaurants. Madison really lacking in the restaurant department. Guilford is more politically divided. The blues are bluer and the reds are redder. Madison used to be red town - old school New England GOP. That has faded and it's now centrist blue town. The high end in Madison, down by the beach, is higher end than Guilford, but that doesn't represent most of the town. Lots and lots regular people here.

The snobby reputation is a bit overblown. Growing up in less bougie town year ago, Madison and Guilford both seemed liked snotty rich towns. Living here now, I don't find that to be true. OK. Let me clarify. If you are a blue collar person who works on their car in your garage and runs the engine loud to blow it out during dinner time, you might have neighbors get an attitude. There are a few people, for sure. Probably the worst we encountered was a non-native who grew up in Waterford, married a doctor, and acted surprised that my wife and I, with our normal jobs, can afford the town. But that was literally one thing in over 2 years.

You can't go wrong with either.

EDIT: Taxes are a bit higher in Guilford. The gap will close a little once the bill for the school renewal and community center hit the budget. Should still be a bit lower than Guilford.