I agree with the New Haven and Mystic/Stonington Borough suggestions. New Haven is packed with great food, music, museums, art, culture, and the latter is quintessential coastal New England.

I love Guilford, Essex, Madison especially inbetween, but tough without a car. Sea Shanty Monday’s at the Griswold Inn would be an amazing experience for you.