Milford is my go-to recommendation for young families. I lived in the Devon section for a while but moved to a bigger house in Orange, which is also a great town.

Milford has lots of family activities and a great downtown. Unfortunately both Milford and Orange do not have much housing inventory. Good luck!