As someone who lives in your town and long ago used to do Waterbury->Grand Central->Poughkeepsie several times and have lived in several train station towns (Waterbury/Shelton/Bridgeport/Poughkeepsie), do yourself a favor and drive down to Bridgeport or Samford and take the train from there. Will save you a lot of time.