Tons of people arrive on every train. The station and area around it are safe, and there are usually a few people walking from the station to the Yale campus and downtown area after each train.

If the weather sucks, or if you are carrying valuables and are walking around the city at like 2am, then you might want to do the cab to get to where you're going. There is also a free public bus that runs from the train station through the downtown and back though I'm not sure the schedule. Overall though, I wouldn't worry too much, just stick to the main, well-lit streets where there are other people around if it's very late at night.

Because the area has improved so much over the past decade or so, people are building giant luxury apartment blocks and high-end laboratory towers in this area, so there are some construction sites that you may want to use caution around due to sidewalk closures and such.