Lived in Sturbridge for awhile years ago, and found myself going into Southbridge for most things. It's really not that bad from what I remember, though apparently there are now smells from a water treatment plant that weren't an issue when I was in the area though.

Personally if I were looking for a first house in towns surrounding Worcester, I'd try to hold out for a bargain in Oxford or Leicester. Or if you want a more city-type life without being in Worcester, Leominster. You can still find houses for less than $300k in those places. But no, Southbridge is not as miserable as people make it out to be.