First thing first: if you want quiet, you have to avoid downtown.
Here you have a choice: living 5 min walk from the hospital, or quiet. Downtown is frequented by a colorful bunch that likes to advertise their coolness with loud noises, plus loud partygoers.
Bikers in State street (especially at the corner with Clark) will make sure be be heard during summer, so that's a no too. My suggestion is to look for narrow streets with potholes if you want quiet.

That being said with you budget you can go anywhere. There are some very new buildings near congress and cedar st that are in an very quiet (almost dead) area and 2 min walk from the hospital, but there's nothing to "explore there".

General advice:
Get a nice warm and waterproof coat.
Don't get attached to your umbrella, you'll lose it, or it will get stolen.
Buy a bike AND a good sturdy lock, if if have a safe place to park it overnight.
Craigslist in New Haven is a gem, use it.