I just did this this past weekend. I have family in Richmond, so we stay there overnight and make the trip into DC. We drive to the Franconia Springfield metro station, park for the day and take the Metro in.

The cherry blossoms were nice (this past weekend was peak for them), but there's still tons of people, especially near the museums. It was difficult to even walk through the sidewalks because there were so many people. I would not recommend driving in DC ever. I wouldn't even say it's worth it with the metro with a 2 year old.

Edit: forgot to mention that it's absolute shit trying to find somewhere to eat. The wait times for the wharf restaurants are 2+ hours, so you HAVE to make a reservation days in advance.