The Druid Hill Y is probably your best bet for something year-round with broad choices of physical activity.

Beyond that, I would investigate the city rec centers and see if any of them offer exercise programs. I get the feeling that they do, but the city website is unhelpful in figuring out which site offers what.

As far as swimming goes, I know there's an outdoor pool in Druid Hill Park, and I believe there's another in Sandtown-Winchester, though they're both closed in the winter. Friends in southwest Baltimore say you can take swim lessons at the Vivien Thomas Medical Arts Academy in Franklin Square, but I don't know if their pool is open to the public beyond that.