In a good example of correlation not meaning causation.

What's going on is this - some small population of people always have the flu at any given time, they're just the unlucky bunch. Now the flu itself needs to survive the journey from person to person, this might mean being suspended in the air in droplets, or lingering on surfaces. The flu does best in moderate conditions, somewhat humid to dry air, and middling to cooler temperatures.

Now think about a nice, hot sunny day. Everyone is generally outside where the air is blowing away the germs AND it's not an environment the flu can survive in easily anyway. So, probably not a lot of chance of a flu epidemic in nice, warm sunny weather.

Now lets go your cold and miserable day. You're focusing on the outdoor exposure, but what were you doing before and after? You were indoors, with all these other people avoiding the weather, in conditions the flu LOVES, no breeze, moderate temperatures, everyone touching everything. That's how the flu spreads.

Yes, cold weather can stress your body and weaken your immune system, but as you mentioned, being cold and wet does not give you a virus.

Another important point is cold air is generally very dry. When you're stuck indoors all winter, breathing that dry air it causes damage to the lining of your nose and throat, kind of like getting chapped lips inside your body. That damage means the flu can travel through the skin in those areas making transmission all the more easy.