>s balloons would be reflecting solar IR

Solar IR is a small part of the downwelling solar spectrum.

https://cdn.shopify.com/s/files/1/0823/0287/files/atmospheric-absorption_848x452.png?v=1670339790

And the near infrared is heavily absorbed and emitted by CO2 and H20 so in those wavelengths that would be the ambient IR. The whole sky will look like it is those wavelengths.

You would also have to account for the upwelling radiation (very very very roughly shown here in Khiel and Trenberth 97 https://chriscolose.wordpress.com/2008/12/10/an-update-to-kiehl-and-trenberth-1997/ )

These wavelengths have always been a huge problem for IR.

Operating in these wavelengths is "the endorsement of the air force you think it is!"