The problem with space being cold is that it’s also empty. You know how a windy day at 40 degrees out feels much colder than when it’s 20 degrees out but with no wind? Take that to its logical extreme. Things do cool down in space, but not by convection or conduction so it’s very slow.