Admittedly a large population of the German refugees were Jewish, and iirc, prior to the Allies liberating Auschwitz, most of America didn't know about the death camps. Hitler probably (correctly) assumed that if he openly said he was committing genocide, it would have gone much worse for him. We didn't really know what he was doing until it was far too late

>I don't recall a time where the US let in citizens from the aggressor nation.

I looked it up bc I was bored and what else are you gonna do on a Tuesday night, and we actually did take in some amount of German refugees during WW2 with the caveat that they must have already escaped Nazi territory, so not directly from Germany, but probably the closest you'll find