It’s not the swearing in. There are several US states that actually prohibit atheists from holding office. I live in Texas. Texas is one of them.

Now granted, it wouldn’t hold up today - hopefully. But there’s definitely still a law for it.