They rarely ask in US. Only once they asked one of my friends at the departure. I travel overseas frequently too. In most cases we have notarized letters. Once I left a letter overseas coming back, and that is the only time they asked me if I have permission to bring the kids back to US. I told them I had a letter but left it back overseas and they were cool with it.