The reason it's repeatedly stated it's all about frequency at the ear drum is because blind listening tests showed that to be the major if not only factor in what determines what people perceive as different about a given sound setup.

Everything else audiophiles talk about is currently completely subjective and "some" hear these differences and some don't.

The reason people don't think it's just frequency is because they can't eq one headphone to sound like another and assume that is proof it's not just that. Ignoring the fact that they most certainly did not achieve the same frequency at the ear drum.

There are some actual differences, such as the feeling of bass with your body from a subwoofer.

it's not necessarily something you hear, but feeling air being pushed at you is a real feeling and some headphones with bigger drivers and thicker pads can give you the feeling of more bass presence and that physical feeling cannot be eq'd.