Well I know Reddit crowd is more on the "if it's loud enough then it's good enough". And I don't know the exact science behind it but this is my finding:

Hifiman R7DX = Sounds about the same on any source, doesn't scale much with higher gear.

Denon D2000/D5200 = easy to drive headphone, sounds pretty good on anything but sounds better on my Violectric V200. But generally most dongle dac seems to be able to power it decently well.

Mad Dog = I can definitely get it "loud enough" but sounds absolutely TERRIBLE on my beloved Dragonfly Red (my favourite dongle dac) which output 2 vrms. On O2/SDAC high gain it sounds pretty good though.

So I would say depending on headphone, easy to drive headphone will scale a little, but hard to drive headphone will definitely scale more. But it really depending on headphone.

There is no 1 answer for all, just have to try it out yourself. But I do find people who swear by "loud enough is enough" tend to read Reddit / ASR but have never physically tried higher gear. So they've listened with eyes but haven't listened with ear yet.

The way I see it is that Tesla and Civic driving at 40 km/h will both measure at 40 km/h speed. But HOW it got to that 40 km/h speed will be vastly different.

Again, it depends on headphone so it's not a yes/no type of answer.