Don't pay off the student loans in advance. I had a similar amount and I paid it off over 20+ years at somewhere between $100-$150 dollars a month. It really is not going to impact you that much, especially decades later when you're earning way more.

I imagine it this way. Imagine if someone assembled 20k from your earnings over 20 years. That's quite an assembly of capital and a decent interest rate. You might as well do something with it you can't otherwise do because you're never going to get a 20k loan with 4% interest ever again (credit cards? LOL) except maybe a new auto loan or a mortgage.