The show has established from the start that it's going to tell smaller stories within its world as well as Joel/Ellie's "main story".

Yet it still manages to give us backstory about Joel, about surviving the past 20 years, about surviving in a bubble without any health services, it gives us a lot of context to appreciate/fear the world and makes us care about people we just met. It also revealed the source of the radio music from the first episode, making us think back to that. There was a lot that ties back to our main characters, albeit indirectly. Worldbuilding is not filler in my eyes.