How can you be outside something that is infinitely large? It just goes on forever.

“Space [...] is big. Really big. You just won't believe how vastly hugely mind-bogglingly big it is. I mean, you may think it's a long way down the road to the chemist, but that's just peanuts to space.” - Douglas Adams.

Are you maybe talking about the observable universe? There's just more galaxies outside of that, "observable universe" is a localised term, and it will just be different spheres visibility centred on wherever you are at the time that may, or my not, overlap with the Earth-local observable sphere.