Here's an analogy that's going to fall down if you think about it too much, but it's still valid.

Consider the universe as the surface of a spherical balloon. Don't consider the void in the centre, just the surface. As the balloon expands all parts of the surface move away from you in all directions. The universe expands. If you were to measure the circumference of the balloon and you knew the starting size and the rate of expansion then you'd know the age.

You could do this from any point on the balloon.