The previous configurations of matter, energy and spacetime are “under” the rubber layer, because it’s where the rubber layer used to be in terms of entropy. You can’t take a ship and just fly to it. It would have to be a space-time machine capable of reversing entropic systems either globally or locally. Yes, the balloon analogy is simple to say, that’s the trade off. I am not going to be able to describe to you the capabilities and perception of any trans dimensional perspective, because I do not and cannot have that vantage point. The rubber layer is all you will ever be able to physically “travel” within as a being bound by length, width, height and time. So “under” the rubber layer… IF you could reverse entropy, you would simply arrive at the universe in what entropic state the universe happens to be in at the time you “stop reversing time”.

Seeing “through the rubber layer” would be akin to saying you have the power to see matter in superposition at scale. It would appear as nonsense multiple output for few input. Nonlinearity, etc… The more finely we tend to look at the universe on a small scale, the more “elephant in the room” this conversation becomes. Imagine watching a movie and seeing every single frame transposed onto one another at once… whatever that would even mean or look like. It’s important to point out here that the movie would lose all narrative meaning in this configuration. Which should prompt you to ponder what narrative in the abstract even is and if it’s somehow transcendental. It is information we might would actually lose by GAINING transdimensionality. Total conjecture but worth thinking about. Usually it’s the other way around; We gain information going up that ladder. Going down usually forces us to give up a little bit of “true” information about the higher rung.

As for saying it would “just” be the past is a little insufficient, depending on your definition of the “past”. You have to remember to account for the spatial distortion as well. So it’s not like the Langoleirs from Stephen King, where you go back to the past and all the structures and inanimate matter just is “left behind”. All of that matter is subject to entropy as well. On that note, I have a serious hunch that quantum computing and the recent work with local realness and entanglement will drastically redefine our capabilities regarding this issue. I could absolutely be completely wrong as well.

Another note on your question of whether or not there is anything happening at the “center” of the balloon still:

From OUR perspective. No, it was the Big Bang. From outside the perspective of time, who knows? Extrapolating (which is NOT necessarily the correct answer!) from an unconstrainment of the time vector, it could be a permanent ON switch eternally happening. This is nearly alllllll conjecture at this point, but in the words of Richard Feynman, it’s fun to imagine.

To use an even less understood phenomenon and an even poorer analogy:

You would be a particle within that rubber layer. It would take more energy than we can currently generate and use to rip you out of it. And it would absolutely require you to not be constrained by any time vector, which isn’t the case. What would that look and feel like? I have no idea, and I doubt anyone else has a fully proven idea of what that is like. Much like the ink cannot just float off the page of your paper, what use is that ink without paper? Will it even keep its spatial configuration so you can read it? Would your brain even be able to function without an entropic “inertia”? Etc? Now you will have to borrow more and more from philosophy to arrive at a conclusion that you and every other human might never be able to prove… but it’s important that we keep trying.