Is it a common belief that all life is just different manifestations of the same consciousness? Not in a vague spiritual way, like a real, physically consistent way. I really can't think of a better explanation. Take consciousness as a fundamental, singular property of the universe (or reality in general) that can manifest in different ways (with some complicated interaction with brain structure/information that results in different qualia).

We obviously can only remember our own lives because physically it's only possible for our brains to generate the qualia for remembering our own past. But different people (and life, and whatever else is conscious), would still be part of the same consciousness, just physically disconnected. In the same way you might not remember being a kid a long time ago, you don't "remember" being other people (because it's physically impossible for you to have those qualia), but it's still all "you," because consciousness is just one thing, not something that gets "duplicated" for each person.

You could "model" it as reincarnation, where you experience another life after death but don't remember you're old one, but really there wouldn't be time dependence (you're this life, then you're this life), they would just be two, causally disconnected lives that don't happen in any definable order (you're this life, and you're this life, no explainable order because there's no causal connection).