One main factor would be the technological progress in GPU power and affordability. In the scenario you described, having a potent GPU for VR + owning a VR headset in the first place would matter (unless it's optional). The VR space has a bit of a cliche image with "gimmick games", even when there are quite a few AAA titles (e.g. Half-Life Alyx). But since most people do not own a VR headset, the big game developers out there do not focus on it.

The second factor would be, at which point lower end GPUs and even consoles would be realistically capable of running large language models. As such a simulation game would require vast amounts of VRAM and computation (for both the graphics and the AI model). Once that would be realistically feasible, this would then catch the interest of (bigger) studios. It might be even from a passionate indie developer at some point, thanks to the massive leap in research and open-sourcing (e.g. on Huggingface)

>do we still have a good 10 or so years before something that really works comes out?

Frankly it's tough to say, and take my opinion with a grain of salt. Personally I think that it will take (at minimum) another console generation. In order to see mainstream adaption and the possibility of such a true-to-detail game. There will be surely games who will attempt this at least, until one nails it perfectly.

But in theory, such a game could still happen at any point. Sites like Character.ai exist, so if such things are done on the serverside, it could happen way sooner. There would be then just more strings attached, such as always-online requirement and requiring a subscription for the GPU calculation costs.