Not quite accurate to describe AI, what you described is just today's dominant "curve fitting" AI which doesn't generalize outside of the training distribution. This particular mathematical modeling style is as you've said, problematic.

However, it is possible to build a different type of AI which runs simulations step by step starting with sounder and minimally biased assumptions, in order to make predictions that exist outside of the existing data distribution.