Yes you need to just accept it as fact.

The view that you have is a common misconception because in our daily lives, we’re surrounded by invisible forces that make it appear as though objects stop moving when we aren’t interacting with them. The common sense explanation is that the universe works as your describing. Fortunately, through rigorous science, we can discover the true (or closer to truth) laws of the universe by controlled experimentation and reproducible measurements that hedge against our inherent biases.