>I feel like we haven't had a fair chance to let our relationship truly flourish

You've been together for 5 years. I know it's hard to accept, but it's over.

If after 5 years he's developed an annoyance for you and no longer wants to live with you, you two are done. Plus, you deserve to be with someone who is happy to come home to you and who doesn't find your presence to be annoying.

>I feel like hes my soulmate and this whole things is breaking my heart and soul

Your soulmate would want to be with you just as much as you want to be with them. It's going to be painful to move on but you'll get through it.