It would not matter as long be dealt with this personally myself and you tell them in any and all ways and they still “won’t know why”. It’s because they do not want to know and just want you (OP in this case) to do what they want to do and to hell with what you want or need.

So OP should just stop playing their game and drop the rope and move on.

That’s laughable. He knows but doesn’t care what he wants but only what he feels is owed to him. Don’t be naive to dysfunctional parental dynamics and their amazing ability to be willfully in denial.

Cease and desist letter from your lawyer (get a good lawyer) and if he persists follow up with a restraining order (or your local equivalent) to legally keep him away and enforce it vigorously as needed.

Divorce him before he has his friend rape you in front of him. He’s a scum bag and in no way was this just a fantasy but a sick need for him to realize (nothing against those who do hot wife but this isn’t that obviously). There’s no saving this as he severely broke your trust and won’t accept your no.

Take you and your child out of this situation ASAP. Talk to your friends and family for support and take this asshole to the cleaners. He can have his fantasy with someone else who's willing to play along.