I mean it sounds like you are both towing the line with other people. He isn't acting in a monogamous way and neither are you. It sounds like you should think about what monogamy means to you both and draw boundaries according to that.