History shows that it is usually better to keep learning as simple as possible and not force the NN to learn the way you think it would be best. If you want the NN to recognize two objects, better don't waste time training it to check their spatial relationship, or something else that isn't what you're looking for (even though related).

Just check whether object 1 is reported as object 1 and object 2 is reported as object 2. This can be two separate losses you add. Don't worry about whether the actually reported objects are left or right from each other, just check whether they are in the right place each for themselves.