I think it's because it seems to be most often done very poorly. Whether it just doesn't make sense for the characters, the writing is bad, or it feels shoe-horned into the book like it was a requirement from the publisher and they're checking a box.

Some people are surely prudes as well, but at least in my experience it's just that it usually takes away from the overall story instead of adding to it.