I think it's because it seems to be most often done very poorly. Whether it just doesn't make sense for the characters, the writing is bad, or it feels shoe-horned into the book like it was a requirement from the publisher and they're checking a box.

Some people are surely prudes as well, but at least in my experience it's just that it usually takes away from the overall story instead of adding to it.

It's just done poorly very often and shoehorned into situations it doesn't belong.

There may be some prudish people but I'm not offended if I read an out of place sex scene, I'm just a little bored.

We live in a world with the internet and anything sexy directly at your fingertips. Incorporating a sex scene just because has lost it's relevance and steaminess.