I think they would be a lot less controversial if they stuck purely to technical upgrades, rather than making major changes to the editing, tone, and environments.

This is why I think Empire Strikes Back is the least controversial special edition (and it's the only one I actually prefer over watching the Despecialized version). It didn't get a bunch of ugly CGI additions like the other two films. Most of the changes were centered around cleaning up '80s effects work, not replacing it. The entire Battle of Hoth looks so much better than it used to since they were able to digitally re-composite all the original film elements that were originally assembled photochemically.

Star Wars (1977) got done the dirtiest by the "Special Edition treatment. It changed some minor plot points, re-introduced a scene that was deleted because it was redundent (everything we need to know about Han and Jabba's relationship was already conveyed by Greedo in the cantina), and all the CGI additions feel extremely out of place in a film that otherwise looks like it was made in the '70s.

I think Blade Runner: The Final Cut is the gold standard for how these kinds of remasters should be handled. The new version itself looks and sounds great, without any added nonsense. It feels like a version Ridley Scott could have released in 1982 if given the power. But even if it was bad, they still had the good sense to include multiple original cuts of the film in the Blu-Ray set.