Broccoli has much less sugar and more fiber than most sweet fruits, but fruit is not just a bunch of sugar. Fruits have fiber, vitamins, minerals, phytochemicals, antioxidants etc. Plus, back when the old "four food groups" was new, people got much more exercise. Sugar isn't bad (especially when it comes with fiber, which reduces blood sugar swings) if you are burning it off. Other carbs, fats and proteins get converted (in part) to sugar by your body.

With the amount of exercise many of us get today, sweet fruit should practically be considered a dessert--and grains should be an exception rather than the rule.

(I prefer to say "sweet fruit" to differentiate from fruits like tomatoes, squash, avocados, etc.)