Tea is only very rarely fermented (puerh and hei cha). It's allowed to oxidize to a greater or lesser extent, which is sometimes incorrectly called fermentation especially in older resources. Green tea is also not simply dried; in order to prevent the tea from oxidizing to make green tea, tea needs to be 'fixed', generally involving some kind of heat treatment; baking, pan-frying, etc.

I would also not say those differences are very small. Tea has the same kind of variation based on terroir that wine has, if not more. You might not be able to pick out specific notes (frankly I'm pretty bad at it myself) but there's a ton of variation within eg just black (red) tea or just green tea or just oolongs, such that if you were given two cups from different regions but prepared roughly the same it would be very clear that they were different. This is less pronounced with commodity grade tea (what you're likely to find in a tea bag) because it's intentionally blended to have a specific and consistent flavor profile.

All that said, the difference between black (red), oolong, yellow, white, green, and dark tea is absolutely one of processing not of variety or region. While certain varieties in certain regions are often used to make a specific kind of tea, they don't have to be, and having eg an oolong of a tea usually used to make green tea can be interesting.