It's probably because of the small amount. Doctor once recommended me to add a few drops of tea tree oil into water for intimate hygiene, and it worked perfectly. It has antibacterial effect, so it should be beneficial for you :)

But I guess the rule of thumb would be this: when it burns, it's probably not good for you.

Tea tree oil as a stand alone product is concentrated, as are most essential oils. That's a big part of why they can harm you. (Also some of them can be toxic if absorbed through the skin or mucus membranes, but that's not relevant to tea tree oil)

Most of the time if you see an oil like that in a product, it's being diluted by carrier oils that are safe for you (coconut oil is a really popular one) to dilute them and make them safe to use directly on your skin. Your soap likely is using a carrier oil to protect you (and a lot of carrier oils are really nice and moisturising for your skin)

Water isn't super effective at diluting in the sense of mixing evenly since it's still an oil, but it does reduce the ppm of the oil, rendering it less damaging to the skin.

Also for anyone using it for skincare; remember to moisturise after, that shit is astringent as hell and your skin needs a barrier to protect it.