Sometimes it just has to guess blindly, it's almost impossible to know the true colors. There are many it gets right based on the texture it's made of, but since clothes can be any color it doesn't have a proper way to know. If you know the sweatshirt has to be red you can specify it in the prompt and it should work.