You don't need conscious awareness to beat me in chess, and maybe you don't need it to beat all of us and take everything.

So what worries me is not that we're copyable, but that maybe we're not. We can't prove whether another person or an animal actually experiences qualia. What if the machine doesn't, but wipes out all life anyway? Then the light of consciousness would go out of the world.