That's a solvable problem. Same discussion as with autopilots in cars.

With the human staff in hospitals getting thinner by the day, some people would rather trust an inexpensive machine, than having to wait ages to talk to a human doctor, who might not even be smarter than the machine. Assuming that AI will grow in popularity in general.

Couldn't you train on the output of Codex itself? Might be legally dubious, but so is a lot of training of these AIs in the first place.