There is nothing about diffusion models that stop it from memorising data. Dall-E 2 can definitely memorise.

Codex is not technically copy pasting; it is generating a new output that is (almost) exactly the same, or indistinguishable on the eyes of a human, to the input. Sounds like semantics, but there is no actual copying. You already have music generating algorithms that can also generate short samples that are indistinguishable to the inputs (memorisation). Dall-E 2 is not there yet, but we are close to prompting "Original Mona Lisa painting" and be given back the original Mona Lisa painting with striking similarities. There are already several generative models of images that can mostly memorise inputs used to train it (quick example found using google: https://github.com/alan-turing-institute/memorization).

Yes, I find it incredibly strange that when speaking about Codex, everyone is worried about the models regurgitating the code they have been trained on while citing GPL and other licenses; but this seems to not be that much of an issue when it comes to images (given anecdotal evidence from these discussions), even though they themselves have licenses. It just goes to show that humans perceive text and images very differently from a creative point of view.