I am experiencing RAM issues when other Adobe Apps are open although I set their allowed RAM usage to the minimum amount. This is different to x86 where this setting really had an effect.

So even a passive Media Encoder or Illustrator are slowing down AE on M1, but when AE is running alone, its very smooth here.