the ones talking about limits are the one's who are creating the processors, they are not pessimists, just realists. silicon has quite a bit of runway left, however other materials will be able to help in the future.

Frankly there may come a time where we use hybridization of classical, quantum, and biological computation since the all have their own strengths and weakness. At least classical and quantum hybridization is considered an obvious future.