We're still going to be limited by fab capacity, rare earth minerals, energy, and maintenance technicians.

Supply chain still rules above all. Trade needs to exist until/unless post scarcity hits.

Energy is still finite, and AI uses an absolute fuck ton compared to the human brain. I don't see a practical way to scale it up with current technology that wouldn't also allow for genetic engineering to make us compete just as well, but more resiliently.

Also, We literally just had a 10-100x carrington event miss us in the last two weeks. That shit would set us back to the industrial era at best, above-human-AI or not.

If it turns out AGI can figure out a way to get infinite energy without destroying everything, hey, problem solved! No more conflict! Dark forest avoided!

When you're in the elbow it's really hard to tell if the growth is logistic, exponential, or hyperbolic.

Yeah, pragmatically I don't see any issues with arithmetic or using any math already proved. Imo it's still to be seen if LLMs can do novel thought, but even if not that's still ... what's a word signifying a greater change than revolutionary? Game changing?

I did see some AI coming up with independent models of physics that have no analog yet were able to properly model real physical systems and make valid predictions with a formula whose variables could not all be determined by the researchers, but idk if that was an LLM

Chat GPT is not running the multiplication algorithm. You're being the human in the loop here by having it iterate through every step in the algorithm. You're manually executing a bunch of constant time operations and feeding the input back into itself.

You're basically writing and running code. If this qualified as being able to derive a multiplication algorithm then all CPUs are already sentient.

Your example is orders of magnitude less likely than the carrington event or the recent one which was 100x more powerful but luckily pointed away from earth

Yeah, you bring up some great points. Honestly I would want to go the opposite direction though and compare growth rates of functions, many classes of which tend to infinity but at vastly different rates.

I totally tried to get a better example but then went on a wiki binge and got lost around the page of hyperbolic growth which contrasts itself against exponential and logistic growth then found my way to robert miles again,....

That said there are still different levels of infinity. The reals have a Lebesgue measure strictly greater than the rationals.

Also the "sum of 1..inf == -1/12" is not the case at all, the whole point of that example is to show how different contexts and definitions can have conflicting answers, similar to 0^0 or 1^(inf)

100%, was a pure math major who sucked at algebra and arithmetic. They're more brutish skills than are often needed to do math, and definitely than needed to understand it.