You should consider diving into the topic a little deeper. What you’re talking about is distributing the computation, which is something that is already being done at some scale or another when there is more than one gpu or multiple machines. An outside example of this that you can donate your computers compute to SETI.

Your question about wether it can beat an existing implementation of gpt is the most complicated question ever posed in the history of humanity. It sounds like you’re assuming that this will have more compute than a dedicated system, but there’s a little more to getting something that performs better than just compute. Compute is a bottle neck, but only one of many.