Another point is that even if the GPU would do very little computing, you would need something able to communicate between your screen and your computer. This is similar to external sound cards and network cards, although these are not so common these days because for most users the integrated cards are fine.