The model is huge though and needs to be in GPU memory for performance calculations (sparse matrix dot product).

Probably one thing teams are working in is reducing the dimensions of the sparse matrix so it can fit on fewer GPUs. Also looking at reduced precision of floating point multiplication, 8 bit floats is probably enough for AI matrix maths. Maybe combining matrix multiplication AND the activation functions (typically ReLU or Sigmoid) so two maths operations can be done in one pass through GPU. That involves refactoring their math library.

Or the build custom TPUs with all this build into the hardware.

The future is bright 🌞 for AI. Until we hit the next brick wall