Dendriform1491 t1_j5ywgiz wrote on January 26, 2023 at 3:08 PM

Reply to comment by manubfr in Few questions about scalability of chatGPT [D] by besabestin

Also, Google doesn't use GPUs, they designed their own cards which they call TPUs.

TPUs are ASICs designed specifically for machine learning, they don't have any graphics related components, they are cheaper to make, use less energy and can make as many as they want.

cdsmith t1_j5z0rrm wrote on January 26, 2023 at 3:37 PM

You don't have to be Google to use special-purpose hardware for machine learning, either. I work for a company (Groq) that makes a machine learning acceleration chip available to anyone. Groq has competitors, like SambaNova and Cerebras, with different architectures.

Taenk t1_j60gdbl wrote on January 26, 2023 at 8:59 PM

Do these also increase inference speed? How much work is it to switch from CUDA based software to one of these?

cdsmith t1_j60q0bs wrote on January 26, 2023 at 9:59 PM

I can only answer about Groq. I'm not trying to sell you Groq hardware, honestly... I just honestly don't know the answers for other accelerator chips.

Groq very likely increases inference speed and power efficiency over GPUs; that's actually its main purpose. How much depends on the model, though. I'm not in marketing so I probably don't have the best resources here, but there are some general performance numbers (unfortunately no comparisons) in this article, and this one talks about a very specific case where a Groq chip gets you a 1000x inference performance advantage over the A100.

To run a model on a Groq chip, you would typically start before CUDA enters the picture at all, and convert from PyTorch, Tensorflow, or a model in several other common formats into a Groq program using https://github.com/groq/groqflow. If you have custom-written CUDA code, then it's likely you've got some programming work ahead of you to run on something besides a GPU.

lucidrage t1_j61so7l wrote on January 27, 2023 at 2:33 AM

>convert from PyTorch, Tensorflow, or a model in several other common formats into a Groq program

Are there any effort spend in adding a plugin for a high level framework like keras to automatically use groq?

cdsmith t1_j62a3yv wrote on January 27, 2023 at 4:55 AM

I'm not aware of any effort to build it into Keras, but Keras models are one of the things you can easily convert to Groq chips using groqflow.

gradientpenalty t1_j61tko2 wrote on January 27, 2023 at 2:40 AM

Okay, so where can I buy it as a small startup for under 10k without signing any NDA for using your proprietary compiler. As far as I can see, we are all still stuck with Nvidia after 10B of funding for all these "AI" hardware startup.

cdsmith t1_j626c0c wrote on January 27, 2023 at 4:21 AM

I honestly don't know the price or terms of use, for this or any other company. I'm not in sales or marketing at all. I said you don't need to be Google; obviously you have to have some amount of money, whether you're buying a GPU or some other piece of hardware.