Viewing a single comment thread. View all comments

sayoonarachu t1_j1408am wrote

If you're savy enough, you can technically run BLOOM 176b . But as others stated, it'll take forever to be usable. I.e 30 minutes for 10 token.

https://towardsdatascience.com/run-bloom-the-largest-open-access-ai-model-on-your-desktop-computer-f48e1e2a9a32

11

avialex t1_j14p22o wrote

There's a VRAM memory leak in that code btw. I haven't tracked it down yet, but it's easy to solve with a torch cache clear in the forward method.

5