Viewing a single comment thread. View all comments

catch23 t1_j9b9upb wrote

Could try something like this: https://github.com/Ying1123/FlexGen

This was only released a few hours ago, so there's no way for you to have discovered this previously. Basically makes use of various strategies if your machine has lots of normal cpu memory. The paper authors were able to fit a 175B parameter model on their lowly 16GB T4 gpu (with a machine with 200GB of normal memory).

56

smallfried t1_j9dtyf7 wrote

That is very interesting!

The paper is not yet on GitHub, but I'm assuming the hardware requirements are as mentioned one beefy consumer GPU (3090) and a whole bunch of DRAM (>210GB) ?

I've played with opt-175b and with a bit of twiddling it can actually generate some Python code :)

This is very exciting as it gets these models into the prosumer range hardware!

8

catch23 t1_j9dxlze wrote

Their benchmark was done on a 16GB T4 which is anything but beefy. The T4 maxes out at 80W power consumption, and was primarily marketed toward model inference. The T4 is the cheapest GPU offered by google cloud.

6

EuphoricPenguin22 t1_j9c51t7 wrote

Does that increase inference time?

1

catch23 t1_j9cd5tw wrote

it does look to be 20-100x slower for those huge models, but still bearable if you're the only user on the machine. Still better than nothing if you don't have lots of GPU memory.

14

EuphoricPenguin22 t1_j9ceqy4 wrote

Yeah, and DDR4 DIMMs are fairly inexpensive as compared to upgrading a GPU for more VRAM.

6

luaks1337 t1_j9cajyf wrote

Yes, at least if I read the documentation correctly.

1