Viewing a single comment thread. View all comments

EuphoricPenguin22 t1_j9c51t7 wrote

Reply to comment by catch23 in [D] Large Language Models feasible to run on 32GB RAM / 8 GB VRAM / 24GB VRAM by head_robotics

Does that increase inference time?

1

catch23 t1_j9cd5tw wrote

it does look to be 20-100x slower for those huge models, but still bearable if you're the only user on the machine. Still better than nothing if you don't have lots of GPU memory.

14

EuphoricPenguin22 t1_j9ceqy4 wrote

Yeah, and DDR4 DIMMs are fairly inexpensive as compared to upgrading a GPU for more VRAM.

6

luaks1337 t1_j9cajyf wrote

Yes, at least if I read the documentation correctly.

1