Viewing a single comment thread. View all comments

Disastrous_Elk_6375 t1_j99xxfa wrote

Reply to comment by ArmagedonAshhole in [D] Large Language Models feasible to run on 32GB RAM / 8 GB VRAM / 24GB VRAM by head_robotics

Are there some rough numbers on prompt size vs. ram usage after the model load? I haven't played yet with GPT-NeoX

10

ArmagedonAshhole t1_j9a1vq3 wrote

it depends mostly on settings so no.

Small context like 200-300 tokens could work with 24GB but then your AI will not remember and connect dots well which would make model worse than 13B

People are working right now on spliting work between gpu(vram) and cpu(ram) in 8bit mode. I think like 10% to RAM would make model work well on 24GB vram card. IT would be a bit slower but still usable.

If you want you can always load whole model to ram and run it via cpu but it is very slow.

12