KerfuffleV2 t1_jd7sb4u wrote on March 22, 2023 at 1:33 PM

Reply to comment by keeplosingmypws in [Project] Alpaca-30B: Facebook's 30b parameter LLaMa fine-tuned on the Alpaca dataset by imgonnarelph

llama.cpp and alpaca.cpp (and also related projects like llama-rs) only use the CPU. So not only are you not getting the most out of your GPU, it's not getting used at all.

I have an old GPU with only 6GB so running larger models on GPU isn't practical for me. I haven't really looked at that aspect of it much. You could start here: https://rentry.org/llama-tard-v2

Keep in mind you will need to be pretty decent with technical stuff to be able to get it working based on those instructions even though they are detailed.

keeplosingmypws t1_jd9wpwm wrote on March 22, 2023 at 9:44 PM

Thanks for leading me in the right direction! I’ll letcha know if I get it working