Select_Beautiful8

Select_Beautiful8 t1_jc9lckr wrote on March 15, 2023 at 7:31 AM

Reply to comment by KerfuffleV2 in [P] ChatRWKV v2 (can run RWKV 14B with 3G VRAM), RWKV pip package, and finetuning to ctx16K by bo_peng

just got time to try it, but it doesn't load nor does it give error message :( Thanks anyways for your help!

Select_Beautiful8 t1_jc0w1px wrote on March 13, 2023 at 5:45 AM

Reply to comment by KerfuffleV2 in [P] ChatRWKV v2 (can run RWKV 14B with 3G VRAM), RWKV pip package, and finetuning to ctx16K by bo_peng

This gave me the "out if memory" error again, which did not happen with the "cuda fp18i8 *16 -> cpu fp32" :(

Select_Beautiful8 t1_jbrbor0 wrote on March 11, 2023 at 2:38 AM

Reply to comment by KerfuffleV2 in [P] ChatRWKV v2 (can run RWKV 14B with 3G VRAM), RWKV pip package, and finetuning to ctx16K by bo_peng

Thanks, I use Windows, but I want to do a dual boot

Select_Beautiful8 t1_jbra2af wrote on March 11, 2023 at 2:25 AM

Reply to comment by KerfuffleV2 in [P] ChatRWKV v2 (can run RWKV 14B with 3G VRAM), RWKV pip package, and finetuning to ctx16K by bo_peng

ok so "cuda fp16i8 *16 -> cpu fp32" would be the most optimal argument for me?

Select_Beautiful8 t1_jbr867y wrote on March 11, 2023 at 2:09 AM

Reply to comment by KerfuffleV2 in [P] ChatRWKV v2 (can run RWKV 14B with 3G VRAM), RWKV pip package, and finetuning to ctx16K by bo_peng

Oh it loaded, it was because I wrote "cuda fp32" instead of "cpu fp32" in the second half of the argument. Thanks

Select_Beautiful8 t1_jbqyth8 wrote on March 11, 2023 at 12:54 AM

Reply to comment by KerfuffleV2 in [P] ChatRWKV v2 (can run RWKV 14B with 3G VRAM), RWKV pip package, and finetuning to ctx16K by bo_peng

Thanks. I'm actually using the oobabooga text generation webui on github

Select_Beautiful8 t1_jbqpd5x wrote on March 10, 2023 at 11:41 PM

Reply to comment by KerfuffleV2 in [P] ChatRWKV v2 (can run RWKV 14B with 3G VRAM), RWKV pip package, and finetuning to ctx16K by bo_peng

>How do I reduce the CUDA layers?

Select_Beautiful8 t1_jbq9m13 wrote on March 10, 2023 at 9:50 PM

Reply to comment by KerfuffleV2 in [P] ChatRWKV v2 (can run RWKV 14B with 3G VRAM), RWKV pip package, and finetuning to ctx16K by bo_peng

No, I wasn't able to load the 7B model, it still says CUDA out of memory :(

Select_Beautiful8 t1_jbp7qq7 wrote on March 10, 2023 at 5:43 PM

Reply to comment by KerfuffleV2 in [P] ChatRWKV v2 (can run RWKV 14B with 3G VRAM), RWKV pip package, and finetuning to ctx16K by bo_peng

Thanks. I have a laptop 3060 and 16GB of RAM, and I successfully ran the 3B one; I will try with the 7B one.

Select_Beautiful8 t1_jbk1hwd wrote on March 9, 2023 at 4:37 PM

Reply to comment by bo_peng in [P] ChatRWKV v2 (can run RWKV 14B with 3G VRAM), RWKV pip package, and finetuning to ctx16K by bo_peng

Thanks, I will try it

Select_Beautiful8 t1_jbijcjl wrote on March 9, 2023 at 8:01 AM

Reply to comment by bo_peng in [P] ChatRWKV v2 (can run RWKV 14B with 3G VRAM), RWKV pip package, and finetuning to ctx16K by bo_peng

I tried the 3B and it said out of memory. I'm now trying 1B5 and it loads correctly.

Select_Beautiful8 t1_jbifwzt wrote on March 9, 2023 at 7:15 AM

Reply to [P] ChatRWKV v2 (can run RWKV 14B with 3G VRAM), RWKV pip package, and finetuning to ctx16K by bo_peng

I have one 6GB vram GPU, which model should I use?