nemorocksharder t1_jdz8kt5 wrote on March 28, 2023 at 7:40 AM

Reply to comment by light24bulbs in [R] Hello Dolly: Democratizing the magic of ChatGPT with open models by austintackaberry

What you're describing is exactly what I have been looking to do too, and am really surprised I'm not hearing more about it. Have you found any useful approaches to essentially adding to the LLM's Corpus with target material/text? or anyone else trying to do this?

light24bulbs t1_jdzzeh4 wrote on March 28, 2023 at 1:01 PM

Yes, I'm into it now. Code like this can be adapted to load bulk data instead of q&a.

I suspect some of the training parameters need to be adjusted a bit to prevent over fitting and obviously the data loading and templating needs to be removed.

https://github.com/lxe/llama-tune Or for a cooler approach where you make a Lora layer https://github.com/serp-ai/LLaMA-8bit-LoRA