[D] Build a ChatGPT from zero

Submitted by manuelfraile t3_12327d1 in MachineLearning

I've recently discovered models such as ChatLLaMA that allows you to create a "ChatGPT" but you need Meta's LLaMA weights (yes, you can find them in torrents but that's not the point of the question). Similar limitations found in other cases.

Therefore I wanted to try to find an open source: dataset (in addition to hugging face), "base model", "chat model" AND that it is feasible to train with a commercial computer with a very good GPU (NVIDIA, etc.). With this get at least decent results.

Also would be interesting to distinguish between solutions with commercial limitations and those who don't.

Thanks!

• EDIT • A first solution I already found is this: https://github.com/databrickslabs/dolly based on this https://huggingface.co/EleutherAI/gpt-j-6B, but looking for some discussion and perhaps other/better solutions.

1

Comments

You must log in or register to comment.

Hands0L0 t1_jdsvv9h wrote

You and everyone else here

28

W_O_H t1_jdswox3 wrote

Ye it will be easy if you have a budget of 200+ mil.

17

BeautifulLazy5257 t1_jdsyfbc wrote

I'd start by going through the Hugging Facd courseware.

You'll learn in the first chapter of their courses that it is just better for people to fine-tune pre-trained models. That's what they are there for.

It costs a lot and produces a lot of energy and heat waste to train a LLM from scratch.

8

m0ushinderu t1_jdtbee7 wrote

GPT's model architecture has not exactly been shrouded in mystery anyways. It is all about the training data and training methodology. Thats why projects that crowd sources training data such as OpenAssistant is so important rn. You can check them out!

5

LastVariation t1_jdsx10i wrote

Start by training a gpt-2, then add 2, then reinforce it to not be crazy by feeding chatGPT responses

3

he_who_floats_amogus t1_jdtwp8t wrote

>open source dataset ... feasible to train with a commercial computer ... decent results

Choose two. Therefore, you can approach this one of three ways:

  1. Use closed source data (eg. where your starting point is a pre-trained model and you're doing additional fine-tuning training)
  2. Use millions of dollars of compute resource (a "very good GPU - nvidia etc" does not meet this standard)
  3. Accept poor results
2

Deep-Station-1746 t1_jduhbth wrote

OP is peaking on Dunning-Kruger curve right now.

2

[deleted] t1_jduqdc3 wrote

[deleted]

2

Deep-Station-1746 t1_jduxvm4 wrote

You know what? provide me with a 2-3 sample "good" responses to the above post, and explain why they make for a better response than what I wrote, and I'll actually use them from now on to respond to low-effort posts from this sub.

−2

suflaj t1_jdvbyog wrote

With the constraints you have I'm afraid the best you could do is:

  • find a person who can quickly copy and paste prompts
  • give them internet access
  • pay for ChatGPT Plus
  • have them copy user prompts into ChatGPT and copy its answer to the user
1