Deep learning neural networks. Imagine that you trained a computer to work like a human brain - taking in information and learning based on that information. Now imagine that brain can be trained on billions of pieces of information and make conclusions and responses based on those billions of pieces of data. That’s essentially what it’s spitting out.

Way over simplified but that’s more or less how it does what it does.

ChatGPT is a modified version of another AI language model, called GPT, made by OpenAI.

While the exact technical details aren't available, we do know it's a complex language model trained on a vast data set of written word. There is also some component to the training that teaches ChatGPT what appropriate responses to prompts look like, so that it can work in a chatroom context.

In more ELI5 terms, it's a program that was designed and trained to learn how language works and how to respond to prompts.

When you send it a message, it does a large, complex calculation to make a good response to your message.

Chat GPT is a form of deep learning model, which is a subsection of a machine learning model. A machine learning (ML) model is one in which the decisions the model makes are based off a ‘training’ step rather than being physically encoded.

A simple example is a model that tried to distinguish between different breeds of flower. So, you give this model some information about each flower (petal length, colour, etc) as well as a ‘truth label’ (what a flower expert has said that flower is).

The model takes these numbers as inputs, these inputs are multiples by a set of numbers, have some numbers added to them, and then get passed to the output, and some value is decided as a cut off (eg, if output >5 it’s flower A, otherwise it’s flower B) If the model is wrong, all those numbers get changed a little bit, in a process known as stochastic gradient descent.

In a deep learning model, the inputs are multiplied, and then passed to a ‘hidden layer’ of nodes (often called neurons). Then these numbers are again multiplied by another set of numbers. This keeps going for multiple layers until you get to the output layer.

This is an over simplification, but is the basis of how things like chatGPT work. They simply look for patterns, and output the next word based on what they think matches the pattern.

What makes chat gpt pretty powerful is (mostly) it’s size. It contains 175 billion of those numbers that have to get updated while training, and so takes a long time + is very expensive to train

Currently, artificial intelligence (AI) and machine learning (ML), which ChatGPT make use of, are simply the science of statistics being applied heavily.

If you take a sample of 10,000 English sentences, you expect to encounter certain patterns. Maybe 3 of the sentences have "rock" after the word "the," maybe 15 of the sentences that contain 6 or fewer words contain "I." Depending on how frequently these patterns appear, you can make predictions; if 9,996/10,000 of those sentences have "rock" after the word "the" and you're given the word "the," you can predict that you should follow it with "rock."

Now take this principle and scale it up greatly with the most sophisticated pattern-finding levers the company could come up with for the program. Feed it examples of countless oceans of language in different contexts associated with different prompts. It's then a matter of calculating based on each model and coming up with the most probable word that should follow the previous word given the entire context (your question, the sentence, the paragraph, the conversation). At that point, you can reasonably expect the program to "act like" whatever the training data was. And the training data was well-labeled and captured across many contexts, allowing the program to feel intelligent.

ChatGPT is a language model that uses deep learning to generate human-like text. It is trained on a large dataset of text and uses a variant of the transformer architecture called the GPT (Generative Pre-trained Transformer) architecture. During training, the model learns patterns and relationships in the text data, allowing it to generate new text that is similar to the training data. When the model is used for generating text, it takes a prompt (a starting text) and generates a continuation of rhat text. The quality of the generated text depends on the quality of the training data and the complexity of the task.

In more eli5 terms:

ChatGPT is a computer program that can talk like a person. We teach it by showing it lots and lots of talking, like in books and on the internet. It learns how people talk and then it can talk like a person too. When you ask it something, it uses what it learned to try to say something that makes sense. It's like when you learn new words, you can use them to talk better.

[deleted]

[removed]

It is a direct decendent of the same techniques used to help you type by predicting the next word on smartphones and chat programs.

People found they could improve those models using trained artificial neural networks to learn prediction patterns. These evolved into very large models that have to train for very long times on humongous amounts of data on a very large number of computers.

Papers on these have been criticized for showing little architectural innovation or cognitive insight., just scaling things up massively.

But the results certainly produce many hitherto obtainable aspects of human intelligent discourse, even it is impossible yet to tell if any cognitive model building is going on in that artificial neural network.

Was this written by ChatGPT?

I can't refrain from not denying that it isn't not, because it maybe would or maybe would not directly break the subreddit rules. But I would say that I could not resist the irony. And if it saves me from being banned, I must say that I did type all the comment, and that 1/3 of the paragraphs are mine.

It's important to note that ChatGPT doesn't "understand" things in the same way we do, and doesn't "think". So it does tends to get a lot of stuff wrong.

That being said, it's quite a good bit better than previous attempts at similar things.

Please read this entire message

Your submission has been removed for the following reason(s):

Whole topic overviews are not allowed on ELI5. This subreddit is meant for explanations of specific concepts, not general introductions to broad topics.

If you would like this removal reviewed, please read the detailed rules first. If you believe this submission was removed erroneously, please use this form and we will review your submission.

You forgot to add in the censorship , and ideological parameters.

Does it just have a database of things it has to conform to / things it’s supposed to ignore?

ie most historical texts and all biology textbooks up to year x would tell you that men cannot get pregnant or menstruate. But in 2023, wouldn’t chatgpt tell you a man can menstruate and get pregnant?

And how does it deal with input data known to be false / lies?

Would it tell you George santos is a Jew”ish” black Hispanic male who’s family avoided the holocaust via being born as white Christians in South America?

[deleted]

You’re gonna be downvoted to oblivion for raising this issue but you are 100% correct in the biases of machine learning. We will only ever hear about the biases the mainstream is interested in though (e.g. face scanners not triggering on people with dark skin), and they will intentionally ignore anything that furthers their perceived reality.

The one that says “in more eli5 terms:”, amirite?