ChatGPT's name helps us understand it a bit more. GPT stands for Generative Pre-trained Transformer.
Pre-trained means that it learned about the structure of language and everything else it knows - that is, those 175 billion parameters were adjusted and fine-tuned - by the system working its way through vast amounts of texts from the internet and other sources. In effect, it built an internal model of human language and much of what language has been used to express.
Transformer means that the architecture of the system - what those 96 different layers do and how they interconnect - was designed to allow it to pay attention to a number of key elements of sequential input at the same time, and analyze how they go together. Here's ChatGPT's summary:
Overall, the Transformer architecture is characterized by its ability to perform efficient attention-based computations over sequences of variable length, and its ability to model long-range dependencies in sequential data. This makes it particularly well-suited for natural language processing tasks, such as language translation and text generation.
Here's an example, using a Jeopardy question:
Input: Look for Aretha, Ray, James Brown, Cab Calloway, and, of course, Twiggy and Frank Oz in this 1980 film comedy.
ChatGPT: The 1980 film comedy that features Aretha Franklin, Ray Charles, James Brown, Cab Calloway, Twiggy, and Frank Oz is "The Blues Brothers". Directed by John Landis, the film stars John Belushi and Dan Aykroyd as "Joliet" Jake and Elwood Blues, who set out on a mission from God to reunite their band and save the orphanage where they were raised. Along the way, they encounter a number of musical legends, including the ones you mentioned.
You can see how effectively ChatGPT extracted the key part of the input - that we're seeking the name of a film from a particular year with particular actors. It then gives us a sample of how much it knows in its accurate, detailed and focused response.
Generative means that the system is capable of creating text that makes sense. Here text might mean a short answer, a summary, an essay, a computer program, a poem, a short story - you name it.
You may have heard that ChatGPT generates its responses by "just" figuring out what word "should" follow the previous word. More accurately, ChatGPT takes into account a whole sequence of words, much as we might read an entire sentence or paragraph before responding. It then chooses what word or words come next on the basis of all the information and relationships stored in those 175 billion internal parameters. In other words, there's an enormous amount of knowledge and information processing hidden in those words "just" and "should."
"This is possible," ChatGPT says, "because the model has learned the statistical patterns and structures of language during pre-training, which it can use to generate new text."
What does it know?
I continue to be amazed by how much factual information ChatGPT knows - from pop culture to quantum physics, from cooking to coding, not to mention dozens of different human and computer languages. Again, it doesn't have current access to the internet or other external sources of information, but it does manage to store an incredible amount of detailed information in those 175 billion parameters.
It knows a lot, and presents what it knows in a well-organized, authoritative way. That's why it concerns educators, since it can turn out high-school or even college-level essays on just about any subject in seconds. It's already passed the US Medical Licensure Examination, earned an A+ on a microbiology quiz, and passed tests for Wharton MBA students, and final exams at the University of Minnesota Law School and Stanford University Medical School. There's good reason for teachers, journalists and other writers, programmers, and most other people dealing with information to be worried.
What doesn't it know (and why doesn't it know it)?
There's a lot it doesn't know, for example the sort of fine details or complex relationships that a real expert on a subject would know. That's not so bad - we can't expect it to know everything. But it does have a major flaw -- it doesn't know when it doesn't know. Not infrequently, instead of simply saying "I don't know," ChatGPT just makes stuff up, while still presenting it authoritatively. AI experts call this hallucinating. Here's an example:
Next Page 1 | 2 | 3 | 4 | 5 | 6 | 7
(Note: You can view every article as one long page if you sign up as an Advocate Member, or higher).
2
2
1
