Building AI Applications - Part 1: What Is a Language Model? (Vectors & Embeddings)

This is the first unit of a course on using and building AI systems and applications, made for everyone, with no math and as little jargon as I can manage. The single thing I want to drive home, before anything else, is this:

A language model is not a person, and it is not a machine that is running. It is a file.

Once that lands, the rest of the technology stops being scary, and you can cut through a lot of the nonsense you hear about it. Here is the whole unit, end to end.

The model is a file, not a mind

The model is a very large file of numbers sitting on a disk. It gets loaded into computer memory and is used by a program. It doesn’t change during the process, it doesn’t change at all, which means:

• It does not learn.
• It has no memory.
• It is not alive.

If you want to think of it like a person’s brain, it is only part of the brain, and if that were all the person had, we would describe them as brain dead because of all the bits that are missing.

So what actually runs? The system around the model. That is software, the model runner, that loads the file, takes what you type, applies the file’s numbers to it, and prints what comes out.

The system is the kitchen, and you bring the ingredients.

Three steps: listen, score, speak

Here is what happens when you type a sentence. Three steps, walked through with one sentence:

Listen. Score. Speak.

The sentence is I went to the store. The question the system always asks is: what comes next?

Step one: listen

The system takes the five words and cooks them up into a mess of numbers with the model. Step after step, each step rewrites the numbers for each word by blending in the numbers of the other words. The model file holds the weighting, which words matter to which, and how much.

Some say each word listens to the others. That is what self-attention is all about. Not what the words are, or what order they are in, or even what the words mean, but what they mean to each other as an entire phrase. So:

• went comes away marked as something that happened in the past, with a doer.
• I comes away marked as the doer, the speaker. This is their story.
• store comes away marked as a place, a destination, the kind of place you go to with a purpose.
• the comes away marked as not any store, a particular one we already know.

No word is understood in isolation. Each word’s numbers are shaped by what is blended in from the rest. And here is the thing people get backwards: those numbers are not a smaller version of your sentence. They are many, many times larger. Five short words come out as thousands of numbers, which represent the entire phrase’s meaning as captured by the math and the model.

It is abstract. No single number means anything. You cannot point at one and ask, “Is this the purpose?” There is no slot for purpose, and no slot for store. All the meaning is in the pattern, across all the numbers at once.

The fingerprint

That abstraction is what makes it possible to emit that fingerprint as another language. The people who created this algorithm were working on translation from one language to another, and on the hard problem of languages that order words differently from one another. The beauty of it is that the fingerprint holds the complete idea, not the word sequence. The same idea in another order, or in another language, produces a close match.

A fingerprint works exactly this way as an abstraction of you. It is not a description of your finger, it is a precise pattern. No single ridge means anything by itself, but the pattern carries who you are completely. Your print from a different day is a close match, and everyone else’s is not.

Same here. I went to the store and I drove to the shop produce similar patterns. A sentence that means something different produces one that is far away.

The pattern does not hold the words, it holds the complete idea, and where that complete idea sits relative to every other complete idea in the model. So I could describe the interpretation of the original as: a speaker who just went to a specific store, probably for a reason. But that sentence is us looking at the numbers and reconstructing the closest human description, the way you describe a face from a face scan.

Step two: score

Once the prompt has been processed, the system can do the math to predict what comes next. It runs the fingerprint to score words in its vocabulary, tens of thousands of them. A score for how well does this word fit the situation. A small number of words score well, and most will be zero. For our sentence, the top of the card looks like this:

• to is high. I went to the store to get milk. Store trips usually have a purpose, and purpose is the natural next beat.
• because is high. Because I was out of coffee. The storyteller is about to explain the reason.
• after is in the middle. After work. Possible, but time comes after purpose in people’s heads.
• near is low. The pattern already encodes a particular store we both know, so describing where it is feels redundant.

Step three: speak

At this point, a few words sit high, a few sit low, and ninety-nine percent of the vocabulary sits near zero. Then the system acts. It takes the tallest peak and rolls a weighted die. The die belongs to the system, and how this selection happens is something we can choose.

If we do it one way, the system will produce the same output from the same input every time. But we can make it a bit random to make it more creative, for example by picking randomly from the top few choices. Two consequences:

• The same prompt can give you different stories, because the landscape is the same and the selection differs.
• Nothing in the output is a fact. It is the shape of what fits, not what is true.

The file knows nothing. The system applies, and a ranking comes out.

Meaning as a whole, not as a list

If you change one word, the score of another one moves. Change the sentence to I went to a store, and near jumps. A store near the bakery now fits perfectly. The score of one word moved because the idea changed. That is meaning as a whole, not as a list.

Now something harder. The last sentence stays the same, but the idea in front of it changes:

I needed to change the oil in my car. I went to the store.

What comes next? When the system runs this, the words in the second sentence do not just blend with each other, they blend with the words in the first sentence too, and this is the point that is easy to miss: nothing is carried over as a summary. There is no little note saying car oil change that gets passed along. Every word still has a direct line of attention to every word in front of it.

So store comes away marked as the place you get car oil, not a generic store, a store for the oil. The fingerprint of the whole thing is different from before, it carries a car, an oil change that is needed, a specific store, a live goal, and the top of the scorecard moves. to was the top pick, now for the oil is on top. The purpose is already on the table.

Consider the same move, but a different situation:

My partner called to remind me we are out of eggs. On my way home, I went to the store.

The last sentence is again five words. The system runs the whole thing, and the fingerprint is different again. Out of eggs, reminded, on the way home. And look at what moves: for is high, and to get is high, but because is low. Remember, in the first sentence because was a top pick. Here the reason is already given, so explaining it again is redundant. And for the oil is near zero. The car was never in this situation, so it is not in the fingerprint, and that word does not fit.

I went to the store is five words, but the fingerprint in front of it carries the complete idea. The system is not matching the last sentence against a list of what usually follows those five words. It is modeling the complete idea those words represent.

Does it understand?

Does the model file understand anything? No. A file of numbers can’t. But the fingerprint the system builds using it feels like a thought, and when you do it over and over again, it looks like thinking.

Nudge the sentence, and the pattern moves the right way. Change one word, and the ranking changes the right way. But that is not understanding. It is a map of meaning that seems practically the same.

It will take time and experience to apply this to the prompts you write. But understanding the importance of words as ideas, and using that understanding to write effective instructions to get what you need, is a core skill for using AI applications.

What this unit left for next

At this point I need to point out that what I just told you was a bit oversimplified, and an expert could rightly take issue with some of it, especially where I said “many thousands of numbers” when talking about the fingerprint and the output. Those numbers are the math behind the fingerprint, the model, and the output, collectively.

So next time, we get a bit more precise and define a few terms, and start to put it all together: what is really going on when you use this kind of application, and what the systems you are using are doing with the models and the other pieces to make the brain seem to be alive.

The video

The full recording is here. Timestamps:

• 00:00 Intro: the course on using and building AI systems and applications, made for everyone, no math.
• 00:52 The model is not a person and not a machine that is running. It is a file.
• 01:12 It does not learn, it has no memory, and it is not alive.
• 01:30 The system around it: the model runner loads the file, applies its numbers to your text, and prints what comes out.
• 01:42 The system is the kitchen, and you bring the ingredients. Three steps: listen, score, speak.
• 01:59 The sentence is I went to the store. The question is, what comes next?
• 02:13 Listen: each step rewrites the numbers for each word by blending in the numbers of the others.
• 02:28 Self-attention: it is what the words mean to each other as an entire phrase.
• 02:58 Five words come out as thousands of numbers. No slot for purpose, no slot for store.
• 03:31 The fingerprint holds the complete idea, not the word sequence, and where it sits relative to every other idea.
• 03:54 The fingerprint analogy: your print from a different day is a close match, everyone else’s is not.
• 04:54 Score: run the fingerprint over the vocabulary. to and because are high, near is low.
• 05:45 Speak: take the tallest peak, roll a weighted die, and choose how random to make it.
• 06:26 Change one word, and the score of another moves. Meaning as a whole, not as a list.
• 06:48 Change the idea in front of it, and the scorecard moves. for the oil is now on top.
• 07:49 The same move, different situation: out of eggs, on the way home. for and to get are high, because is low.
• 08:29 Five words, but the fingerprint in front of them carries the complete idea.
• 08:41 Does the model file understand anything? No. But it looks like thinking.
• 09:21 It is a map of meaning that seems practically the same, and why that matters for the prompts you write.

Notes

The transcript for this article was produced automatically from the video’s audio, and it misheard two names. The course is by Doug Ware, founder of Elumenotion and creator of GuideAnts, the free, open source AI operating system, and that is how they are written here. The audio was transcribed as “Illumination” and “Guidance”.