Building Your OwnAI

Most explanations of language models are either three equations or three hundred pages. This is the middle: enough to actually plan a build, decide what data you need, and know what you are looking at when the numbers on screen start moving.

No code here on purpose. Code goes stale and every framework spells things differently. The shape of the thing does not change, and the shape is what nobody explains.

Chapter 1What a language model actually is

It predicts the next word. That is the whole job.

Everything else, answering questions, writing code, having a personality, falls out of doing that one thing extremely well over a very large amount of text. Nobody programs in "be helpful." You train next-word prediction, and helpfulness is a behaviour that emerges when the training text contains helpful exchanges.

This matters practically: you never write rules. If the model does something wrong, the fix is almost always in the data, not in a rule you add afterward. A hand-written rule is a patch over a hole in the training set.

Chapter 2The parts, and what each one does

A model is a stack of identical layers. Text goes in the bottom, comes out the top as a prediction. Each layer does the same two jobs.

Job one: attention (looking around)

Attention lets every word look at the other words and decide which ones matter. In "the dog that chased the cat was tired," attention is what connects "tired" back to "dog" and not to "cat." Without it a model only sees the previous word or two, which produces text that is topically right and grammatically scrambled.

Job two: the feed-forward block (thinking about it)

After looking around, each word gets processed on its own through a small expansion-and-contraction. This is where most of the model's actual stored knowledge lives, roughly two thirds of the parameters.

The supporting cast

Chapter 3Parameters: what people mean by "a billion"

You hear that a model "has 7 billion parameters" and it sounds like a filing cabinet with 7 billion facts in it. It is not. It is closer to 7 billion dials.

Picture an old mixing desk, the kind with rows of knobs. Sound goes in one end, comes out the other, and every knob nudges what happens in between. Nobody labelled the knobs. Nobody decided what any single one does. You just turn them until the output sounds right.

A parameter is one of those knobs. It is a single number, usually something small and unremarkable like 0.0384 or -1.27. On its own it means nothing at all, and you cannot look at one and say what it is for.

So who turns them?

Nobody. That is the whole trick, and it is the part people find hardest to believe.

Training is a loop that runs millions of times:

1. Show the model some real text with the next word hidden.
2. Let it guess.
3. Compare its guess to the actual word.
4. Nudge every dial a tiny amount in whichever direction would have made that guess better.
5. Do it again with different text.

Step 4 is the whole of machine learning. The nudge is tiny on purpose, far too small to fix the current mistake. It is not meant to. Each nudge is a fraction of a correction, and the model only becomes good because it gets billions of them, from billions of different sentences, and the pressures that keep pointing the same way accumulate while the random ones cancel out.

Nobody chooses what any dial means. Meaning arrives on its own: some dials end up tracking whether the subject is plural, some end up caring about the tone of a sentence, some do jobs nobody has a word for. You do not assign these roles and you cannot read them off afterward. They emerge because they were useful for guessing the next word.

Why more dials is not automatically better

More dials means more capacity to represent things. It also means more dials that need enough examples to settle sensibly. Give a model 600 million dials and only a few million sentences to learn from, and most of those dials never receive enough pressure to point anywhere useful, they end up memorising the training text rather than learning language, which has its own chapter, because it is the failure that hides best (Chapter 11).

That is the ratio in the next chapter, and it is the reason a small well-fed model beats a large starved one.

Chapter 4Size: picking your numbers

Four dials decide how big your model is:

• Width: how many numbers represent each word. 256 is tiny, 1536 is small-serious, 4096+ is large.
• Depth: how many layers stacked. 6 to 24 for anything you train at home.
• Heads: how many things attention can track at once. Usually width divided by 64 to 128.
• Context: how many words it can see at once. This one costs memory fast.

Real configurations from a working build, so these are not invented:

Chapter 5Data: the part that decides everything

This chapter matters more than the rest combined. The architecture is nearly a solved problem you can copy. The data is where your model becomes good or stays useless, and it is where nearly all the real work goes.

The trap nobody warns you about

The easiest text to get legally is old: public-domain books, out-of-copyright encyclopedias. It is free, it is clean, and it will give your model the voice of a Victorian gentleman. It will not know what a phone is. Everything it says will sound like 1890 because it is 1890.

You need modern text, and you need conversation, and those are different problems from getting text at all.

The kinds of data, and what each buys

Quantity, honestly

Text is measured in tokens, roughly three-quarters of a word each. A useful yardstick: a full-length novel is about 120,000 tokens. So one billion tokens is around 8,000 books.

• Under 100M tokens: toy. It will produce grammatical noise.
• 1 billion: a small model becomes genuinely coherent.
• 10 billion+: where a mid-size model starts being useful.

Chapter 6The mix, in percentages

Ratios matter as much as volume. A defensible starting mix for a general conversational model:

Chapter 7Tokenizer: turning text into numbers

Models cannot read letters. A tokenizer chops text into pieces and assigns each a number. Pieces are usually chunks of words: "unhappiness" might become "un" + "happi" + "ness."

Why chunks, and not just letters or whole words

Both obvious options are bad, and understanding why tells you what a tokenizer is actually for.

One number per letter gives you a tiny alphabet, 26 letters and some punctuation. Lovely and simple. But now the word "unhappiness" is twelve separate steps, and the model has to work out from scratch that those twelve things go together before it can even start on what they mean. Every sentence becomes enormous, and since a model can only hold so many steps at once, you have just cut how much text it can see in one go by a factor of four.

One number per word gives you short sentences, which is what you want. But English has a few hundred thousand words, plus names, plus typos, plus "unhappinesses" which nobody has ever written before. Every word you did not plan for becomes a shrug, the model has literally no number for it. And every word you did plan for needs its own row of storage, which gets expensive fast.

Chunks split the difference. Common words get one piece each because they earn it. Rare words get built from parts. Nothing is ever unspellable, because worst case it falls back to letters. That is the whole trick.

Decisions that matter:

• Vocabulary size. 8,000 for small models, 32,000–128,000 for large. Bigger vocab means fewer tokens per sentence (faster) but a larger embedding table (heavier).
• Train it on your actual data. A tokenizer built on different text will waste tokens on yours.
• Lock it before training. Changing the tokenizer afterward invalidates the entire model. There is no migration. You start over.

Chapter 8The actual kit: what to install, where to get data

Everything so far is the shape of the thing. This is the shopping list: names, what each one is for, and where it lives.

The one non-negotiable piece

PyTorch is what nearly everyone trains in. It runs the maths on the graphics card and, critically, works out the nudge for every single dial automatically, so you never write that part yourself. If you install one thing, this is it. Match the version to your card's CUDA release; the site gives you the exact command.

The alternative is JAX, which is genuinely good and common in research. But the tutorials, the example code and the answers to your questions are overwhelmingly PyTorch. Start where the help is.

Things you do not have to write yourself

Where the data actually comes from

This is the part that is hardest to find written down anywhere, and it is the part that decides whether your model is any good.

Code worth reading before writing your own

• nanoGPT: a complete, trainable model in a few hundred readable lines. If you read one thing, read this one.
• build-nanogpt: the same thing constructed step by step, with a video walkthrough.
• minbpe: the tokenizer algorithm, minimal and clear.
• Llama reference code: a production architecture small enough to read in an afternoon.

Hardware, honestly

Memory matters more than speed. A slower card with more of it beats a faster card that cannot hold your model.

When it breaks

• PyTorch docs and forums: your error has been seen before.
• The Hugging Face course: free, practical, current.
• "Attention Is All You Need": the original paper. Short, and more readable than its reputation.
• The Chinchilla paper: where the data-per-parameter ratio in Chapter 4 comes from.

Chapter 9The training run, start to finish

1. Gather and clean: deduplicate, strip junk, fix encoding. Slow, unglamorous, decides your outcome.
2. Train the tokenizer on a sample of the final mix.
3. Pack: convert all text to token numbers in one big file. Do this once; doing it per-run wastes hours.
4. Smoke test: 50 steps at full size. Catches memory blowups before you have burned a night.
5. Train: the long part. Hours to weeks.
6. Fine-tune: shorter runs on specific data to add behaviour or personality.
7. Shrink and ship: compress for the target device.

The knobs during training

Chapter 10Reading the numbers while it runs

A training run prints a line every few seconds. It looks like this, and this one is real:

step 2457/3000  lr 5.27e-05  loss 2.550  140,221 tok/s

• step: progress. 2,457 rounds done of 3,000.
• lr: learning rate, shrinking on purpose as it settles.
• loss: the number that matters. How wrong its guesses are. It should fall.
• tok/s: throughput. Only useful for estimating time remaining.

What loss values mean

Chapter 11When it stops learning and starts reciting

Every number on your screen will tell you this run is going well. That is the problem, and it is worth its own chapter because the previous chapter's advice cannot catch it.

What is actually happening in there

Think about how a child learns to read. Show them a picture book fifty times and they will "read" it to you perfectly with the book closed. They have not learned to read. They have learned that book. Hand them a new one and you find out.

A model has exactly two ways to get better at predicting your text, and it does not get to choose which:

• Learn the rules. Adjectives come before nouns. Questions end in question marks. This character is blunt and uses short sentences. Rules are cheap, one rule covers thousands of sentences, so a small number of dials can hold a lot of them.
• Learn the answers. "After the words the cat sat on the comes mat." That is expensive, it buys you one sentence. But it is worth infinite points on that one sentence.

Training is a hill-climb: each step nudges the dials whichever way makes the next prediction less wrong. On the first pass through your data, rules win easily, because a rule that helps on this sentence also helps on the next thousand it has not seen yet. There is nothing to memorise, every sentence is new.

The second time it sees the same sentence, that changes. Now storing the literal answer pays off, because that exact sentence really will come around again. Given enough repeats and enough spare dials, storing beats generalising, and the hill-climb takes the cheaper win every time. It is not a bug and it is not the model being lazy. You told it to be less wrong on this text, and the fastest way to be less wrong on text you keep showing it is to remember it.

That is also why the trade depends on your two numbers. A small model on a big corpus cannot memorise it, not enough dials to store that much, so rules are the only affordable option. A big model on a small corpus has room to spare and will memorise almost immediately. Same code, same settings, opposite outcome. This is why you must measure it on your own data rather than copying a number out of a guide, including this one.

Why the loss curve cannot tell you

Loss measures one thing: how surprised the model was by the next word. It does not care why the model was unsurprised. A model that has memorised your corpus is not just good at predicting it, it is close to perfect on it, better than any amount of genuine understanding could get. So memorising does not make the curve look bad. It makes the curve look fantastic. Ours reached 0.2 and I was pleased with it.

The standard answer is to hold text back, split off a slice the model never trains on, measure loss on that, and trust it. It is good advice, it is better than nothing, and it is not enough. Here is a real measurement, one small model fine-tuned on one 4.85M-token corpus at five different repeat counts:

Read the last two rows against each other. Between 1.5 passes and 2.0 passes the held-out loss got better, 2.930 down to 2.750, on text the model had never seen, while the longest verbatim quote went from five words to twenty-five, and the share of copied text went from 7% to 32%. One quarter of one extra pass.

The held-out number did not merely fail to warn. It pointed the wrong way. If you had been watching it, you would have concluded that the reciting model was the better one and shipped it.

Why held-out loss missed it

Because both things were happening at once, and held-out loss only sees one of them.

Your held-out slice comes from the same corpus, so it is the same voice, the same subject, the same format. A model getting better at the general shape of your data will score better on it. That was real, and it is why the number improved.

Meanwhile the model was also stuffing literal passages into its spare capacity. That work is invisible to a held-out sentence, a memorised paragraph does not help predict a sentence the model has never seen, but it does not hurt either. It just sits there. The held-out number does not go up. It has no opinion.

So the two effects do not cancel and they do not fight. One shows on the meter and one does not. You watch a needle that is genuinely moving in the right direction while something the needle cannot see gets worse underneath it.

The classic textbook picture, held-out loss bottoms out and then climbs while training loss keeps falling, is real, but it is what happens well past the point where you should have stopped. By the time the meter turns around you have been reciting for a while. Waiting for the turn means catching it late.

The test that does work

Stop asking a number whether the model is copying. Check whether it is copying.

Take a passage from your training data, the stuff it did see, which is the opposite of what you were told to do, and that is the point. Feed the model the first half. Compare what it produces against the real second half.

A model that learned the rules writes something that fits: right voice, right subject, different words. It cannot reproduce the original because it never stored the original. A model that memorised hands the original straight back, because that is the highest-scoring thing it can say and it has it on file.

This works where loss fails because it is not a measure of quality at all. It is a measure of overlap. You are not asking "is this good", you are asking "is this the same", and a model cannot be accidentally word-for-word identical to a paragraph it never memorised. Thirty lines of code:

prefix, truth = passage[:60], passage[60:120]
guess = model.generate(prefix, max_new_tokens=60)

overlap = shared_words(guess, truth) / len(truth)
longest = longest_common_run(guess, truth)

# Over ~8 samples: overlap above 0.5, or any run of
# 12+ words, means it is reciting.

Two thresholds, and here is where they come from. Overlap is what share of the real continuation's words show up in the model's version, it will never be zero, because "the" and "and" appear in both, and normal English on the same subject shares a lot of vocabulary. Ours sits around 2-7% when the model is behaving. Half the words matching is not coincidence.

Longest run is the killer, and it is the one to trust. Two people writing about the same thing might land on the same four or five words in a row. Twelve in a row does not happen by chance, the odds compound with every word, and past about ten there is no innocent explanation left. Our clean runs top out at 5. The reciting one hit 25, which is most of a paragraph.

Run it on samples from several parts of the corpus, not one, because a model can memorise the distinctive passages and generalise on the boring ones. Run it on a checkpoint partway through, not at the end, the whole point is finding out before you have paid for the full run, and if you are only going to run it once, run it early.

How many passes, then

The honest answer is measure it on your own data. You now know why: the trade between learning rules and storing answers depends on how much spare room your model has, which is your parameter count against your corpus size. Nobody else's number is about your two numbers.

Measuring is cheaper than it sounds. Train short runs at 0.5, 1, and 2 passes and test each one. Ours were four minutes apiece, the whole sweep cost less than the single bad run it replaced. If you are about to spend hours on a fine-tune, spend twenty minutes finding the cliff first.

What we can tell you is the shape of the thing:

• It is a cliff, not a slope. Do not expect recital to creep up so you can spot it. In the run above it went 7% to 32% across a quarter of a pass. There is a number where you are fine and a number just past it where you are not.
• Fine-tuning is where this bites. Pretraining runs over enormous corpora rarely see the same text twice. Fine-tunes use small corpora and people reach for high repeat counts to "make it stick".
• One to two passes is a sane starting bracket for a character fine-tune, and if that surprises you, it surprised us too. We guessed 22 (memorised outright, whole paragraphs coming back), then guessed 4 (still reciting), then measured, and the answer was 1.5.
• Put the number in the code, not in your head. Derive the step count from the corpus size and refuse to start above your ceiling. A warning printed forty lines above a scrolling loss curve is a warning nobody reads, ours printed MEMORISATION RANGE and the run went to completion at 22 passes anyway.

Chapter 12Personality: LoRA, fine-tuning, and not paying twice

You have a base model. It writes decent English and has no self. Now you want a character: a support agent, a narrator, someone with a voice. There are three ways to do that and they cost wildly different amounts.

1. Just ask it (prompting)

Put "you are a helpful pirate" at the front of every conversation. Free, instant, changes nothing permanent. It also eats context space, and the character drifts as the conversation gets long, because nothing about the model actually changed.

Fine for trying an idea. Not how you ship a character.

2. Full fine-tuning

Keep training the whole base model on text in your character's voice. Every dial is still free to move, so the voice sinks in properly.

The catch is cost: you need memory for every dial plus the optimiser's bookkeeping for every dial, which is typically several times the model's own size. And you get a whole new model per character. Ten characters, ten full copies.

It is the right call when you have one character and enough memory. Our own runs do exactly this: the base is trained once over hours, then each character is a short run of a few hundred steps starting from a copy of it, about twelve minutes each, because the grammar is already paid for and only the voice has to move.

3. LoRA, the one most people should use

LoRA stands for Low-Rank Adaptation, and the idea is better than the name.

Rather than adjusting the model's millions of dials, you freeze all of them and bolt a small extra set of dials alongside. Only the small set trains. At runtime the model's original output and the little add-on's output are added together, and the result behaves like a model that learned your character.

Why it works: the change from "general English" to "this specific voice" is a much simpler change than the model itself. It does not need millions of free dials to express. A few million will carry it.

The practical dial in LoRA is its rank: how many extra dials you allow. Small (8–16) is plenty for a voice or an output format. Larger (32–64) if the behaviour is more involved. Bigger is not better here either; too large and it starts memorising your examples instead of learning the pattern.

PEFT implements this and it is a handful of lines to wire in. The original paper is readable. QLoRA combines it with a shrunk base model so you can adapt a large model on modest hardware.

What to feed it

Far less than you would guess. A few hundred to a few thousand examples in the voice you want, and quality beats quantity every time. A hundred sharp examples beat ten thousand mediocre ones.

Write them in the shape the model will actually be used in. If it will answer questions, write questions and answers, not prose about the character. The model learns the form as much as the content, which is why a character trained only on monologue will monologue at you forever, in a perfect voice, about nothing you asked.

Which to pick

• Trying an idea? Prompt it.
• Several characters, or limited memory? LoRA. Almost always the right answer.
• One character, deep change, memory to spare? Full fine-tune.
• Need it to know new facts? Neither, really. Look at retrieval, where the model reads from a document store at answer time. Teaching facts by fine-tuning is expensive and unreliable.

Chapter 13How much room each number takes

Every one of those billions of dials has to be stored somewhere, and each takes up a certain amount of memory. How much room you give each one decides how big a model fits on the hardware you already own.

Think of it as how many decimal places you keep. A number stored generously might be 0.038471629. Stored leanly it becomes 0.038. Same number, less room, slightly less exact. Cut the storage in half and your model is half the size. Cut it to a quarter and a model that needed four graphics cards now needs one.

The two places precision matters, and they are not the same

People conflate these constantly, and the difference is the whole story.

Shrinking a finished model is easy and everyone does it. Train it generously, then round all the dials down afterward. This is called quantization. It works well, it is well understood, and it is why you can run a capable model on a laptop.

Training in low precision is a different problem entirely, and it is genuinely hard.

Remember that each training nudge is tiny, deliberately far smaller than the error it is correcting. Now store the dial with very few decimal places. The nudge is smaller than the smallest change that storage can represent. So the dial receives the nudge and does not move. Then it happens again. And again.

The model stops learning entirely. Not slowly, not badly. It sits there while the training loop runs, and the numbers never change. Everyone hits this, which is why nearly all training happens at generous precision and the shrinking is saved for the end.

It has been done, and the point is quality

This is not hypothetical. The remaining hard part, getting the whole model into the lean format without training falling apart, has been solved and run to completion on an ordinary consumer graphics card. The published techniques did not transfer; what was actually killing the training turned out to be something not previously written down, and fixing that was the work.

The result that matters is not the smaller file, and it is not about avoiding expensive hardware. It is this: on the same card, you can now train a bigger model, and the bigger model trained in the lean format beat the best smaller model trained the generous way.

So it never was a tradeoff where you accept a worse model to save money. Same hardware, better model. From here the constraint is not what you can afford to run on. It is your corpus, your mix, and your patience, which is why the data chapters in this manual are the long ones and this one is short.

Chapter 14Checking your corpus before you spend the hours

A training run turns text into a model and cannot tell you whether the text was right. It will faithfully learn whatever is in there, and you find out hours later by talking to the result. This chapter is the set of checks worth running first, because every one of them costs minutes and every one of them has caught something real.

Check the packed file, not the script that made it

Your corpus goes through a pipeline: read files, clean them, filter them, tokenize, write a binary. Every stage is a place something can go quietly wrong, and the log line at the end reports only that the stage ran.

A packer printing 39,596 names replaced tells you the substitution executed. It does not tell you the result reads correctly. Decode the finished binary back into text and search that. It is a few lines of code and it is the only check with nothing between it and the model.

Worth building as a standing script rather than a one-off, because you will run it before every pack. It should assert the things you believe are true: no foreign names, no markup, the character's own name present at a healthy rate, and exit non-zero when one fails.

A substitution is not a rename

Filters replace one string with another, and the replacement lands in a context the filter cannot see.

Replacing a name with a generic word inside a dialogue corpus formatted as Name: what they said does not remove the speaker tag. It renames it. Thousands of instances of Assistant: at the start of a line teach the model that this string is what precedes speech, and it will emit it. The name is gone and the damage is new.

Before running a filter across a corpus, run it across a sample and print each changed line before and after. Read them. It takes a minute and it is the only way to see the context the pattern is landing in.

Watch out for filters that fight your own data

If you write a file teaching a character to reject a name (are you [other name] / No. I'm [character].) and you also run a filter that scrubs that other name from the corpus, the two cancel. The rejection line becomes are you the assistant and you have taught the exact opposite lesson.

This is invisible in both files. Each is correct alone. Only the diff shows it.

Count what your character says about itself

Identity failures come in two flavours and they look identical from outside.

Contamination is a foreign name present in the corpus. A filter fixes it.

Dilution is your character's own name being too rare to learn. Nothing filters that. In one of our persona corpora the character's name appeared 64 times across 21.7 million characters, 0.0003%, because the general-conversation padding vastly outweighed the hand-written voice. The model learned a personality and never learned what it was called.

So count it. A persona corpus should mention its own character in the low thousands, not the dozens. If four of your nine voice files never use the name at all, that is the number to fix before you train.

Your padding is a voice too

Fine-tuning one character on a narrow set of examples will drift, so most people add a slice of general text to hold the grammar steady. Reasonable. But whatever that slice is written in becomes the model's fallback register, what it produces when a prompt gives it nothing specific to grab.

Public-domain dialogue is the tempting choice because it is free and it looks like conversation. It is also a century old. A modern character padded with it will answer ordinary questions in period idiom, complete with characters who are not in your corpus at all. Identity checks pass cleanly; the register is what is wrong, and no name filter can see it.

Pad with text from the era you want the model to sound like. Quality-filtered modern web prose has no characters and no speaker tags, so drifting into it costs you tone rather than replacing it.

Test the form, not just the content

A model trained on loose lines learns a voice. A model trained on question-then-answer pairs learns to answer. These are different skills and the first does not imply the second.

The symptom is distinctive: you ask something ordinary and get a reply in exactly the right voice that has nothing to do with what you asked. The character is intact. The comprehension was never trained. Fix it in the data, write pairs where the answer is specifically about the question, rather than reaching for sampling settings, which cannot manufacture a relationship that was never in the text.

Read the file you were handed

Corpora arrive from other people and from your own past self. Open them and look before they go in.

A persona file we received was 7.66 MB, and the character's real dialogue in the same folder was 35 KB. The big file was a configuration document written about the character in the third person. At 200:1 the model dutifully learned to recite its own spec sheet, grammatical, confident, and completely wrong. A quick ls would have shown the ratio; opening the file would have shown everything.

Chapter 15What goes wrong

If you are starting today

1. Build the small config, not the large one. Get all the way through the pipeline once.
2. Gather 1 billion tokens in roughly the Chapter 5 mix.
3. Train until loss is around 2.0, then look at what it says.
4. Fix the data, not the architecture. That is where the improvement is.
5. Only then go bigger.