Day 08 · Overview

Week 2FNX 101 · 08

Machines That Read

Tokenizers, embeddings, and attention: open up GPT-2, watch it predict one token at a time, and build your own language lab.

Today you will

See how text becomes tokens and then embeddings
Watch a language model predict the next word and learn what temperature does
Peek inside attention: every word choosing which other words to look at

Today's items

2 h 30 min

Start: Slides

Lab

FNX 101 · 08

day08_language_lab.ipynb0/21 cells runPython 3

Day 8 — Machines That Read 💬

Mission briefing: Your Studio can see. Today it learns to read — and to write. We crack open the machinery behind ChatGPT-class systems: transformers, powered by one absurdly simple idea repeated a trillion times — predict the next word.

Previously in the Lab: On Day 7 you fine-tuned a giant with transfer learning and unlocked 🪨📄✂️ RPS Arena.

Today you will:

See how text becomes tokens and then embeddings (words as arrows in space)
Watch a language model predict the next word, step by step, and learn what temperature does
Tour four pipelines: mood, fill-in-the-blank, topic-sorting, and story writing
Peek inside attention — every word choosing which other words to look at
Learn why models produce plausible text, not true text

Today you unlock: 🔓 Language Lab — four language superpowers in your Studio, running on the transformer models you explore today.

A precise reading

GPT-2 predicts the next token, which may be a word fragment. Encoder models such as DistilBERT can use both sides of a masked token. Tokenization can make letter counting harder but does not make it impossible. An attention heatmap shows model weights, not proof of grammatical reference or a complete explanation of a prediction. A lower temperature changes sampling, not factual reliability.

# ╔═══════════════════════════════════════════════════════════════════╗
# ║ SETUP — run me first! (identical in every Lab notebook)           ║
# ╚═══════════════════════════════════════════════════════════════════╝
import os, sys, pathlib

IN_COLAB = "google.colab" in sys.modules
SMOKE = os.environ.get("SMOKE_TEST", "0") == "1"   # tiny fast run for automated tests

REPO_URL = "https://github.com/A-Halimi/ai-studio-internship.git"  # instructor: set once

if IN_COLAB:
    if not pathlib.Path("/content/ai-studio-internship").exists():
        !git clone -q {REPO_URL} /content/ai-studio-internship
    %cd /content/ai-studio-internship/notebooks/day08
    %pip -q install transformers==5.13.0 gradio==6.19.0

HERE = pathlib.Path.cwd()
REPO = HERE.parents[1]                       # .../notebooks/dayNN -> repo root
DATA_DIR = pathlib.Path(os.environ.get("COURSE_DATA_DIR", REPO / "data"))
SAMPLES_DIR = REPO / "data_samples"          # small datasets shipped with the repo
MODELS_DIR = REPO / "ai_studio" / "models"   # where your Studio modules unlock
for p in (REPO / "data", MODELS_DIR):
    p.mkdir(parents=True, exist_ok=True)

import random
import numpy as np
SEED = 42
random.seed(SEED); np.random.seed(SEED)
print(f"✅ Setup done | Colab: {IN_COLAB} | Smoke test: {SMOKE}")
print(f"   data: {DATA_DIR}")

import torch
torch.manual_seed(SEED)
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
if DEVICE.type == "cuda":
    print(f"⚡ Running on GPU: {torch.cuda.get_device_name(0)}")
else:
    print("🐢 No GPU found — running on CPU. Everything today still works, just a little slower.")

# ── CONFIG — the Lab's control panel ───────────────────────────────
GEN_TOKENS = 20 if SMOKE else 60    # how many new words the story writer adds

# First run downloads a few small models from Hugging Face (~1.5 GB total:
# gpt2 + three distilbert models). They're pre-baked in the course Docker image;
# on a laptop or Colab they download once and then cache. Grab a coffee. ☕

1. The idea that changed everything ⚡

In 2017 a paper called "Attention Is All You Need" introduced the transformer. Within a few years it powered GPT, translation, search, and the chatbots you use today. The wild part: under all the hype sits one tiny training goal — look at some text, predict the next token, and repeat. Do that on a big chunk of the internet and the model soaks up grammar, facts, styles, and a surprising amount of reasoning.

Today we don't just use these models — we look inside one. We'll use GPT-2, the small, open ancestor of ChatGPT (124 million parameters — tiny by today's standards, perfect for learning).

2. Step 1 — text becomes tokens 🔤

GPT-2 receives token IDs rather than a separate input for every letter. First it chops text into tokens — common chunks that are often whole words, sometimes word-pieces. GPT-2 knows exactly 50,257 of them. Every token maps to a number, and that's all the model ever sees: a list of numbers.

Run the demo — notice the little Ġ marks a space in front of a token.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
pieces = tok.tokenize("The penguins in the lab love matrix multiplication")
print(pieces)
print(f"\n{len(pieces)} tokens for that sentence.")

🧪 Exercise 1 — Find a word that shatters

Some words are a single token; rarer ones get split into pieces. Below, several words are tokenized for you. Collect the ones that split into 3 or more pieces.

tok.tokenize(word) returns the list of pieces
keep the words where len(pieces) >= 3

You should find that long or unusual words (like a full name) shatter the most.

Exercise

candidates = ["supercomputer", "Abdelghafour", "strawberry", "penguin", "robot", "cat"]
splits = {w: tok.tokenize(w) for w in candidates}          # given: tokenize each word
for w, t in splits.items():
    print(f"{w!r:16} -> {len(t)} piece(s)  {t}")

# ==================== YOUR CODE HERE ====================
### HINT: [w for w, t in splits.items() if len(t) >= 3]
...
print("\n3+ pieces:", three_plus)
assert any(len(t) >= 3 for t in splits.values()), "Expected at least one word to split into 3+ pieces."

🧪 Exercise 2 — The strawberry problem 🍓

Ever seen a chatbot miscount the letters in a word? Here's why. Tokenize "strawberry" and count the pieces. The model never sees ten individual letters — it sees a couple of chunks. Counting the r's requires recovering character structure from those chunks; a model can learn this, but may make mistakes.

Set n_pieces to how many tokens "strawberry" becomes.

Exercise

pieces = tok.tokenize("strawberry")
print("strawberry ->", pieces)

# ==================== YOUR CODE HERE ====================
### HINT: n_pieces = len(pieces)
...
print(f"The model sees {n_pieces} chunk(s), not 10 letters — so 'how many r's?' is genuinely hard for it.")

# Every token has an ID; encode -> IDs, decode -> back to text (a perfect round-trip).
print("Vocabulary size:", tok.vocab_size)               # 50257
ids = tok.encode("Matrix multiplication is everywhere.")
print("IDs:    ", ids)
print("Decoded:", tok.decode(ids))

3. Step 2 — tokens become arrows 🏹

A number like token 1820 means nothing on its own. So the model gives every token an embedding: a list of 768 numbers — think of it as an arrow pointing to a spot in a 768-dimensional space. The magic is that meaning becomes geometry: words used in similar ways end up near each other.

We can't picture 768 dimensions, so we squish them down to 2 with PCA and plot. Watch related words cluster.

from transformers import AutoModelForCausalLM
from sklearn.decomposition import PCA

gpt2 = AutoModelForCausalLM.from_pretrained("gpt2").to(DEVICE).eval()
emb = gpt2.transformer.wte.weight        # (50257, 768): one 768-D arrow per token

# GPT-2 tokens usually carry a leading space, so we write " king" not "king".
words = [" king", " queen", " man", " woman", " boy", " girl",
         " cat", " dog", " pizza", " bread", " fire", " water",
         " science", " music", " happy", " sad", " summer", " winter"]
words = [w for w in words if len(tok.encode(w)) == 1]        # keep true single-token words
vecs = torch.stack([emb[tok.encode(w)[0]] for w in words]).detach().cpu().numpy()

xy = PCA(n_components=2, random_state=SEED).fit_transform(vecs)
import matplotlib.pyplot as plt
plt.figure(figsize=(8, 6))
plt.scatter(xy[:, 0], xy[:, 1])
for (x, y), w in zip(xy, words):
    plt.annotate(w.strip(), (x, y), fontsize=11)
plt.title("GPT-2 word embeddings (768-D) squished to 2-D")
plt.xlabel("PCA dimension 1"); plt.ylabel("PCA dimension 2")
plt.tight_layout(); plt.show()

🧪 Exercise 3 — Put your own words on the map

Add three of your own words to my_words. Two rules so nothing breaks:

start each with a space (e.g. " ocean"), and
pick words likely to be a single token (short, common words are safest — the code quietly drops any that aren't).

Re-run and see where they land. Do the neighbors make sense? Meaning = geometry: each word is really 768 numbers, and closeness on the plot hints at closeness in meaning.

Exercise

# ==================== YOUR CODE HERE ====================
### HINT: three single-token words, each starting with a space, e.g. " ocean"
...

all_words = words + [w for w in my_words if len(tok.encode(w)) == 1]
vecs2 = torch.stack([emb[tok.encode(w)[0]] for w in all_words]).detach().cpu().numpy()
xy2 = PCA(n_components=2, random_state=SEED).fit_transform(vecs2)

new_set = {w for w in my_words if len(tok.encode(w)) == 1}
plt.figure(figsize=(8, 6))
for (x, y), w in zip(xy2, all_words):
    is_new = w in new_set
    plt.scatter([x], [y], color="crimson" if is_new else "steelblue")
    plt.annotate(w.strip(), (x, y), fontsize=11,
                 color="crimson" if is_new else "black")
plt.title("Your words (red) among the neighbors")
plt.xlabel("PCA dimension 1"); plt.ylabel("PCA dimension 2")
plt.tight_layout(); plt.show()

4. ★ The whole secret — predict the next word 🔮

Here it is, the entire engine. Given some text, GPT-2 outputs a probability for every one of its 50,257 tokens — how likely each is to come next. Pick one, stick it on the end, and ask again. That loop is all that "generating text" means.

Let's watch it. The helper below returns the model's top-5 guesses for what comes after a prompt.

🔢 Math Corner — how much computation is one word? (optional)

Each next-word guess flows through GPT-2's 12 transformer blocks, each with an attention step and a small MLP — hundreds of matrix multiplications across 124 million parameters, per token. Generating a sentence does this dozens of times. Making those multiplications scream on a GPU is the whole field of high-performance computing.

import torch.nn.functional as F

@torch.no_grad()
def next_token_probs(prompt, k=5):
    """Return the model's top-k (token, probability) guesses for what comes next."""
    ids = tok.encode(prompt, return_tensors="pt").to(DEVICE)
    logits = gpt2(ids).logits[0, -1]          # scores for the token AFTER the prompt
    probs = F.softmax(logits, dim=0)          # turn scores into probabilities
    top = torch.topk(probs, k)
    return [(tok.decode([int(i)]), float(p)) for p, i in zip(top.values, top.indices)]

print("Prompt: 'The best thing about science is'\n")
for tk, p in next_token_probs("The best thing about science is"):
    print(f"{p:6.1%}   {tk!r}")

# Greedily glue on the single most likely token, five times. Watch the sentence assemble.
text = "The best thing about science is"
for step in range(5):
    top = next_token_probs(text, k=5)
    nxt = top[0][0]                                        # greedy = always pick #1
    options = ", ".join(f"{t.strip()!r}:{p:.0%}" for t, p in top)
    print(f"step {step + 1}: ...{text[-40:]!r}")
    print(f"        choices -> {options}\n")
    text += nxt
print("Final sentence:", repr(text))

🧪 Exercise 4 — Predict from your own prompt

Set my_prompt to any sentence-start you like and look at the model's top-5 next-word guesses. Try something factual ("The capital of France is") and something silly — the shape of the probabilities changes with how "sure" the model is.

Exercise

# ==================== YOUR CODE HERE ====================
### HINT: any string, e.g. "In the year 3000, computers will"
...

for tk, p in next_token_probs(my_prompt):
    print(f"{p:6.1%}   {tk!r}")

5. Four superpowers — the pipeline tour 🧰

Writing model code by hand is a lot. Hugging Face pipelines wrap a model + tokenizer into a single call. We'll tour four, each a different transformer trained for a different job. First up: text-classification — reading the mood (sentiment) of a sentence.

from transformers import pipeline

# device=DEVICE puts it on the GPU when there is one; plain CPU works fine too.
clf = pipeline("text-classification",
               model="distilbert-base-uncased-finetuned-sst-2-english", device=DEVICE)

for s in ["This course is the best thing ever!",
          "I am so bored and tired today.",
          "The GPU cluster caught fire again."]:
    r = clf(s)[0]
    print(f"{r['label']:8} {r['score']:.0%}   <- {s}")

🧪 Exercise 5 — Fool the mood detector 😏

These models read surface words, not deep meaning. Write a sarcastic sentence — cheerful words, sour meaning — that gets labeled POSITIVE even though a human hears the opposite. (Something like "Oh wonderful, another bug.")

Exercise

# ==================== YOUR CODE HERE ====================
### HINT: sarcasm works well — happy words, unhappy meaning
...

r = clf(tricky)[0]
print(f"{r['label']} ({r['score']:.0%})  <- {tricky}")
print("Did it read the sarcasm, or just the happy words?")

# Superpower 2 — fill-mask: guess a hidden word. Write the blank as ___ and we swap in
# the model's special mask token (that replace trick comes straight from the Studio module).
fm = pipeline("fill-mask", model="distilbert-base-uncased", device=DEVICE)

sentence = "Students in the lab love to ___ every day."
masked = sentence.replace("___", fm.tokenizer.mask_token)
print(sentence, "\n")
for r in fm(masked)[:5]:
    print(f"{r['score']:.1%}   {r['token_str'].strip()!r}")

# Superpower 3 — zero-shot: sort text into ANY labels you invent, with no extra training.
zs = pipeline("zero-shot-classification",
              model="typeform/distilbert-base-uncased-mnli", device=DEVICE)

res = zs("The team trained a neural network on the GPU cluster overnight.",
         candidate_labels=["science", "sports", "cooking", "gossip"])
for lab, score in zip(res["labels"], res["scores"]):
    print(f"{score:5.0%}   {lab}")

🧪 Exercise 6 — Two temperatures 🌡️

Superpower 4 — text-generation: the next-word loop from section 4, automated. The knob that controls it is temperature: how adventurously the model picks from its list of likely next words.

Low (0.2) → almost always the top pick: safe, repetitive, sometimes boring
High (1.2) → happily grabs unlikely words: creative, surprising, sometimes nonsense

Generate the same story starter twice — once at temperature=0.2, once at temperature=1.2 — and read the difference.

Exercise

gen = pipeline("text-generation", model="gpt2", device=DEVICE)
prompt = "Late one night in the lab, a small robot woke up and said"
torch.manual_seed(SEED)          # seed so the demo is the same for everyone

# ==================== YOUR CODE HERE ====================
### HINT: call gen(...) twice — temperature=0.2 for `cool`, temperature=1.2 for `wild`
...

print("🧊 temperature 0.2 (plays it safe):\n", cool, "\n")
print("🔥 temperature 1.2 (takes big risks):\n", wild)

6. Peeking inside — attention 👀

How does a model know that "it" refers to the penguin and not the fish? Attention. At every layer, each word gets to look at every other word and decide how much to borrow from each. That table of "who looks at whom" is literally a grid of numbers — and computing it is (you guessed it) matrix multiplication, the same operation HPC engineers make fly on the GPU.

Let's visualize it. We run a small model (DistilBERT) with output_attentions=True and draw one attention head as a heatmap: bright square at row it, column penguin means "it is paying attention to penguin."

from transformers import AutoModel

dtok = AutoTokenizer.from_pretrained("distilbert-base-uncased")
dmodel = AutoModel.from_pretrained("distilbert-base-uncased",
                                   output_attentions=True).to(DEVICE).eval()

def show_attention(sentence, layer=4, head=0, title=None):
    enc = dtok(sentence, return_tensors="pt").to(DEVICE)
    with torch.no_grad():
        out = dmodel(**enc)
    grid = out.attentions[layer][0, head].cpu().numpy()      # (tokens, tokens)
    toks = dtok.convert_ids_to_tokens(enc["input_ids"][0])
    plt.figure(figsize=(6.5, 5.5))
    plt.imshow(grid, cmap="viridis")
    plt.xticks(range(len(toks)), toks, rotation=90)
    plt.yticks(range(len(toks)), toks)
    plt.xlabel("attending TO"); plt.ylabel("word doing the looking (FROM)")
    plt.title(title or f"Attention — layer {layer + 1}, head {head + 1}")
    plt.colorbar(label="attention weight"); plt.tight_layout(); plt.show()

show_attention("The penguin ate the fish because it was hungry",
               title="'it' should point at the PENGUIN (it was hungry)")

🧪 Exercise 7 — Follow the pronoun

Change the ending so "it" now means the fish instead of the penguin — for example, "...because it was delicious." Re-draw the heatmap and compare: does the bright spot in the it row shift toward fish?

(Attention is subtle and no single head is perfect — but you'll often see the pattern move. Set sentence2 below.)

Exercise

# ==================== YOUR CODE HERE ====================
### HINT: keep the start, change the last word so "it" = the fish (e.g. "delicious")
...

show_attention(sentence2, title="Now where does 'it' look?")

7. Reality check ⚠️

One minute of honesty before you unlock the module. A language model predicts plausible text — words that sound right following the prompt. That is not the same as true text. Watch GPT-2 answer something it knows, then something it can't possibly know — and say both with the exact same confidence.

for prompt in ["The largest planet in our Solar System is",
               "In the year 3000, computers will be able to"]:
    out = gen(prompt, max_new_tokens=GEN_TOKENS, do_sample=False,
              pad_token_id=50256)[0]["generated_text"]
    print(out)
    print("-" * 65)

The first answer is probably fine (that fact appeared all over GPT-2's training text). The second is invented — GPT-2 never saw 2026, so it confidently makes something up. This is called a hallucination, and it's the single biggest gotcha when using these tools: always verify facts.

So how do modern systems (ChatGPT, Claude) get so much more reliable and helpful? They add an extra ingredient on top of next-word prediction: training from human feedback — rewarding good answers, penalizing bad ones. Rewards. Which is exactly tomorrow's topic. 🐕

🔓 Unlock your Studio module

The Language Lab module bundles all four superpowers into your Studio. The transformer models load themselves from the cache — all you ship is a small settings token that personalizes it. Fill in your name and (optionally) rewrite the story starters, then run the cell.

import json

MADE_BY = "an AI explorer at the Lab"     # ← put YOUR name here
DEFAULT_TEMPERATURE = 0.9                       # the Story Writer's starting creativity
STORY_STARTERS = [                              # write at least 3 — make them yours!
    "Late one night in the lab, a small robot woke up and said",
    "The last penguin on the iceberg looked at the satellite and",
    "In the year 3000, the fastest supercomputer on Earth whispered",
]

cfg = {"made_by": MADE_BY,
       "default_temperature": DEFAULT_TEMPERATURE,
       "story_starters": STORY_STARTERS}
(MODELS_DIR / "language_lab.json").write_text(json.dumps(cfg, indent=2), encoding="utf-8")

REQUIRED_FILES = ["language_lab.json"]
missing = [f for f in REQUIRED_FILES if not (MODELS_DIR / f).exists()]
assert not missing, f"Something didn't save: {missing}"
assert len(STORY_STARTERS) >= 3, "Write at least 3 story starters before unlocking."
print("🔓 UNLOCKED: Language Lab! Run your Studio and try all four tabs:")
print("   python ai_studio/app.py   (from the repo root)   ->   💬 Language Lab")

🏁 Checkpoint

Today you looked inside a language model. You watched text become tokens and then embeddings (meaning as geometry), saw the one trick behind it all — predict the next token — and learned that temperature tunes how wild the guessing gets. You toured four pipelines, peeked at attention (every word choosing what to look at — matrix multiplication again), and learned the crucial caveat: models produce plausible, not true, text.

Studio unlocked: 💬 Language Lab — mood, fill-in-the-blank, topic-sorting, and a story writer, all in your app.

Seeing ✓. Reading ✓. But the Studio still can't act. Tomorrow: machines that learn by trial, error, and reward — like training a puppy with treats. Welcome to reinforcement learning. 🐕