Day 2 — The Prediction Machine 🐧

Mission briefing: A field station just radioed in with a spreadsheet of penguin measurements — and no species labels. Their expert is out sick. The Lab's job: build a machine that identifies a penguin's species from its measurements alone, so the station can label the whole backlog automatically. This is your first real model.

Previously in the Lab: as the Data Detective you cleaned the penguin data and spotted three species forming clear clusters in a scatter plot.

Today you will:

• Learn supervised learning: features, labels, and the sacred train/test split
• Build k-Nearest Neighbors from scratch with NumPy, then in 3 lines of scikit-learn
• See a model think, with decision boundaries and a readable decision tree
• Diagnose mistakes with a confusion matrix, and uncover the feature-scaling trap

Today you unlock: 🔓 Prediction Machine — slide four measurements, get a live species prediction in your AI Studio.

A precise reading

Choose k and other settings using validation data or cross-validation within the training set, then evaluate once on an untouched test set. Fit scaling on training data only. A larger k makes a smoother boundary; too large can underfit. Comparing many settings on the same test set makes that score optimistic.

# ╔═══════════════════════════════════════════════════════════════════╗
# ║ SETUP — run me first! (identical in every Lab notebook)           ║
# ╚═══════════════════════════════════════════════════════════════════╝
import os, sys, pathlib

IN_COLAB = "google.colab" in sys.modules
SMOKE = os.environ.get("SMOKE_TEST", "0") == "1"   # tiny fast run for automated tests

REPO_URL = "https://github.com/A-Halimi/ai-studio-internship.git"  # instructor: set once

if IN_COLAB:
    if not pathlib.Path("/content/ai-studio-internship").exists():
        !git clone -q {REPO_URL} /content/ai-studio-internship
    %cd /content/ai-studio-internship/notebooks/day02

HERE = pathlib.Path.cwd()
REPO = HERE.parents[1]                       # .../notebooks/dayNN -> repo root
DATA_DIR = pathlib.Path(os.environ.get("COURSE_DATA_DIR", REPO / "data"))
SAMPLES_DIR = REPO / "data_samples"          # small datasets shipped with the repo
MODELS_DIR = REPO / "ai_studio" / "models"   # where your Studio modules unlock
for p in (REPO / "data", MODELS_DIR):
    p.mkdir(parents=True, exist_ok=True)

import random
import numpy as np
SEED = 42
random.seed(SEED); np.random.seed(SEED)
print(f"✅ Setup done | Colab: {IN_COLAB} | Smoke test: {SMOKE}")
print(f"   data: {DATA_DIR}")

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

print("Toolkit ready 🧰")

Supervised learning: teaching by example

Yesterday you saw the penguin species form clusters. Today you'll teach a machine to draw the boundaries itself — using the most common flavor of machine learning: supervised learning.

The recipe has two ingredients:

• Features (X) — the measurements we know for each penguin (bill length, flipper length, ...).
• Label (y) — the answer we want to predict (the species).

We show the machine hundreds of (X, y) pairs — "these measurements → that species" — and it learns the mapping. Then we hand it new measurements with the label hidden and ask it to guess.

The golden rule: never test on data you trained on. We split the penguins into a training set (the machine studies these) and a test set (locked in a drawer). It's the difference between studying with past exam papers and then sitting a brand-new exam. If you only ever graded yourself on the exact questions you'd memorized, you'd have no idea whether you actually learned anything. Hiding the test set is the only honest way to measure real learning.

penguins = pd.read_csv(SAMPLES_DIR / "penguins.csv").dropna().reset_index(drop=True)

FEATURES = ["bill_length_mm", "bill_depth_mm", "flipper_length_mm", "body_mass_g"]
X = penguins[FEATURES]     # features: a DataFrame with 4 columns
y = penguins["species"]    # label: the species of each penguin

print(f"{len(penguins)} clean penguins")
print(f"Features: {FEATURES}")
print(f"Species : {sorted(y.unique())}")
penguins.head()

🧪 Exercise 1 — Split the data honestly

Use train_test_split to lock away 25% of the penguins as a test set.

• Pass test_size=0.25, random_state=SEED (so the split is reproducible), and stratify=y.
• stratify=y keeps each species at the same proportion in both halves — important when one species is rarer than the others.
• Expected: 249 training penguins and 84 test penguins.

from sklearn.model_selection import train_test_split
# ==================== YOUR CODE HERE ====================
### HINT: train_test_split(X, y, test_size=0.25, random_state=SEED, stratify=y)
...
print(f"Training penguins: {len(X_train)}")
print(f"Testing penguins : {len(X_test)}")
assert len(X_test) == 84, "With 333 penguins and test_size=0.25 you should hold out 84."

Machine #1: k-Nearest Neighbors

The simplest classifier there is — and it works just how you'd guess a stranger's taste in music: look at who they're standing next to.

To label a new penguin, k-Nearest Neighbors (k-NN):

1. measures the distance from the new penguin to every training penguin,
2. finds the k closest ones (its "nearest neighbors"),
3. takes a majority vote of their species.

There's no "training" beyond memorizing the examples. Let's build it from scratch with NumPy so nothing is hidden — using just two features at first, so we can literally see it work: bill length and flipper length.

TWO = ["bill_length_mm", "flipper_length_mm"]
X_train2 = X_train[TWO].to_numpy()
X_test2  = X_test[TWO].to_numpy()

# k-NN votes by tallying labels, so turn species names into integer codes 0/1/2
classes = np.array(sorted(y.unique()))               # ['Adelie', 'Chinstrap', 'Gentoo']
name_to_code = {name: i for i, name in enumerate(classes)}
y_train_codes = np.array([name_to_code[s] for s in y_train])
y_test_codes  = np.array([name_to_code[s] for s in y_test])

print("Two-feature training matrix:", X_train2.shape)
print("Classes:", classes.tolist())

🧪 Exercise 2 — Build k-NN from scratch

Fill in the body of knn_predict. Given one new penguin x_new, it should return the majority species-code among its k nearest neighbors.

• Distance: np.sqrt(((X_train - x_new) ** 2).sum(axis=1)) computes the distance to all training points at once.
• Nearest: np.argsort(distances) returns the indices from closest to farthest — take the first k.
• Vote: np.bincount(votes).argmax() returns the most common code.

def knn_predict(x_new, X_train, y_train, k=5):
    """Predict the label of x_new by majority vote of its k nearest neighbors."""
    # ==================== YOUR CODE HERE ====================
    ### HINT: np.sqrt(((X_train - x_new)**2).sum(axis=1)) gives all distances at once
    ### HINT: np.argsort gives the indices that would sort the array
    ...


# sanity check: predict the very first test penguin
pred_code = knn_predict(X_test2[0], X_train2, y_train_codes, k=5)
print("Predicted:", classes[pred_code], "  |  Actual:", classes[y_test_codes[0]])

🧪 Exercise 3 — Grade it on the test set

Now run your knn_predict on every test penguin and measure the accuracy.

• Build an array preds with one prediction per row of X_test2.
• Expected: accuracy around 0.95 — and the self-check insists on beating 0.80.

# ==================== YOUR CODE HERE ====================
### HINT: call knn_predict on every row of X_test2 and collect the results into an array
...
accuracy = (preds == y_test_codes).mean()
print(f"Scratch k-NN accuracy on {len(X_test2)} unseen penguins: {accuracy:.3f}")
assert accuracy > 0.8, "Your k-NN should beat 0.80 — check the distance and voting logic."

Watch it think: decision boundaries

Here's how to see what the classifier learned. Take every point on the plane, ask the model which species it would predict there, and color the background to match. The result is a map of decision boundaries — the borders between "I'd call this an Adelie" and "I'd call this a Gentoo."

We give you the plotting helper; you supply the model. Let's compare two extremes: k = 1 (trust only the single closest neighbor) versus k = 15 (poll fifteen of them).

from matplotlib.colors import ListedColormap

BG = ListedColormap(["#cfe8cf", "#fdf0c0", "#f6cccc"])   # faint background fills
FG = ListedColormap(["#1b7837", "#b8860b", "#b2182b"])   # bold dot colors


def plot_decision_boundary(predict_fn, X2, y_codes, ax, title):
    """Color the whole plane by the model's prediction, then scatter the data on top.

    predict_fn takes an (N, 2) array of points and returns (N,) class codes.
    """
    res = 60 if SMOKE else 200
    x_min, x_max = X2[:, 0].min() - 1, X2[:, 0].max() + 1
    y_min, y_max = X2[:, 1].min() - 5, X2[:, 1].max() + 5
    xx, yy = np.meshgrid(np.linspace(x_min, x_max, res),
                         np.linspace(y_min, y_max, res))
    grid = np.c_[xx.ravel(), yy.ravel()]
    Z = predict_fn(grid).reshape(xx.shape)
    ax.contourf(xx, yy, Z, levels=[-0.5, 0.5, 1.5, 2.5], alpha=0.35, cmap=BG)
    ax.scatter(X2[:, 0], X2[:, 1], c=y_codes, cmap=FG, edgecolor="k", s=20)
    ax.set_xlabel("Bill length (mm)")
    ax.set_ylabel("Flipper length (mm)")
    ax.set_title(title)


def knn_grid(grid, k):
    """Wrap the scratch k-NN so it can label a whole grid of points."""
    return np.array([knn_predict(pt, X_train2, y_train_codes, k=k) for pt in grid])

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(11, 5))
plot_decision_boundary(lambda g: knn_grid(g, 1),  X_train2, y_train_codes, ax1,
                       "k = 1  (memorizes every point)")
plot_decision_boundary(lambda g: knn_grid(g, 15), X_train2, y_train_codes, ax2,
                       "k = 15  (smooth and general)")
plt.tight_layout()
plt.show()

🧠 Memorizing vs. generalizing

• k = 1 carves a jagged little island around every training point — including the odd penguin sitting in the "wrong" place. It has essentially memorized the training data, noise and all: perfect on what it's seen, brittle on anything new. That's overfitting.
• k = 15 is smooth. By polling many neighbors it shrugs off individual weird penguins and captures the general trend. It might miss a true outlier, but it will generalize better to new penguins.

This tension — memorizing the past vs. generalizing to the future — is one of the deepest ideas in all of machine learning. You'll meet it every single day.

🧪 Exercise 4 — The same thing in 3 lines

You just built k-NN by hand. In practice nobody does — scikit-learn ships a fast, battle-tested version. See how little code it takes.

• Create KNeighborsClassifier(n_neighbors=5), .fit(...) it on the same two features, and .score(...) it on the test set.
• Expected: almost exactly the accuracy your from-scratch version got. It's the same algorithm.

from sklearn.neighbors import KNeighborsClassifier
# ==================== YOUR CODE HERE ====================
### HINT: KNeighborsClassifier(n_neighbors=5), then .fit(X_train2, y_train_codes), then .score(...)
...
print(f"sklearn k-NN accuracy : {sklearn_acc:.3f}")
print(f"your scratch accuracy : {accuracy:.3f}")
assert abs(sklearn_acc - accuracy) < 0.05, "Scratch and sklearn k-NN should give nearly the same score."

Machine #2: the Decision Tree

k-NN never really explains itself. A decision tree does — it's a flowchart of yes/no questions the machine writes for itself: "Is the bill longer than 43 mm? If yes, go left; if no, go right..." Follow the branches to a species.

🧪 Exercise 5 — Grow a tree

• Create a DecisionTreeClassifier(max_depth=3, random_state=SEED), fit it on the two features, and score it.
• max_depth=3 keeps the tree shallow enough to actually read — we'll draw it next.

from sklearn.tree import DecisionTreeClassifier, plot_tree
# ==================== YOUR CODE HERE ====================
...
print(f"Decision tree accuracy: {tree_acc:.3f}")

plt.figure(figsize=(11, 6))
plot_tree(tree, feature_names=["bill_length", "flipper_length"],
          class_names=list(classes), filled=True, rounded=True, fontsize=9)
plt.title("The machine's if/else brain (max_depth = 3)")
plt.show()

📖 Read the tree out loud

Start at the top box and follow one path down, reading it as a sentence: "If the bill length is below this threshold, and the flipper is below that one, then I predict Adelie." Every box splits on one measurement; every leaf at the bottom is a final answer, colored by the species that dominates it.

This is why decision trees are prized when a prediction must be explained — a doctor or a bank can trace the exact chain of reasoning. The "who's standing next to me" logic of k-NN can't offer that.

Where does it go wrong? The confusion matrix

Accuracy is a single number, and a single number hides where mistakes happen. A confusion matrix lays out, for every true species, what the model actually guessed — so a mix-up between two look-alike species stands out from a wild miss.

🧪 Exercise 6 — Build the confusion matrix

• Call confusion_matrix(actual, predicted) on the test set, using the sklearn k-NN's predictions.
• We'll draw it as a heatmap. The diagonal is correct; everything off the diagonal is a mix-up.

from sklearn.metrics import confusion_matrix
y_pred_codes = knn_sklearn.predict(X_test2)   # the sklearn k-NN's guesses on the test set
# ==================== YOUR CODE HERE ====================
### HINT: confusion_matrix(actual, predicted) — rows are the true species, columns the guesses
...
plt.figure(figsize=(6, 5))
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues",
            xticklabels=classes, yticklabels=classes)
plt.xlabel("Predicted species")
plt.ylabel("Actual species")
plt.title("Confusion matrix — where the mistakes hide")
plt.show()
print(cm)

🔎 Reading the mistakes

The diagonal is bright — most penguins are classified correctly. The few off-diagonal cells are the interesting part: with only two features, the model's slip-ups cluster around Chinstrap.

That's not random. On these two measurements Chinstrap sits in the middle — its bill length is close to Gentoo's, and its flipper length is close to Adelie's — so a borderline penguin can drift into Chinstrap territory. Our hypothesis: hand the model more measurements (bill depth and body mass too) and those extra dimensions should pull the species further apart. Let's test that hunch.

The model tournament — and a trap

Time to use all four features and find the best k. We'll hold a tournament: train a k-NN for every k from 1 to 15, and see which scores highest on the test set.

🧪 Exercise 7 — Run the tournament (raw features)

• Complete the call to tournament(...) using the raw four-feature arrays X_train4 and X_test4.
• Watch closely: two features already scored about 0.98. Four features carry more information, so surely this does better... right?

X_train4 = X_train.to_numpy()
X_test4  = X_test.to_numpy()
k_values = list(range(1, 16))


def tournament(train_X, test_X):
    """Fit a k-NN for every k in k_values; return the list of test accuracies."""
    scores = []
    for k in k_values:
        m = KNeighborsClassifier(n_neighbors=k).fit(train_X, y_train_codes)
        scores.append(m.score(test_X, y_test_codes))
    return scores


# ==================== YOUR CODE HERE ====================
### HINT: run the tournament on the RAW four-feature arrays, X_train4 and X_test4
...
print(f"Best accuracy with 4 raw features: {max(raw_scores):.3f}")
print("(Remember: two features already scored about 0.98...)")

⚠️ Wait — more information made it worse?

That's the trap. k-NN decides who's "nearest" using distance, and distance sums up every feature's differences. Look at the raw scales:

• bill length differs between penguins by a few millimeters (values ≈ 40)
• body mass differs by hundreds or thousands of grams (values ≈ 4000)

So body_mass_g drowns out everything else — "nearest" collapses to "weighs about the same." And Adelie and Chinstrap weigh almost exactly the same, so the model can no longer tell them apart. It's like measuring the distance between two cities with one ruler in millimeters and another in kilometers, then adding the two numbers.

The fix: put every feature on the same scale. StandardScaler rescales each column to mean 0 with a comparable spread, so a millimeter of bill and a gram of mass each get a fair vote.

🧪 Exercise 8 — Scale, then rerun

• We've already fit a StandardScaler on the training features and transformed both sets for you.
• Run the same tournament(...) on the scaled arrays X_train4s and X_test4s, storing the result in scaled_scores.
• Expected: accuracy jumps above 0.95 — now clearly beating the two-feature model.

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().fit(X_train4)
X_train4s = scaler.transform(X_train4)
X_test4s  = scaler.transform(X_test4)

# ==================== YOUR CODE HERE ====================
### HINT: call tournament(...) again, but on the scaled arrays X_train4s and X_test4s
...
best_k = k_values[int(np.argmax(scaled_scores))]
print(f"Best accuracy with 4 SCALED features: {max(scaled_scores):.3f}  at k = {best_k}")
assert max(scaled_scores) > 0.9, "After scaling, accuracy should climb well above 0.90."

plt.figure(figsize=(8, 5))
plt.plot(k_values, raw_scores, marker="o", label="raw features")
plt.plot(k_values, scaled_scores, marker="s", label="scaled features")
plt.axvline(best_k, color="red", linestyle="--", label=f"best k = {best_k}")
plt.xlabel("k (number of neighbors)")
plt.ylabel("Test accuracy")
plt.title("Putting features on the same scale changes everything")
plt.legend()
plt.show()

Ship the champion

The tournament chose the best k honestly, on penguins the model never trained on. Now two professional moves give us the final, deployable model:

1. Bundle the scaler and the classifier into a single Pipeline, so raw measurements can go straight in and get scaled automatically — exactly what the Studio app needs.
2. Retrain on every penguin we have — now that k is locked in, more data makes a stronger model.

from sklearn.pipeline import make_pipeline

# One object that scales AND classifies. Feed it raw measurements (like the Studio's
# sliders) and it handles the scaling internally before predicting.
final_model = make_pipeline(
    StandardScaler(),
    KNeighborsClassifier(n_neighbors=best_k),
)
final_model.fit(X, y)   # X is a DataFrame (keeps column names); train on ALL penguins

honest_estimate = max(scaled_scores)   # accuracy measured earlier on held-out test penguins
print(f"Final model: StandardScaler + {best_k}-NN, trained on all {len(X)} penguins.")
print(f"Honest accuracy estimate (from the held-out test set): {honest_estimate:.3f}")
print(f"Species it can predict: {list(final_model.classes_)}")

🔓 Unlock your Studio module

You trained a real classifier — now ship it. The cell below saves your model into ai_studio/models/ in the exact shape the 🐧 Prediction Machine tab expects:

• model — your fitted scaler + classifier pipeline
• feature_names — the four columns, in order, so the app feeds inputs correctly
• target_names — the species names in the model's own class order, so the probability bars line up with the right label

(No blanks to fill here — just run it and watch the padlock pop open.)

import joblib

bundle = {
    "model": final_model,
    "feature_names": FEATURES,
    "target_names": list(final_model.classes_),   # SAME order as predict_proba's columns
}
out_path = MODELS_DIR / "prediction_machine.joblib"
joblib.dump(bundle, out_path)

REQUIRED_FILES = ["prediction_machine.joblib"]
missing = [f for f in REQUIRED_FILES if not (MODELS_DIR / f).exists()]
assert not missing, f"Something didn't save: {missing}"

print("🔓 UNLOCKED: Prediction Machine!")
print(f"   saved: {out_path}")
print("   Run your Studio and drag the sliders — your model predicts live:")
print("   python ai_studio/app.py   (from the repo root)")

🏁 Checkpoint

You built a real classifier — twice from scratch, then again with the pros' tools:

• ✅ Split data into train/test and learned why hiding the test set is the only honest measure
• ✅ Coded k-Nearest Neighbors yourself with NumPy
• ✅ Saw decision boundaries and the memorize-vs-generalize tradeoff (k = 1 vs k = 15)
• ✅ Grew a readable decision tree and traced its reasoning
• ✅ Read a confusion matrix to find which species get mixed up
• ✅ Sprang the feature-scaling trap — why more data can hurt, and how to fix it
• ✅ Unlocked 🐧 Prediction Machine — your first Studio tab is live!

Run python ai_studio/app.py, open the Prediction Machine tab, and drag the sliders. That's your model answering.

Tomorrow (Day 3): every penguin today came with its species already written down. But what can a machine learn when nobody labels the data — when you just hand it a pile of measurements and ask, "what groups are hiding in here?" That's unsupervised learning, and it unlocks your 🎨 Pattern Finder.