Day 04 · Overview

Week 1FNX 101 · 04

Inside the Brain

Build a neural network from scratch in pure NumPy — neurons, layers, gradient descent — and meet the matrix multiplication hiding underneath.

Today you will

Understand a single neuron and why one is never enough
Discover that a layer of neurons is really one matrix multiplication
Write the forward pass and a loss, then teach the network by the nudge method

Today's items

2 h 30 min

Start: Slides

Lab

FNX 101 · 04

day04_inside_the_brain.ipynb0/17 cells runPython 3

Day 4 — Inside the Brain 🧠

Mission briefing: The Chief Scientist drops by with a challenge and a grin: "Anyone can call a library. Today you build a neural network with your bare hands — just NumPy — and you'll train it to think. After this, the black box is never black again." No PyTorch. No shortcuts. Let's open the brain.

Previously in the Lab: yesterday you unlocked the 🎨 Pattern Finder and watched machines find structure with no labels. Today you build the thing everyone means when they say "AI": a neural network — from scratch.

Today you will:

Understand a single neuron and why one is never enough
Discover that a layer of neurons is really one matrix multiplication (remember this — it comes back on Day 10)
Write the forward pass, a loss, and teach the network to learn by the nudge method
Watch the decision boundary bend around the data as it trains
Peek at the loss landscape the network rolls down

Today you unlock: nothing to save — today you forge the engine that tomorrow's Studio module runs on. Day 5 hands you PyTorch and a GPU to turbocharge exactly what you build today.

A precise reading

The gradient points toward increasing loss; subtract it to descend. Finite differences approximate derivatives. Backpropagation uses one forward pass and one backward pass, not two forward passes. Matrix products are central, alongside biases, nonlinear activations and other operations. A sigmoid output between 0 and 1 is not automatically a calibrated probability.

# ╔═══════════════════════════════════════════════════════════════════╗
# ║ SETUP — run me first! (identical in every Lab notebook)           ║
# ╚═══════════════════════════════════════════════════════════════════╝
import os, sys, pathlib

IN_COLAB = "google.colab" in sys.modules
SMOKE = os.environ.get("SMOKE_TEST", "0") == "1"   # tiny fast run for automated tests

REPO_URL = "https://github.com/A-Halimi/ai-studio-internship.git"  # instructor: set once

if IN_COLAB:
    if not pathlib.Path("/content/ai-studio-internship").exists():
        !git clone -q {REPO_URL} /content/ai-studio-internship
    %cd /content/ai-studio-internship/notebooks/day04

HERE = pathlib.Path.cwd()
REPO = HERE.parents[1]                       # .../notebooks/dayNN -> repo root
DATA_DIR = pathlib.Path(os.environ.get("COURSE_DATA_DIR", REPO / "data"))
SAMPLES_DIR = REPO / "data_samples"          # small datasets shipped with the repo
MODELS_DIR = REPO / "ai_studio" / "models"   # where your Studio modules unlock
for p in (REPO / "data", MODELS_DIR):
    p.mkdir(parents=True, exist_ok=True)

import random
import numpy as np
SEED = 42
random.seed(SEED); np.random.seed(SEED)
print(f"✅ Setup done | Colab: {IN_COLAB} | Smoke test: {SMOKE}")
print(f"   data: {DATA_DIR}")

# ── CONFIG — the Lab's control panel. Tweak me later! ──────────────
EPOCHS        = 50 if SMOKE else 400     # how many full passes over the data
LEARNING_RATE = 0.5                      # how big a step each weight takes downhill
HIDDEN        = 16                       # neurons in the hidden layer
print(f"Training {EPOCHS} epochs | learning rate {LEARNING_RATE} | {HIDDEN} hidden neurons")

1 · The neuron: a bouncer at a door

A neuron is almost embarrassingly simple. Picture a bouncer deciding whether to open a door.

Each reason to open (an input) has a weight — how much that reason matters. The bouncer multiplies each input by its weight, adds them all up, tosses in a personal bias (a good or bad mood), and if the total clears the bar, the door opens.

In math that's just:

output = activation( input₁·w₁ + input₂·w₂ + … + bias )

The weights and bias are the knobs we'll tune. The activation is the "does it clear the bar?" squashing step. Stack a few of these bouncers into layers and you get a network like this one — 2 inputs, a hidden layer of 16, and 1 output:

import numpy as np
import matplotlib.pyplot as plt

# A quick diagram of our network (just for intuition).
fig, ax = plt.subplots(figsize=(8, 5))
layers = [2, 16, 1]
xs = [0, 1, 2]
pos = {}
for li, (x, n) in enumerate(zip(xs, layers)):
    for ni, y in enumerate(np.linspace(0.05, 0.95, n)):
        pos[(li, ni)] = (x, y)
        ax.add_patch(plt.Circle((x, y), 0.02, color="#1f77b4", zorder=3))
for li in range(len(layers) - 1):
    for a in range(layers[li]):
        for b in range(layers[li + 1]):
            x0, y0 = pos[(li, a)]
            x1, y1 = pos[(li + 1, b)]
            ax.plot([x0, x1], [y0, y1], color="gray", lw=0.3, alpha=0.5, zorder=1)
for x, name in zip(xs, ["2 inputs", "16 hidden neurons", "1 output"]):
    ax.text(x, 1.03, name, ha="center", fontsize=11)
ax.set_xlim(-0.3, 2.3)
ax.set_ylim(0, 1.12)
ax.axis("off")
ax.set_title("our network: information flows left → right")
plt.show()

2 · The challenge data: two interlocking moons

To make this real we need data that's hard — something no single straight line can split. make_moons gives us two interleaving crescents: think of them as two teams tangled together. We also standardize the inputs (mean 0, spread 1), which keeps the sigmoids happy and helps the network learn faster.

from sklearn.datasets import make_moons

X_raw, y_raw = make_moons(n_samples=300, noise=0.20, random_state=SEED)
X = (X_raw - X_raw.mean(axis=0)) / X_raw.std(axis=0)   # standardize
y = y_raw.reshape(-1, 1).astype(float)                 # shape (300, 1)

plt.figure(figsize=(6, 5))
plt.scatter(X[:, 0], X[:, 1], c=y.ravel(), cmap="RdBu", edgecolor="k", s=25)
plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.title("two interlocking teams — no straight line separates them")
plt.show()

3 · The squashing step: sigmoid

Our bouncer needs to turn a "sum of reasons" (any number, positive or negative, big or small) into a clean answer between 0 and 1 — a confidence. The sigmoid function does exactly that: huge positive → near 1, huge negative → near 0, and zero → exactly ½.

sigmoid(z) = 1 / (1 + e^(−z))

🧪 Exercise 1 — Implement the sigmoid

Fill in sigmoid(z) using np.exp.
It should work on a whole array at once (NumPy handles that for free).

Sanity check: sigmoid(0) must equal 0.5, and the plot should be a smooth S-curve from 0 to 1.

Exercise

def sigmoid(z):
    """Squash any number into the range (0, 1)."""
    # ==================== YOUR CODE HERE ====================
    ### HINT: the formula is 1 / (1 + np.exp(-z))
    ...


zs = np.linspace(-10, 10, 200)
plt.figure(figsize=(6, 3.5))
plt.plot(zs, sigmoid(zs))
plt.axhline(0.5, color="gray", ls="--", lw=1)
plt.xlabel("z  (sum of weighted inputs)")
plt.ylabel("sigmoid(z)  =  confidence")
plt.title("sigmoid: any number in → a 0-to-1 confidence out")
plt.show()

assert abs(sigmoid(0) - 0.5) < 1e-9, "sigmoid(0) should be exactly 0.5"
print("sigmoid(0) =", sigmoid(0))

4 · One neuron tries… and fails

Let's see how far a single neuron gets on the moons. One neuron can only draw a single straight cut across the plane — so it's doomed here, but the failure is the whole point. The helper below trains one neuron for us (that's just logistic regression, in pure NumPy) and draws its decision boundary. We'll reuse this plot_boundary picture all day long.

def plot_boundary(predict_fn, X, y, ax=None, title=""):
    """Shade the plane by the model's prediction, then scatter the real points."""
    if ax is None:
        _, ax = plt.subplots(figsize=(5, 4))
    pad = 0.5
    xx, yy = np.meshgrid(
        np.linspace(X[:, 0].min() - pad, X[:, 0].max() + pad, 200),
        np.linspace(X[:, 1].min() - pad, X[:, 1].max() + pad, 200),
    )
    grid = np.c_[xx.ravel(), yy.ravel()]
    zz = predict_fn(grid).reshape(xx.shape)
    ax.contourf(xx, yy, zz, levels=20, cmap="RdBu", alpha=0.7)
    ax.scatter(X[:, 0], X[:, 1], c=y.ravel(), cmap="RdBu", edgecolor="k", s=18)
    ax.set_xlabel("feature 1")
    ax.set_ylabel("feature 2")
    ax.set_title(title)


# Train a single neuron with a tiny gradient loop (pure NumPy).
rng = np.random.default_rng(SEED)
w = 0.1 * rng.standard_normal((2, 1))
b = 0.0
for _ in range(1500):
    p = 1 / (1 + np.exp(-(X @ w + b)))
    error = p - y                       # (300, 1)
    w -= 1.0 * (X.T @ error) / len(X)
    b -= 1.0 * float(error.mean())

plot_boundary(lambda g: 1 / (1 + np.exp(-(g @ w + b))), X, y,
              title="one neuron = one straight cut (it can't win)")
plt.show()

See the problem? One straight line, two tangled moons — it always misclassifies a chunk. A single neuron draws a single cut. To bend the boundary we need many neurons working together.

5 · ⭐ The big secret: a neural network is matrix multiplication

Here's the idea this entire course eventually rests on.

Our hidden layer has 16 neurons, and each one looks at all 300 data points. That sounds like a nested loop — 16 × 300 weighted sums. But line the weights up in a grid and it collapses into a single matrix multiply:

(300 × 2) data @ (2 × 16) weights → (300 × 16) hidden values

One @. That's the whole layer. Stack layers and a neural network is, underneath, just a chain of matrix multiplications with a squash in between. Training a network means doing zillions of these multiplies.

Which raises the question an entire field — high-performance computing — is built on: how fast can you multiply matrices? Let's feel the difference right now.

🧪 Exercise 2 — Matrix multiply, by hand vs NumPy

Fill in matmul_by_hand(A, B) with the classic triple loop (rows × columns × the shared dimension).
We check it against NumPy's @, then time both on 200×200 matrices.

Expect the exact same answer — and NumPy winning by hundreds of times.

Exercise

import time


def matmul_by_hand(A, B):
    """Multiply A (n×m) by B (m×p) with explicit loops — the definition of matmul."""
    n, m = A.shape
    m2, p = B.shape
    assert m == m2, "inner dimensions must match"
    C = np.zeros((n, p))
    # ==================== YOUR CODE HERE ====================
    ### HINT: three nested loops — i over rows of A, j over cols of B, k over the shared dim
    ...
    return C


A = np.random.default_rng(0).standard_normal((200, 200))
B = np.random.default_rng(1).standard_normal((200, 200))

t0 = time.perf_counter(); C_hand = matmul_by_hand(A, B); t_hand = time.perf_counter() - t0
t0 = time.perf_counter(); C_np = A @ B;                   t_np = time.perf_counter() - t0

assert np.allclose(C_hand, C_np), "your matmul doesn't match NumPy — check the loops!"
print(f"by hand : {t_hand:8.3f} s")
print(f"NumPy @ : {t_np:8.5f} s")
print(f"NumPy is about {t_hand / t_np:.0f}x faster — same math, same answer.")

Write that number down. That gap — the same multiplication, hundreds of times faster — is not a NumPy trick you can forget. It's the reason GPUs exist, the reason training big models is even possible, and the entire subject of high-performance computing. Same math. Wildly different speed. Everything a network does, it does by multiplying matrices — so making that fast makes everything fast.

6 · The forward pass: information flows through

Now we assemble the network. Two layers, each a matrix multiply followed by a sigmoid squash:

hidden values: H = sigmoid(X @ W1 + b1) → shape (300, 16)
output: ŷ = sigmoid(H @ W2 + b2) → shape (300, 1)

That's the forward pass — inputs in, prediction out.

🧪 Exercise 3 — Write the forward pass

Fill in forward so it chains the two layers exactly as above.
Then we initialize small random weights and run it once. Untrained, it should output a (300, 1) array of mush near 0.5 — it hasn't learned anything yet.

Exercise

def forward(X, W1, b1, W2, b2):
    """Run inputs through the 2-layer network. Returns predictions in (0, 1)."""
    # ==================== YOUR CODE HERE ====================
    ### HINT: layer 1 -> H = sigmoid(X @ W1 + b1);  layer 2 -> sigmoid(H @ W2 + b2)
    ...


def init_weights(hidden=HIDDEN, seed=SEED):
    """Small random starting weights — the knobs we'll tune. Returns [W1, b1, W2, b2]."""
    r = np.random.default_rng(seed)
    W1 = 0.5 * r.standard_normal((2, hidden))
    b1 = np.zeros((1, hidden))
    W2 = 0.5 * r.standard_normal((hidden, 1))
    b2 = np.zeros((1, 1))
    return [W1, b1, W2, b2]


preds = forward(X, *init_weights())
print("prediction shape:", preds.shape)
print("a few untrained predictions:", np.round(preds[:4, 0], 3))
assert preds.shape == (300, 1), "forward should return one prediction per point"

7 · Loss: how wrong are we?

To improve, the network needs a single number saying how bad its guesses are. We'll use mean squared error: the average of (prediction − truth)². Zero means perfect; bigger means worse.

🧪 Exercise 4 — Implement the loss

Fill in mse(y_pred, y_true) = the mean of the squared differences.
Print the loss of the untrained network. Guessing ~0.5 everywhere gives a loss near 0.25.

Exercise

def mse(y_pred, y_true):
    """Mean squared error — the average squared gap between guess and truth."""
    # ==================== YOUR CODE HERE ====================
    ### HINT: np.mean((y_pred - y_true) ** 2)
    ...


start_loss = mse(forward(X, *init_weights()), y)
print(f"untrained loss: {start_loss:.4f}")

8 · Learning without calculus: the nudge method

Here's the beautiful part. The network has 65 knobs (all the weights and biases). We want to know, for each one: should I turn it up or down to lower the loss?

The simplest possible answer: try it. Nudge a knob up a hair, check the loss. Nudge it down a hair, check again. Whichever direction lowers the loss, step that way a little. Do it for every knob, over and over. That slope-from-nudging is exactly what a gradient is.

The helper below computes that nudge for every weight. You'll write the one line that actually takes the step.

def numerical_gradient(loss_fn, params, eps=1e-4):
    """The nudge method: estimate the gradient of loss_fn for every weight.

    For each weight w we measure loss(w + eps) and loss(w - eps); their
    difference over 2*eps is the slope (gradient) of the loss along that weight.
    Two forward passes per weight — remember that number for the Math Corner.
    """
    grads = []
    for p in params:
        g = np.zeros_like(p)
        it = np.nditer(p, flags=["multi_index"])
        while not it.finished:
            idx = it.multi_index
            original = p[idx]
            p[idx] = original + eps          # nudge up
            loss_up = loss_fn()
            p[idx] = original - eps          # nudge down
            loss_down = loss_fn()
            p[idx] = original                # put it back exactly
            g[idx] = (loss_up - loss_down) / (2 * eps)
            it.iternext()
        grads.append(g)
    return grads

🧪 Exercise 5 — Take the step (this is learning)

Everything below is given except the heart of it. Once per epoch we compute the nudge for every weight; you write the update that moves each weight downhill:

For each weight array p and its gradient g, subtract lr * g from p, in place (p -= …).

Run it and watch the loss fall epoch by epoch as the network figures out the moons.

(Cost check: 65 weights × 2 forward passes × 400 epochs ≈ 52,000 tiny forward passes on a 300-point network — a handful of seconds.)

Exercise

def train_network(hidden=HIDDEN, epochs=EPOCHS, lr=LEARNING_RATE, record_every=50):
    """Train with the nudge method. Returns (params, history)."""
    params = init_weights(hidden)
    loss_fn = lambda: mse(forward(X, *params), y)

    history = {"epoch": [], "loss": [], "snapshots": []}
    for epoch in range(epochs):
        grads = numerical_gradient(loss_fn, params)
        # ↓ the one line that makes learning happen: step every weight downhill
        # ==================== YOUR CODE HERE ====================
        ### HINT: in place, so use  p -= lr * g   (not  p = p - lr*g)
        ...
        if epoch % record_every == 0 or epoch == epochs - 1:
            L = loss_fn()
            history["epoch"].append(epoch)
            history["loss"].append(L)
            history["snapshots"].append([p.copy() for p in params])
            print(f"epoch {epoch:4d}   loss {L:.4f}")
    return params, history


trained_params, history = train_network()

9 · Watch it learn

We saved a snapshot of the network every 50 epochs. Let's replay them: watch the boundary start as a vague blob and bend itself around the two moons.

snaps = history["snapshots"]
eps_rec = history["epoch"]
n = len(snaps)
cols = 3
rows = int(np.ceil(n / cols))
fig, axes = plt.subplots(rows, cols, figsize=(11, 3.4 * rows))
axes = np.atleast_1d(axes).ravel()
for ax, ep, snap in zip(axes, eps_rec, snaps):
    plot_boundary(lambda g, s=snap: forward(g, *s), X, y, ax=ax, title=f"epoch {ep}")
for ax in axes[n:]:
    ax.axis("off")
fig.suptitle("the decision boundary learning to bend around the moons")
fig.tight_layout()
plt.show()

plt.figure(figsize=(6, 4))
plt.plot(history["epoch"], history["loss"], "o-")
plt.xlabel("epoch")
plt.ylabel("MSE loss")
plt.title("the loss falling as the network learns")
plt.show()

🧪 Exercise 6 — Score it

Loss is abstract; accuracy is human. Turn the network's confidences into 0/1 decisions and measure the fraction it gets right.

In accuracy, threshold the predictions at 0.5, then compare to the truth and take the mean.
Our hand-built net should land around 88% here — a giant leap from the single neuron's straight-line failure. (The ceiling on this noisy data is ~97%; tomorrow's power tools close most of that gap. Our simple "nudge every weight" trainer is honest but slow.)

Exercise

def accuracy(y_pred, y_true):
    """Fraction correct, calling anything above 0.5 a '1'."""
    # ==================== YOUR CODE HERE ====================
    ### HINT: (y_pred > 0.5) turns confidences into 0/1; compare with == and take the mean
    ...


final_acc = accuracy(forward(X, *trained_params), y)
print(f"final accuracy: {final_acc:.1%}")
if not SMOKE:
    assert final_acc > 0.85, "accuracy is low — check forward(), the loss, or the update step"

10 · Experiments: how many neurons do you need?

The hidden layer's size controls how wiggly a boundary the network can draw. Too few neurons and it can't bend enough; too many and it can over-wiggle (memorizing noise instead of the shape). Let's see both extremes.

🧪 Exercise 7 — Retrain with 2 and 64 hidden neurons

Inside the loop, train a network with hidden=h and plot its boundary.
Then compare the shapes you get.

Exercise

fig, axes = plt.subplots(1, 2, figsize=(11, 4.5))
for ax, h in zip(axes, [2, 64]):
    # ==================== YOUR CODE HERE ====================
    ### HINT: params, _ = train_network(hidden=h);  then plot_boundary(lambda g,s=params: forward(g,*s), X, y, ax=ax, ...)
    ...
fig.tight_layout()
plt.show()

Describe what you see (double-click to edit):

With 2 neurons, the boundary is ...
With 64 neurons, the boundary is ...

Fewer neurons → simpler boundary. More neurons → more flexible (and, past a point, prone to memorizing noise). Choosing the right size is part of the art.

And one knob you should not crank: the learning rate. A sensible step (our 0.5) walks steadily downhill. But make the step wildly too big and the weights get slammed into saturation — the network collapses to guessing one class and the loss flatlines. Watch it break.

import warnings

with warnings.catch_warnings():
    warnings.simplefilter("ignore")            # giant steps overflow the sigmoids — that's the point
    chaos_params, chaos = train_network(lr=100.0)

broken_acc = accuracy(forward(X, *chaos_params), y)
print(f"accuracy with a giant learning rate: {broken_acc:.1%}  (≈ a coin flip — it never learned)")

plt.figure(figsize=(6, 4))
plt.plot(chaos["epoch"], chaos["loss"], "o-", color="crimson")
plt.xlabel("epoch")
plt.ylabel("MSE loss")
plt.title("learning rate 100 — too big: the loss flatlines, the network never learns")
plt.show()

11 · The loss landscape: what "downhill" means

Every setting of the weights produces some loss. Picture that as a landscape — the weights are your map coordinates, the loss is the altitude. Learning is just walking downhill on this surface, and the gradient is the compass pointing down.

We can't draw a 65-dimensional landscape, but we can take a 2-weight slice: freeze every weight at its trained value except two, sweep those two across a grid, and plot the loss as height. The red star marks where training actually landed.

i_a, i_b = 0, 2                       # vary W1[0,0] and W2[0,0]
a_star = trained_params[i_a][0, 0]
b_star = trained_params[i_b][0, 0]
span = 3.0
a_vals = np.linspace(a_star - span, a_star + span, 25)
b_vals = np.linspace(b_star - span, b_star + span, 25)

Z = np.zeros((25, 25))
for r, bv in enumerate(b_vals):
    for c, av in enumerate(a_vals):
        trial = [p.copy() for p in trained_params]
        trial[i_a][0, 0] = av
        trial[i_b][0, 0] = bv
        Z[r, c] = mse(forward(X, *trial), y)

AA, BB = np.meshgrid(a_vals, b_vals)
plt.figure(figsize=(7, 5))
cs = plt.contourf(AA, BB, Z, levels=30, cmap="viridis")
plt.colorbar(cs, label="loss")
plt.scatter([a_star], [b_star], color="red", marker="*", s=280,
            edgecolor="white", linewidth=1.2, label="where training landed", zorder=5)
plt.xlabel("weight W1[0, 0]")
plt.ylabel("weight W2[0, 0]")
plt.title("a 2-weight slice of the loss landscape")
plt.legend()
plt.show()

12 · Math Corner (optional)

🔢 Math Corner — the nudge method vs. real backprop (optional)

A derivative is just a slope: nudge the input a tiny bit, see how much the output moves, divide by the nudge. That's literally what numerical_gradient does — one pair of nudges per weight.

It works, but it's slow. Our tiny network has 2·16 + 16 + 16 + 1 = 65 weights. The nudge method needs two forward passes per weight to estimate its slope — that's 130 forward passes for a single training step. A network with a million weights would need two million forward passes per step. Hopeless.

Real networks use backpropagation: with the chain rule from calculus, it computes the exact slope for every weight in just two passes total (one forward, one backward) — no matter how many weights there are. 130 versus 2. That efficiency is the only reason training large networks is even possible.

Tomorrow, PyTorch's autograd runs this backward pass for you automatically — and on a GPU, where (remember Section 5) those matrix multiplies fly.

🏁 Checkpoint

Today, with nothing but NumPy, you:

built a neuron and saw why one straight cut isn't enough
learned the course's biggest secret — a network is chained matrix multiplications (and clocked NumPy beating a hand-written loop by hundreds of times)
wrote the forward pass and a loss, then trained the net with the nudge method
watched the decision boundary bend around the moons and read the loss landscape it descended

You forged the engine by hand. Tomorrow the Lab hands you power tools — PyTorch, automatic gradients, and a GPU — and you'll teach a network to read handwritten digits, the very same ones you mapped with t-SNE yesterday.

Next: Day 5 — The Digit Reader ⚡