{ "cells": [ { "cell_type": "markdown", "id": "41ae32cd", "metadata": {}, "source": [ "# Day 4 β€” Inside the Brain 🧠\n", "\n", "> **Mission briefing:** The Chief Scientist drops by with a challenge and a grin: *\"Anyone can call a library. Today you build a neural network with your bare hands β€” just NumPy β€” and you'll train it to think. After this, the black box is never black again.\"* No PyTorch. No shortcuts. Let's open the brain.\n", "\n", "**Previously in the Lab:** yesterday you unlocked the 🎨 **Pattern Finder** and watched machines find structure with no labels. Today you build the thing everyone *means* when they say \"AI\": a **neural network** β€” from scratch.\n", "\n", "**Today you will:**\n", "- Understand a single **neuron** and why one is never enough\n", "- Discover that a layer of neurons is really **one matrix multiplication** (remember this β€” it comes back on Day 10)\n", "- Write the **forward pass**, a **loss**, and teach the network to learn by the **nudge method**\n", "- *Watch* the decision boundary bend around the data as it trains\n", "- Peek at the **loss landscape** the network rolls down\n", "\n", "**Today you unlock:** nothing to save β€” today you **forge the engine** that tomorrow's Studio module runs on. Day 5 hands you PyTorch and a GPU to turbocharge exactly what you build today.\n", "\n", "
A precise reading

The gradient points toward increasing loss; subtract it to descend. Finite differences approximate derivatives. Backpropagation uses one forward pass and one backward pass, not two forward passes. Matrix products are central, alongside biases, nonlinear activations and other operations. A sigmoid output between 0 and 1 is not automatically a calibrated probability.

\n" ] }, { "cell_type": "code", "execution_count": null, "id": "8c7f2191", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:29.077099Z", "iopub.status.busy": "2026-08-17T12:54:29.076940Z", "iopub.status.idle": "2026-08-17T12:54:29.188894Z", "shell.execute_reply": "2026-08-17T12:54:29.187718Z" } }, "outputs": [], "source": [ "# ╔═══════════════════════════════════════════════════════════════════╗\n", "# β•‘ SETUP β€” run me first! (identical in every Lab notebook) β•‘\n", "# β•šβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•\n", "import os, sys, pathlib\n", "\n", "IN_COLAB = \"google.colab\" in sys.modules\n", "SMOKE = os.environ.get(\"SMOKE_TEST\", \"0\") == \"1\" # tiny fast run for automated tests\n", "\n", "REPO_URL = \"https://github.com/A-Halimi/ai-studio-internship.git\" # instructor: set once\n", "\n", "if IN_COLAB:\n", " if not pathlib.Path(\"/content/ai-studio-internship\").exists():\n", " !git clone -q {REPO_URL} /content/ai-studio-internship\n", " %cd /content/ai-studio-internship/notebooks/day04\n", "\n", "HERE = pathlib.Path.cwd()\n", "REPO = HERE.parents[1] # .../notebooks/dayNN -> repo root\n", "DATA_DIR = pathlib.Path(os.environ.get(\"COURSE_DATA_DIR\", REPO / \"data\"))\n", "SAMPLES_DIR = REPO / \"data_samples\" # small datasets shipped with the repo\n", "MODELS_DIR = REPO / \"ai_studio\" / \"models\" # where your Studio modules unlock\n", "for p in (REPO / \"data\", MODELS_DIR):\n", " p.mkdir(parents=True, exist_ok=True)\n", "\n", "import random\n", "import numpy as np\n", "SEED = 42\n", "random.seed(SEED); np.random.seed(SEED)\n", "print(f\"βœ… Setup done | Colab: {IN_COLAB} | Smoke test: {SMOKE}\")\n", "print(f\" data: {DATA_DIR}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "47fb8300", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:29.190503Z", "iopub.status.busy": "2026-08-17T12:54:29.190300Z", "iopub.status.idle": "2026-08-17T12:54:29.194318Z", "shell.execute_reply": "2026-08-17T12:54:29.193353Z" } }, "outputs": [], "source": [ "# ── CONFIG β€” the Lab's control panel. Tweak me later! ──────────────\n", "EPOCHS = 50 if SMOKE else 400 # how many full passes over the data\n", "LEARNING_RATE = 0.5 # how big a step each weight takes downhill\n", "HIDDEN = 16 # neurons in the hidden layer\n", "print(f\"Training {EPOCHS} epochs | learning rate {LEARNING_RATE} | {HIDDEN} hidden neurons\")" ] }, { "cell_type": "markdown", "id": "15928ad1", "metadata": {}, "source": [ "## 1 Β· The neuron: a bouncer at a door\n", "\n", "A neuron is almost embarrassingly simple. Picture a bouncer deciding whether to open a door.\n", "\n", "Each reason to open (an **input**) has a **weight** β€” how much that reason matters. The bouncer multiplies each input by its weight, adds them all up, tosses in a personal **bias** (a good or bad mood), and if the total clears the bar, the door opens.\n", "\n", "In math that's just:\n", "\n", "`output = activation( input₁·w₁ + inputβ‚‚Β·wβ‚‚ + … + bias )`\n", "\n", "The **weights** and **bias** are the knobs we'll tune. The **activation** is the \"does it clear the bar?\" squashing step. Stack a few of these bouncers into layers and you get a network like this one β€” 2 inputs, a hidden layer of 16, and 1 output:" ] }, { "cell_type": "code", "execution_count": null, "id": "16c2528b", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:29.196048Z", "iopub.status.busy": "2026-08-17T12:54:29.195890Z", "iopub.status.idle": "2026-08-17T12:54:29.534015Z", "shell.execute_reply": "2026-08-17T12:54:29.532863Z" } }, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "# A quick diagram of our network (just for intuition).\n", "fig, ax = plt.subplots(figsize=(8, 5))\n", "layers = [2, 16, 1]\n", "xs = [0, 1, 2]\n", "pos = {}\n", "for li, (x, n) in enumerate(zip(xs, layers)):\n", " for ni, y in enumerate(np.linspace(0.05, 0.95, n)):\n", " pos[(li, ni)] = (x, y)\n", " ax.add_patch(plt.Circle((x, y), 0.02, color=\"#1f77b4\", zorder=3))\n", "for li in range(len(layers) - 1):\n", " for a in range(layers[li]):\n", " for b in range(layers[li + 1]):\n", " x0, y0 = pos[(li, a)]\n", " x1, y1 = pos[(li + 1, b)]\n", " ax.plot([x0, x1], [y0, y1], color=\"gray\", lw=0.3, alpha=0.5, zorder=1)\n", "for x, name in zip(xs, [\"2 inputs\", \"16 hidden neurons\", \"1 output\"]):\n", " ax.text(x, 1.03, name, ha=\"center\", fontsize=11)\n", "ax.set_xlim(-0.3, 2.3)\n", "ax.set_ylim(0, 1.12)\n", "ax.axis(\"off\")\n", "ax.set_title(\"our network: information flows left β†’ right\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "e694c215", "metadata": {}, "source": [ "## 2 Β· The challenge data: two interlocking moons\n", "\n", "To make this real we need data that's *hard* β€” something no single straight line can split. `make_moons` gives us two interleaving crescents: think of them as two teams tangled together. We also **standardize** the inputs (mean 0, spread 1), which keeps the sigmoids happy and helps the network learn faster." ] }, { "cell_type": "code", "execution_count": null, "id": "19ece337", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:29.536022Z", "iopub.status.busy": "2026-08-17T12:54:29.535821Z", "iopub.status.idle": "2026-08-17T12:54:30.735726Z", "shell.execute_reply": "2026-08-17T12:54:30.734601Z" } }, "outputs": [], "source": [ "from sklearn.datasets import make_moons\n", "\n", "X_raw, y_raw = make_moons(n_samples=300, noise=0.20, random_state=SEED)\n", "X = (X_raw - X_raw.mean(axis=0)) / X_raw.std(axis=0) # standardize\n", "y = y_raw.reshape(-1, 1).astype(float) # shape (300, 1)\n", "\n", "plt.figure(figsize=(6, 5))\n", "plt.scatter(X[:, 0], X[:, 1], c=y.ravel(), cmap=\"RdBu\", edgecolor=\"k\", s=25)\n", "plt.xlabel(\"feature 1\")\n", "plt.ylabel(\"feature 2\")\n", "plt.title(\"two interlocking teams β€” no straight line separates them\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "a2282054", "metadata": {}, "source": [ "## 3 Β· The squashing step: sigmoid\n", "\n", "Our bouncer needs to turn a \"sum of reasons\" (any number, positive or negative, big or small) into a clean answer between 0 and 1 β€” a **confidence**. The **sigmoid** function does exactly that: huge positive β†’ near 1, huge negative β†’ near 0, and zero β†’ exactly Β½.\n", "\n", "`sigmoid(z) = 1 / (1 + e^(βˆ’z))`\n", "\n", "### πŸ§ͺ Exercise 1 β€” Implement the sigmoid\n", "\n", "- Fill in `sigmoid(z)` using `np.exp`.\n", "- It should work on a whole array at once (NumPy handles that for free).\n", "\n", "Sanity check: `sigmoid(0)` must equal `0.5`, and the plot should be a smooth S-curve from 0 to 1." ] }, { "cell_type": "code", "execution_count": null, "id": "dbfc5e2b", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:30.737812Z", "iopub.status.busy": "2026-08-17T12:54:30.737534Z", "iopub.status.idle": "2026-08-17T12:54:30.749129Z", "shell.execute_reply": "2026-08-17T12:54:30.748147Z" } }, "outputs": [], "source": [ "def sigmoid(z):\n", " \"\"\"Squash any number into the range (0, 1).\"\"\"\n", " # ==================== YOUR CODE HERE ====================\n", " ### HINT: the formula is 1 / (1 + np.exp(-z))\n", " ...\n", "\n", "\n", "zs = np.linspace(-10, 10, 200)\n", "plt.figure(figsize=(6, 3.5))\n", "plt.plot(zs, sigmoid(zs))\n", "plt.axhline(0.5, color=\"gray\", ls=\"--\", lw=1)\n", "plt.xlabel(\"z (sum of weighted inputs)\")\n", "plt.ylabel(\"sigmoid(z) = confidence\")\n", "plt.title(\"sigmoid: any number in β†’ a 0-to-1 confidence out\")\n", "plt.show()\n", "\n", "assert abs(sigmoid(0) - 0.5) < 1e-9, \"sigmoid(0) should be exactly 0.5\"\n", "print(\"sigmoid(0) =\", sigmoid(0))" ] }, { "cell_type": "markdown", "id": "7704e13a", "metadata": {}, "source": [ "## 4 Β· One neuron tries… and fails\n", "\n", "Let's see how far a *single* neuron gets on the moons. One neuron can only draw a single straight cut across the plane β€” so it's doomed here, but the failure is the whole point. The helper below trains one neuron for us (that's just logistic regression, in pure NumPy) and draws its decision boundary. We'll reuse this `plot_boundary` picture all day long." ] }, { "cell_type": "code", "execution_count": null, "id": "42693b29", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:30.750741Z", "iopub.status.busy": "2026-08-17T12:54:30.750574Z", "iopub.status.idle": "2026-08-17T12:54:30.802693Z", "shell.execute_reply": "2026-08-17T12:54:30.801540Z" } }, "outputs": [], "source": [ "def plot_boundary(predict_fn, X, y, ax=None, title=\"\"):\n", " \"\"\"Shade the plane by the model's prediction, then scatter the real points.\"\"\"\n", " if ax is None:\n", " _, ax = plt.subplots(figsize=(5, 4))\n", " pad = 0.5\n", " xx, yy = np.meshgrid(\n", " np.linspace(X[:, 0].min() - pad, X[:, 0].max() + pad, 200),\n", " np.linspace(X[:, 1].min() - pad, X[:, 1].max() + pad, 200),\n", " )\n", " grid = np.c_[xx.ravel(), yy.ravel()]\n", " zz = predict_fn(grid).reshape(xx.shape)\n", " ax.contourf(xx, yy, zz, levels=20, cmap=\"RdBu\", alpha=0.7)\n", " ax.scatter(X[:, 0], X[:, 1], c=y.ravel(), cmap=\"RdBu\", edgecolor=\"k\", s=18)\n", " ax.set_xlabel(\"feature 1\")\n", " ax.set_ylabel(\"feature 2\")\n", " ax.set_title(title)\n", "\n", "\n", "# Train a single neuron with a tiny gradient loop (pure NumPy).\n", "rng = np.random.default_rng(SEED)\n", "w = 0.1 * rng.standard_normal((2, 1))\n", "b = 0.0\n", "for _ in range(1500):\n", " p = 1 / (1 + np.exp(-(X @ w + b)))\n", " error = p - y # (300, 1)\n", " w -= 1.0 * (X.T @ error) / len(X)\n", " b -= 1.0 * float(error.mean())\n", "\n", "plot_boundary(lambda g: 1 / (1 + np.exp(-(g @ w + b))), X, y,\n", " title=\"one neuron = one straight cut (it can't win)\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "e7a6f1f7", "metadata": {}, "source": [ "See the problem? One straight line, two tangled moons β€” it always misclassifies a chunk. A single neuron draws a single cut. To *bend* the boundary we need **many neurons working together.**" ] }, { "cell_type": "markdown", "id": "351404f4", "metadata": {}, "source": [ "## 5 Β· ⭐ The big secret: a neural network is matrix multiplication\n", "\n", "Here's the idea this entire course eventually rests on.\n", "\n", "Our hidden layer has 16 neurons, and each one looks at all 300 data points. That sounds like a nested loop β€” 16 Γ— 300 weighted sums. But line the weights up in a grid and it collapses into **a single matrix multiply**:\n", "\n", "`(300 Γ— 2) data @ (2 Γ— 16) weights β†’ (300 Γ— 16) hidden values`\n", "\n", "One `@`. That's the whole layer. Stack layers and a neural network is, underneath, **just a chain of matrix multiplications** with a squash in between. Training a network means doing *zillions* of these multiplies.\n", "\n", "Which raises the question an entire field β€” high-performance computing β€” is built on: **how fast can you multiply matrices?** Let's feel the difference right now.\n", "\n", "### πŸ§ͺ Exercise 2 β€” Matrix multiply, by hand vs NumPy\n", "\n", "- Fill in `matmul_by_hand(A, B)` with the classic **triple loop** (rows Γ— columns Γ— the shared dimension).\n", "- We check it against NumPy's `@`, then time both on 200Γ—200 matrices.\n", "\n", "Expect the *exact same answer* β€” and NumPy winning by **hundreds of times**." ] }, { "cell_type": "code", "execution_count": null, "id": "50ab7781", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:30.804560Z", "iopub.status.busy": "2026-08-17T12:54:30.804389Z", "iopub.status.idle": "2026-08-17T12:54:32.835132Z", "shell.execute_reply": "2026-08-17T12:54:32.833609Z" } }, "outputs": [], "source": [ "import time\n", "\n", "\n", "def matmul_by_hand(A, B):\n", " \"\"\"Multiply A (nΓ—m) by B (mΓ—p) with explicit loops β€” the definition of matmul.\"\"\"\n", " n, m = A.shape\n", " m2, p = B.shape\n", " assert m == m2, \"inner dimensions must match\"\n", " C = np.zeros((n, p))\n", " # ==================== YOUR CODE HERE ====================\n", " ### HINT: three nested loops β€” i over rows of A, j over cols of B, k over the shared dim\n", " ...\n", " return C\n", "\n", "\n", "A = np.random.default_rng(0).standard_normal((200, 200))\n", "B = np.random.default_rng(1).standard_normal((200, 200))\n", "\n", "t0 = time.perf_counter(); C_hand = matmul_by_hand(A, B); t_hand = time.perf_counter() - t0\n", "t0 = time.perf_counter(); C_np = A @ B; t_np = time.perf_counter() - t0\n", "\n", "assert np.allclose(C_hand, C_np), \"your matmul doesn't match NumPy β€” check the loops!\"\n", "print(f\"by hand : {t_hand:8.3f} s\")\n", "print(f\"NumPy @ : {t_np:8.5f} s\")\n", "print(f\"NumPy is about {t_hand / t_np:.0f}x faster β€” same math, same answer.\")" ] }, { "cell_type": "markdown", "id": "87d6d5c6", "metadata": {}, "source": [ "**Write that number down.** That gap β€” the same multiplication, hundreds of times faster β€” is not a NumPy trick you can forget. It's the reason GPUs exist, the reason training big models is even possible, and the entire subject of **high-performance computing**. Same math. Wildly different speed. Everything a network does, it does by multiplying matrices β€” so making *that* fast makes *everything* fast." ] }, { "cell_type": "markdown", "id": "e1433c4e", "metadata": {}, "source": [ "## 6 Β· The forward pass: information flows through\n", "\n", "Now we assemble the network. Two layers, each a matrix multiply followed by a sigmoid squash:\n", "\n", "1. hidden values: `H = sigmoid(X @ W1 + b1)` β†’ shape (300, 16)\n", "2. output: `Ε· = sigmoid(H @ W2 + b2)` β†’ shape (300, 1)\n", "\n", "That's the **forward pass** β€” inputs in, prediction out.\n", "\n", "### πŸ§ͺ Exercise 3 β€” Write the forward pass\n", "\n", "- Fill in `forward` so it chains the two layers exactly as above.\n", "- Then we initialize small random weights and run it once. Untrained, it should output a (300, 1) array of mush near 0.5 β€” it hasn't learned anything yet." ] }, { "cell_type": "code", "execution_count": null, "id": "812e4962", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:32.837628Z", "iopub.status.busy": "2026-08-17T12:54:32.837439Z", "iopub.status.idle": "2026-08-17T12:54:32.844089Z", "shell.execute_reply": "2026-08-17T12:54:32.842767Z" } }, "outputs": [], "source": [ "def forward(X, W1, b1, W2, b2):\n", " \"\"\"Run inputs through the 2-layer network. Returns predictions in (0, 1).\"\"\"\n", " # ==================== YOUR CODE HERE ====================\n", " ### HINT: layer 1 -> H = sigmoid(X @ W1 + b1); layer 2 -> sigmoid(H @ W2 + b2)\n", " ...\n", "\n", "\n", "def init_weights(hidden=HIDDEN, seed=SEED):\n", " \"\"\"Small random starting weights β€” the knobs we'll tune. Returns [W1, b1, W2, b2].\"\"\"\n", " r = np.random.default_rng(seed)\n", " W1 = 0.5 * r.standard_normal((2, hidden))\n", " b1 = np.zeros((1, hidden))\n", " W2 = 0.5 * r.standard_normal((hidden, 1))\n", " b2 = np.zeros((1, 1))\n", " return [W1, b1, W2, b2]\n", "\n", "\n", "preds = forward(X, *init_weights())\n", "print(\"prediction shape:\", preds.shape)\n", "print(\"a few untrained predictions:\", np.round(preds[:4, 0], 3))\n", "assert preds.shape == (300, 1), \"forward should return one prediction per point\"" ] }, { "cell_type": "markdown", "id": "0207129e", "metadata": {}, "source": [ "## 7 Β· Loss: how wrong are we?\n", "\n", "To improve, the network needs a single number saying how bad its guesses are. We'll use **mean squared error**: the average of (prediction βˆ’ truth)Β². Zero means perfect; bigger means worse.\n", "\n", "### πŸ§ͺ Exercise 4 β€” Implement the loss\n", "\n", "- Fill in `mse(y_pred, y_true)` = the mean of the squared differences.\n", "- Print the loss of the untrained network. Guessing ~0.5 everywhere gives a loss near 0.25." ] }, { "cell_type": "code", "execution_count": null, "id": "4443fb5e", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:32.845968Z", "iopub.status.busy": "2026-08-17T12:54:32.845806Z", "iopub.status.idle": "2026-08-17T12:54:32.850427Z", "shell.execute_reply": "2026-08-17T12:54:32.849193Z" } }, "outputs": [], "source": [ "def mse(y_pred, y_true):\n", " \"\"\"Mean squared error β€” the average squared gap between guess and truth.\"\"\"\n", " # ==================== YOUR CODE HERE ====================\n", " ### HINT: np.mean((y_pred - y_true) ** 2)\n", " ...\n", "\n", "\n", "start_loss = mse(forward(X, *init_weights()), y)\n", "print(f\"untrained loss: {start_loss:.4f}\")" ] }, { "cell_type": "markdown", "id": "e447d175", "metadata": {}, "source": [ "## 8 Β· Learning without calculus: the nudge method\n", "\n", "Here's the beautiful part. The network has **65 knobs** (all the weights and biases). We want to know, for each one: *should I turn it up or down to lower the loss?*\n", "\n", "The simplest possible answer: **try it.** Nudge a knob up a hair, check the loss. Nudge it down a hair, check again. Whichever direction lowers the loss, step that way a little. Do it for every knob, over and over. That slope-from-nudging is exactly what a **gradient** is.\n", "\n", "The helper below computes that nudge for every weight. You'll write the one line that actually *takes the step*." ] }, { "cell_type": "code", "execution_count": null, "id": "cbfbe814", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:32.852365Z", "iopub.status.busy": "2026-08-17T12:54:32.852211Z", "iopub.status.idle": "2026-08-17T12:54:32.856587Z", "shell.execute_reply": "2026-08-17T12:54:32.855446Z" } }, "outputs": [], "source": [ "def numerical_gradient(loss_fn, params, eps=1e-4):\n", " \"\"\"The nudge method: estimate the gradient of loss_fn for every weight.\n", "\n", " For each weight w we measure loss(w + eps) and loss(w - eps); their\n", " difference over 2*eps is the slope (gradient) of the loss along that weight.\n", " Two forward passes per weight β€” remember that number for the Math Corner.\n", " \"\"\"\n", " grads = []\n", " for p in params:\n", " g = np.zeros_like(p)\n", " it = np.nditer(p, flags=[\"multi_index\"])\n", " while not it.finished:\n", " idx = it.multi_index\n", " original = p[idx]\n", " p[idx] = original + eps # nudge up\n", " loss_up = loss_fn()\n", " p[idx] = original - eps # nudge down\n", " loss_down = loss_fn()\n", " p[idx] = original # put it back exactly\n", " g[idx] = (loss_up - loss_down) / (2 * eps)\n", " it.iternext()\n", " grads.append(g)\n", " return grads" ] }, { "cell_type": "markdown", "id": "0d4f35b5", "metadata": {}, "source": [ "### πŸ§ͺ Exercise 5 β€” Take the step (this is learning)\n", "\n", "Everything below is given except the heart of it. Once per epoch we compute the nudge for every weight; you write the update that moves each weight **downhill**:\n", "\n", "- For each weight array `p` and its gradient `g`, subtract `lr * g` from `p`, **in place** (`p -= …`).\n", "\n", "Run it and watch the loss fall epoch by epoch as the network figures out the moons.\n", "\n", "*(Cost check: 65 weights Γ— 2 forward passes Γ— 400 epochs β‰ˆ 52,000 tiny forward passes on a 300-point network β€” a handful of seconds.)*" ] }, { "cell_type": "code", "execution_count": null, "id": "98082aec", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:32.858398Z", "iopub.status.busy": "2026-08-17T12:54:32.858248Z", "iopub.status.idle": "2026-08-17T12:54:34.847260Z", "shell.execute_reply": "2026-08-17T12:54:34.846010Z" } }, "outputs": [], "source": [ "def train_network(hidden=HIDDEN, epochs=EPOCHS, lr=LEARNING_RATE, record_every=50):\n", " \"\"\"Train with the nudge method. Returns (params, history).\"\"\"\n", " params = init_weights(hidden)\n", " loss_fn = lambda: mse(forward(X, *params), y)\n", "\n", " history = {\"epoch\": [], \"loss\": [], \"snapshots\": []}\n", " for epoch in range(epochs):\n", " grads = numerical_gradient(loss_fn, params)\n", " # ↓ the one line that makes learning happen: step every weight downhill\n", " # ==================== YOUR CODE HERE ====================\n", " ### HINT: in place, so use p -= lr * g (not p = p - lr*g)\n", " ...\n", " if epoch % record_every == 0 or epoch == epochs - 1:\n", " L = loss_fn()\n", " history[\"epoch\"].append(epoch)\n", " history[\"loss\"].append(L)\n", " history[\"snapshots\"].append([p.copy() for p in params])\n", " print(f\"epoch {epoch:4d} loss {L:.4f}\")\n", " return params, history\n", "\n", "\n", "trained_params, history = train_network()" ] }, { "cell_type": "markdown", "id": "2e6efe22", "metadata": {}, "source": [ "## 9 Β· Watch it learn\n", "\n", "We saved a snapshot of the network every 50 epochs. Let's replay them: watch the boundary start as a vague blob and bend itself around the two moons." ] }, { "cell_type": "code", "execution_count": null, "id": "af15ad2b", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:34.849219Z", "iopub.status.busy": "2026-08-17T12:54:34.849054Z", "iopub.status.idle": "2026-08-17T12:54:35.397414Z", "shell.execute_reply": "2026-08-17T12:54:35.396249Z" } }, "outputs": [], "source": [ "snaps = history[\"snapshots\"]\n", "eps_rec = history[\"epoch\"]\n", "n = len(snaps)\n", "cols = 3\n", "rows = int(np.ceil(n / cols))\n", "fig, axes = plt.subplots(rows, cols, figsize=(11, 3.4 * rows))\n", "axes = np.atleast_1d(axes).ravel()\n", "for ax, ep, snap in zip(axes, eps_rec, snaps):\n", " plot_boundary(lambda g, s=snap: forward(g, *s), X, y, ax=ax, title=f\"epoch {ep}\")\n", "for ax in axes[n:]:\n", " ax.axis(\"off\")\n", "fig.suptitle(\"the decision boundary learning to bend around the moons\")\n", "fig.tight_layout()\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "3dc6b558", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:35.399179Z", "iopub.status.busy": "2026-08-17T12:54:35.399036Z", "iopub.status.idle": "2026-08-17T12:54:35.408749Z", "shell.execute_reply": "2026-08-17T12:54:35.407833Z" } }, "outputs": [], "source": [ "plt.figure(figsize=(6, 4))\n", "plt.plot(history[\"epoch\"], history[\"loss\"], \"o-\")\n", "plt.xlabel(\"epoch\")\n", "plt.ylabel(\"MSE loss\")\n", "plt.title(\"the loss falling as the network learns\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "c82b9e6c", "metadata": {}, "source": [ "### πŸ§ͺ Exercise 6 β€” Score it\n", "\n", "Loss is abstract; accuracy is human. Turn the network's confidences into 0/1 decisions and measure the fraction it gets right.\n", "\n", "- In `accuracy`, threshold the predictions at 0.5, then compare to the truth and take the mean.\n", "- Our hand-built net should land around **88%** here β€” a giant leap from the single neuron's straight-line failure. (The ceiling on this noisy data is ~97%; tomorrow's power tools close most of that gap. Our simple \"nudge every weight\" trainer is honest but slow.)" ] }, { "cell_type": "code", "execution_count": null, "id": "20924365", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:35.410561Z", "iopub.status.busy": "2026-08-17T12:54:35.410411Z", "iopub.status.idle": "2026-08-17T12:54:35.414898Z", "shell.execute_reply": "2026-08-17T12:54:35.413979Z" } }, "outputs": [], "source": [ "def accuracy(y_pred, y_true):\n", " \"\"\"Fraction correct, calling anything above 0.5 a '1'.\"\"\"\n", " # ==================== YOUR CODE HERE ====================\n", " ### HINT: (y_pred > 0.5) turns confidences into 0/1; compare with == and take the mean\n", " ...\n", "\n", "\n", "final_acc = accuracy(forward(X, *trained_params), y)\n", "print(f\"final accuracy: {final_acc:.1%}\")\n", "if not SMOKE:\n", " assert final_acc > 0.85, \"accuracy is low β€” check forward(), the loss, or the update step\"" ] }, { "cell_type": "markdown", "id": "693b10ff", "metadata": {}, "source": [ "## 10 Β· Experiments: how many neurons do you need?\n", "\n", "The hidden layer's size controls how *wiggly* a boundary the network can draw. Too few neurons and it can't bend enough; too many and it can over-wiggle (memorizing noise instead of the shape). Let's see both extremes.\n", "\n", "### πŸ§ͺ Exercise 7 β€” Retrain with 2 and 64 hidden neurons\n", "\n", "- Inside the loop, train a network with `hidden=h` and plot its boundary.\n", "- Then compare the shapes you get." ] }, { "cell_type": "code", "execution_count": null, "id": "40533361", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:35.416388Z", "iopub.status.busy": "2026-08-17T12:54:35.416241Z", "iopub.status.idle": "2026-08-17T12:54:51.974324Z", "shell.execute_reply": "2026-08-17T12:54:51.973386Z" } }, "outputs": [], "source": [ "fig, axes = plt.subplots(1, 2, figsize=(11, 4.5))\n", "for ax, h in zip(axes, [2, 64]):\n", " # ==================== YOUR CODE HERE ====================\n", " ### HINT: params, _ = train_network(hidden=h); then plot_boundary(lambda g,s=params: forward(g,*s), X, y, ax=ax, ...)\n", " ...\n", "fig.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "cab9fe8b", "metadata": {}, "source": [ "**Describe what you see** *(double-click to edit):*\n", "- With **2 neurons**, the boundary is ...\n", "- With **64 neurons**, the boundary is ...\n", "\n", "Fewer neurons β†’ simpler boundary. More neurons β†’ more flexible (and, past a point, prone to memorizing noise). Choosing the right size is part of the art." ] }, { "cell_type": "markdown", "id": "0fce4991", "metadata": {}, "source": [ "And one knob you should *not* crank: the learning rate. A sensible step (our 0.5) walks steadily downhill. But make the step wildly too big and the weights get slammed into saturation β€” the network collapses to guessing one class and the loss flatlines. Watch it break." ] }, { "cell_type": "code", "execution_count": null, "id": "9e8792ba", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:51.983070Z", "iopub.status.busy": "2026-08-17T12:54:51.982933Z", "iopub.status.idle": "2026-08-17T12:54:55.680728Z", "shell.execute_reply": "2026-08-17T12:54:55.679488Z" } }, "outputs": [], "source": [ "import warnings\n", "\n", "with warnings.catch_warnings():\n", " warnings.simplefilter(\"ignore\") # giant steps overflow the sigmoids β€” that's the point\n", " chaos_params, chaos = train_network(lr=100.0)\n", "\n", "broken_acc = accuracy(forward(X, *chaos_params), y)\n", "print(f\"accuracy with a giant learning rate: {broken_acc:.1%} (β‰ˆ a coin flip β€” it never learned)\")\n", "\n", "plt.figure(figsize=(6, 4))\n", "plt.plot(chaos[\"epoch\"], chaos[\"loss\"], \"o-\", color=\"crimson\")\n", "plt.xlabel(\"epoch\")\n", "plt.ylabel(\"MSE loss\")\n", "plt.title(\"learning rate 100 β€” too big: the loss flatlines, the network never learns\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "952b51a3", "metadata": {}, "source": [ "## 11 Β· The loss landscape: what \"downhill\" means\n", "\n", "Every setting of the weights produces some loss. Picture that as a landscape β€” the weights are your map coordinates, the loss is the altitude. **Learning is just walking downhill** on this surface, and the gradient is the compass pointing down.\n", "\n", "We can't draw a 65-dimensional landscape, but we can take a **2-weight slice**: freeze every weight at its trained value except two, sweep those two across a grid, and plot the loss as height. The red star marks where training actually landed." ] }, { "cell_type": "code", "execution_count": null, "id": "d315635f", "metadata": { "execution": { "iopub.execute_input": "2026-08-17T12:54:55.682388Z", "iopub.status.busy": "2026-08-17T12:54:55.682237Z", "iopub.status.idle": "2026-08-17T12:54:55.734705Z", "shell.execute_reply": "2026-08-17T12:54:55.733386Z" } }, "outputs": [], "source": [ "i_a, i_b = 0, 2 # vary W1[0,0] and W2[0,0]\n", "a_star = trained_params[i_a][0, 0]\n", "b_star = trained_params[i_b][0, 0]\n", "span = 3.0\n", "a_vals = np.linspace(a_star - span, a_star + span, 25)\n", "b_vals = np.linspace(b_star - span, b_star + span, 25)\n", "\n", "Z = np.zeros((25, 25))\n", "for r, bv in enumerate(b_vals):\n", " for c, av in enumerate(a_vals):\n", " trial = [p.copy() for p in trained_params]\n", " trial[i_a][0, 0] = av\n", " trial[i_b][0, 0] = bv\n", " Z[r, c] = mse(forward(X, *trial), y)\n", "\n", "AA, BB = np.meshgrid(a_vals, b_vals)\n", "plt.figure(figsize=(7, 5))\n", "cs = plt.contourf(AA, BB, Z, levels=30, cmap=\"viridis\")\n", "plt.colorbar(cs, label=\"loss\")\n", "plt.scatter([a_star], [b_star], color=\"red\", marker=\"*\", s=280,\n", " edgecolor=\"white\", linewidth=1.2, label=\"where training landed\", zorder=5)\n", "plt.xlabel(\"weight W1[0, 0]\")\n", "plt.ylabel(\"weight W2[0, 0]\")\n", "plt.title(\"a 2-weight slice of the loss landscape\")\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "fd7676ec", "metadata": {}, "source": [ "## 12 Β· Math Corner (optional)\n", "\n", "
πŸ”’ Math Corner β€” the nudge method vs. real backprop (optional)\n", "\n", "A **derivative** is just a slope: nudge the input a tiny bit, see how much the output moves, divide by the nudge. That's literally what `numerical_gradient` does β€” one pair of nudges per weight.\n", "\n", "It works, but it's *slow*. Our tiny network has **2Β·16 + 16 + 16 + 1 = 65 weights**. The nudge method needs two forward passes **per weight** to estimate its slope β€” that's **130 forward passes for a single training step.** A network with a million weights would need two million forward passes per step. Hopeless.\n", "\n", "Real networks use **backpropagation**: with the chain rule from calculus, it computes the *exact* slope for **every** weight in just **two passes total** (one forward, one backward) β€” no matter how many weights there are. 130 versus 2. That efficiency is the only reason training large networks is even possible.\n", "\n", "Tomorrow, PyTorch's **autograd** runs this backward pass for you automatically β€” and on a GPU, where (remember Section 5) those matrix multiplies fly.\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "36b8c3c9", "metadata": {}, "source": [ "## 🏁 Checkpoint\n", "\n", "Today, with nothing but NumPy, you:\n", "\n", "- built a **neuron** and saw why one straight cut isn't enough\n", "- learned the course's biggest secret β€” **a network is chained matrix multiplications** (and clocked NumPy beating a hand-written loop by hundreds of times)\n", "- wrote the **forward pass** and a **loss**, then trained the net with the **nudge method**\n", "- watched the decision **boundary bend** around the moons and read the **loss landscape** it descended\n", "\n", "You forged the engine by hand. **Tomorrow the Lab hands you power tools** β€” PyTorch, automatic gradients, and a GPU β€” and you'll teach a network to read handwritten digits, the very same ones you mapped with t-SNE yesterday.\n", "\n", "**Next: Day 5 β€” The Digit Reader ⚑**" ] }, { "cell_type": "markdown", "id": "81cdd0d3", "metadata": {}, "source": [ "---\n", "\n", "
\n", " \n", " \"Dr.\n", " \n", "
\n", "\n", "
The Lab Β· AI Studio β€” Intro to AI Β· Β© 2026 Dr. Abdelghafour HALIMI Β· ahalimi.com Β· All rights reserved.
" ] } ], "metadata": { "authors": [ { "name": "Dr. Abdelghafour HALIMI" } ], "copyright": "Β© 2026 Dr. Abdelghafour HALIMI Β· ahalimi.com Β· All rights reserved", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }