{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8e56c4b6",
   "metadata": {},
   "source": [
    "# Day 9 — Machines That Act 🎮\n",
    "\n",
    "> **Mission briefing:** Everything the Lab has built so far *perceives* — it classifies penguins, clusters colors, reads handwriting, understands words. Today we build something new: an agent that *decides*. There are no labels and no answer key — only a world, actions, and **rewards**. You will train a brain the way you would train a puppy: a treat for every good move.\n",
    "\n",
    "**Previously in the Lab:** yesterday you unlocked the 💬 **Language Lab** and made transformer models write stories, read moods, and fill in the blanks.\n",
    "\n",
    "**Today you will:**\n",
    "- Meet **reinforcement learning** — the loop of *act → get a reward → learn*\n",
    "- Teach an agent to cross a frozen lake with a **Q-table** (64 numbers that become an instinct)\n",
    "- Write the single line of math — the **Q-learning update** — that sits at the heart of it\n",
    "- Watch a neural network (a **Deep Q-Network**) learn to balance a pole from nothing\n",
    "- See why this is how AlphaGo won, how robots learn to walk, and (as **RLHF**) part of how ChatGPT was polished\n",
    "\n",
    "**Today you unlock:** 🔓 **🎮 Game Master** — replay both agents you trained, live in your Studio.\n",
    "\n",
    "<details data-resource-precision=\"9\"><summary>A precise reading</summary><p>RL learns from experience: states, actions, rewards and transitions. It does not require a correct-action label for every step. For a truly terminal transition, set the next-state value to zero in the Q target. A time-limit truncation is a separate case and can still bootstrap. The FrozenLake exercise uses deterministic, non-slippery transitions.</p></details>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97a259ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ╔═══════════════════════════════════════════════════════════════════╗\n",
    "# ║ SETUP — run me first! (identical in every Lab notebook)           ║\n",
    "# ╚═══════════════════════════════════════════════════════════════════╝\n",
    "import os, sys, pathlib\n",
    "\n",
    "IN_COLAB = \"google.colab\" in sys.modules\n",
    "SMOKE = os.environ.get(\"SMOKE_TEST\", \"0\") == \"1\"   # tiny fast run for automated tests\n",
    "\n",
    "REPO_URL = \"https://github.com/A-Halimi/ai-studio-internship.git\"  # instructor: set once\n",
    "\n",
    "if IN_COLAB:\n",
    "    if not pathlib.Path(\"/content/ai-studio-internship\").exists():\n",
    "        !git clone -q {REPO_URL} /content/ai-studio-internship\n",
    "    %cd /content/ai-studio-internship/notebooks/day09                    # ← ADAPT day folder\n",
    "    %pip -q install \"gymnasium[classic-control,toy-text]==1.3.0\" stable-baselines3==2.9.0 \"imageio[ffmpeg]==2.37.3\" gradio==6.19.0   # ← ADAPT per-day installs\n",
    "\n",
    "HERE = pathlib.Path.cwd()\n",
    "REPO = HERE.parents[1]                       # .../notebooks/dayNN -> repo root\n",
    "DATA_DIR = pathlib.Path(os.environ.get(\"COURSE_DATA_DIR\", REPO / \"data\"))\n",
    "SAMPLES_DIR = REPO / \"data_samples\"          # small datasets shipped with the repo\n",
    "MODELS_DIR = REPO / \"ai_studio\" / \"models\"   # where your Studio modules unlock\n",
    "for p in (REPO / \"data\", MODELS_DIR):\n",
    "    p.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "import random\n",
    "import numpy as np\n",
    "SEED = 42\n",
    "random.seed(SEED); np.random.seed(SEED)\n",
    "print(f\"✅ Setup done | Colab: {IN_COLAB} | Smoke test: {SMOKE}\")\n",
    "print(f\"   data: {DATA_DIR}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9621873",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "torch.manual_seed(SEED)\n",
    "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "if DEVICE.type == \"cuda\":\n",
    "    print(f\"⚡ Running on GPU: {torch.cuda.get_device_name(0)}\")\n",
    "else:\n",
    "    print(\"🐢 No GPU found — running on CPU. Everything still works, just slower.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d072430b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── CONFIG — the Lab's control panel. Tweak me later! ──────────────\n",
    "EPISODES  = 300 if SMOKE else 3000     # Q-learning episodes on FrozenLake\n",
    "TIMESTEPS = 3_000 if SMOKE else 60_000 # Deep-Q-Network training steps on CartPole\n",
    "ALPHA = 0.8    # learning rate for the Q-table (how big a step we take)\n",
    "GAMMA = 0.95   # discount: how much the agent values *future* rewards (patience)\n",
    "print(f\"EPISODES={EPISODES} | TIMESTEPS={TIMESTEPS} | ALPHA={ALPHA} | GAMMA={GAMMA}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c044fc10",
   "metadata": {},
   "source": [
    "## 1 · The reinforcement-learning loop\n",
    "\n",
    "Supervised learning (for example, Day 2) needs an **answer key**: a photo *labeled* \"cat\", a review *labeled* \"positive\". Reinforcement learning throws the answer key away. Instead there is a loop:\n",
    "\n",
    "> The **agent** looks at the current **state**, picks an **action**, and the **environment** replies with a **reward** and a new state. Repeat thousands of times. The agent's only goal: collect as much reward as possible over the long run.\n",
    "\n",
    "```\n",
    "        ┌──────────────────────────────────────────┐\n",
    "        │                                          │\n",
    "   ┌────▼────┐   action a    ┌───────────────┐     │\n",
    "   │  AGENT  │ ────────────▶ │  ENVIRONMENT  │     │\n",
    "   │ (brain) │               │   (a world)   │     │\n",
    "   └─────────┘ ◀──────────── └───────────────┘     │\n",
    "        │        reward r + new state s'           │\n",
    "        └──────────────────────────────────────────┘\n",
    "```\n",
    "\n",
    "The agent is never *told* the right move. It has to **discover** which moves pay off, purely from the rewards. That is the whole idea — and it is remarkably general:\n",
    "\n",
    "| Situation | The reward the agent chases |\n",
    "|---|---|\n",
    "| Playing a video game | the score going up |\n",
    "| A robot learning to walk | distance covered without falling over |\n",
    "| A self-driving car | staying in its lane and reaching the destination |\n",
    "| Polishing ChatGPT (**RLHF**) | a human clicking 👍 instead of 👎 |\n",
    "\n",
    "One loop, a thousand applications. Let's build the smallest possible version of it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4579058",
   "metadata": {},
   "source": [
    "## 2 · Meet FrozenLake ❄️\n",
    "\n",
    "Our first world is a tiny 4×4 frozen pond from the **Gymnasium** library (the standard toolbox of RL practice environments). The rules:\n",
    "\n",
    "- The agent starts at **S** (top-left) and wants the **gift** 🎁 at **G** (bottom-right).\n",
    "- Some tiles are **holes** (H) — step on one and the episode ends with nothing.\n",
    "- **16 states** (the tiles, numbered 0–15) and **4 actions**: `0=LEFT  1=DOWN  2=RIGHT  3=UP`.\n",
    "- **Reward = 1** *only* when it reaches the gift. Every other step gives **0**.\n",
    "\n",
    "That last point is what makes this hard: the agent gets almost no feedback. It has to stumble onto the gift by luck at least once, then learn to repeat the path. We use `is_slippery=False` so the ice behaves predictably — a move you choose is the move you make."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6d4abbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ.setdefault(\"SDL_VIDEODRIVER\", \"dummy\")   # let pygame draw with no screen (cloud/headless)\n",
    "import gymnasium as gym\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "env = gym.make(\"FrozenLake-v1\", map_name=\"4x4\", is_slippery=False,\n",
    "               render_mode=\"rgb_array\")\n",
    "state, _ = env.reset(seed=SEED)\n",
    "\n",
    "print(\"Number of states :\", env.observation_space.n)   # 16 tiles\n",
    "print(\"Number of actions:\", env.action_space.n)        # 4 moves (L, D, R, U)\n",
    "print(\"The pond (S=start  F=frozen  H=hole  G=gift):\")\n",
    "print(env.unwrapped.desc.astype(str))\n",
    "\n",
    "plt.figure(figsize=(4, 4))\n",
    "plt.imshow(env.render())\n",
    "plt.axis(\"off\")\n",
    "plt.title(\"FrozenLake 4x4 — get from S (top-left) to the gift (bottom-right)\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a87c094e",
   "metadata": {},
   "source": [
    "### 🧪 Exercise 1 — how far does *pure luck* get?\n",
    "\n",
    "Before we teach the agent anything, let's measure the baseline: an agent that ignores everything and picks **random** moves.\n",
    "\n",
    "- Run many episodes where every action comes from `env.action_space.sample()`.\n",
    "- Count how often it stumbles onto the gift, and report the success rate.\n",
    "\n",
    "Expected outcome: a dismal **~1–5%**. Wandering blindly across ice rarely ends well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55948a7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "N_RANDOM = 200 if SMOKE else 1000\n",
    "env.action_space.seed(SEED)          # make the \"random\" moves reproducible\n",
    "\n",
    "wins = 0\n",
    "for episode in range(N_RANDOM):\n",
    "    state, _ = env.reset()\n",
    "    done = False\n",
    "    # ==================== YOUR CODE HERE ====================\n",
    "    ### HINT: env.action_space.sample() returns a random action (0-3)\n",
    "    ### HINT: after the episode, reward > 0 means it finished on the gift\n",
    "    ...\n",
    "\n",
    "success_rate = wins / N_RANDOM\n",
    "print(f\"🎲 Random agent reached the gift in {success_rate:.1%} of {N_RANDOM} episodes\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac739126",
   "metadata": {},
   "source": [
    "## 3 · The Q-table — the agent's cheat sheet\n",
    "\n",
    "Here is the key idea. Imagine a table with **one row per state** and **one column per action**. Each cell holds a single number, `Q(state, action)` — the agent's estimate of *\"how good is it to take this action from this state, if I play well afterward?\"*\n",
    "\n",
    "For FrozenLake that's a **16 × 4 = 64-number** table. If the agent could fill it in correctly, choosing the best move would be trivial: in any state, look at that row and pick the column with the biggest number.\n",
    "\n",
    "The agent starts knowing **nothing**, so every cell begins at zero:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3eebf72",
   "metadata": {},
   "outputs": [],
   "source": [
    "q = np.zeros((16, 4))     # 16 states (rows) x 4 actions (columns)\n",
    "print(\"The Q-table starts as all zeros — the agent knows nothing yet:\")\n",
    "print(q)\n",
    "print(\"Shape:\", q.shape, \"→ one row per state, one column per action\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "326e505a",
   "metadata": {},
   "source": [
    "### 🧪 Exercise 2 — explore vs. exploit (`choose_action`)\n",
    "\n",
    "An agent that always picks the highest number in its table will keep repeating whatever it tried first — it never discovers better moves. An agent that always moves randomly never *uses* what it learned. The fix is **ε-greedy**:\n",
    "\n",
    "- With probability **ε** (epsilon): **explore** — take a random action.\n",
    "- Otherwise: **exploit** — take the action with the largest Q-value for this state.\n",
    "\n",
    "It's the restaurant dilemma: most nights you go to your favorite (exploit), but once in a while you try somewhere new (explore) in case it's even better. Fill in `choose_action`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c76a3f59",
   "metadata": {},
   "outputs": [],
   "source": [
    "def choose_action(state, q, eps):\n",
    "    \"\"\"Epsilon-greedy: explore with probability eps, otherwise exploit the Q-table.\"\"\"\n",
    "    # ==================== YOUR CODE HERE ====================\n",
    "    ### HINT: np.random.random() draws a float in [0, 1)\n",
    "    ### HINT: np.argmax(q[state]) is the column with the biggest Q-value for this state\n",
    "    ...\n",
    "\n",
    "# quick check: with eps=0 it must ALWAYS pick the greedy action\n",
    "_probe = np.zeros((16, 4)); _probe[3] = [0, 0, 9, 0]\n",
    "assert choose_action(3, _probe, eps=0.0) == 2, \"With eps=0 you should always exploit!\"\n",
    "print(\"✅ choose_action works — with eps=0 it picks the best action (column 2 here).\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "648d2cbc",
   "metadata": {},
   "source": [
    "### 🧪 Exercise 3 — start bold, end wise (`get_epsilon`)\n",
    "\n",
    "Early in training the agent should **explore a lot** (it knows nothing). Later it should mostly **exploit** what it has learned. So we let ε **decay** over the run — from `1.0` (all exploration) down to `0.01` (almost always greedy).\n",
    "\n",
    "Write a **linear** schedule: on episode `0` return `EPS_START`; on the final episode return (about) `EPS_END`; blend smoothly in between."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fe9b40d",
   "metadata": {},
   "outputs": [],
   "source": [
    "EPS_START, EPS_END = 1.0, 0.01\n",
    "def get_epsilon(episode, n_episodes):\n",
    "    \"\"\"Linear decay from EPS_START (all exploration) to EPS_END (almost all exploit).\"\"\"\n",
    "    # ==================== YOUR CODE HERE ====================\n",
    "    ### HINT: frac should run from 0 (first episode) to 1 (last episode)\n",
    "    ### HINT: blend EPS_START and EPS_END using frac\n",
    "    ...\n",
    "\n",
    "assert abs(get_epsilon(0, EPISODES) - 1.0) < 1e-9, \"Episode 0 should be full exploration\"\n",
    "assert get_epsilon(EPISODES - 1, EPISODES) <= 0.02, \"The last episode should be near-greedy\"\n",
    "print(f\"✅ epsilon starts at {get_epsilon(0, EPISODES):.2f} \"\n",
    "      f\"and ends at {get_epsilon(EPISODES-1, EPISODES):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f253cff",
   "metadata": {},
   "source": [
    "## 4 · ★ The Q-learning update — the heart of it all\n",
    "\n",
    "Every time the agent takes action `a` in state `s`, lands in state `s'`, and collects reward `r`, it improves **one cell** of its table with this rule:\n",
    "\n",
    "> **new opinion of (s, a)  =  old opinion  +  ALPHA × ( reward  +  GAMMA × best future value  −  old opinion )**\n",
    "\n",
    "In code, that's a single line:\n",
    "\n",
    "```python\n",
    "q[s, a] += ALPHA * (r + GAMMA * np.max(q[s_next]) - q[s, a])\n",
    "```\n",
    "\n",
    "Read it slowly. `r + GAMMA * np.max(q[s_next])` is a **fresh estimate** of how good `(s, a)` really was — the reward we just got, plus the discounted value of the *best* move available from where we landed. We nudge the old value a fraction `ALPHA` of the way toward that fresh estimate. Do this a few thousand times and the table fills itself in.\n",
    "\n",
    "<details><summary>🔢 <b>Math Corner</b> — the update in symbols (optional)</summary>\n",
    "\n",
    "$$Q(s,a) \\leftarrow Q(s,a) + \\alpha\\,\\big[\\, r + \\gamma \\max_{a'} Q(s',a') - Q(s,a) \\,\\big]$$\n",
    "\n",
    "- $Q(s,a)$ — how good we *currently* think action $a$ is in state $s$.\n",
    "- $r$ — the reward we just received.\n",
    "- $\\max_{a'} Q(s',a')$ — the value of the **best** move from the new state $s'$.\n",
    "- $\\gamma$ (GAMMA) — a **patience knob** in $[0,1]$: near $0$ chases reward *now*; near $1$ plans far ahead.\n",
    "- $\\alpha$ (ALPHA) — the **learning rate**: the step size toward the new estimate.\n",
    "\n",
    "The bracketed part is the **temporal-difference error** — the gap between what we expected and what happened. Learning is just shrinking that gap.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55ecc402",
   "metadata": {},
   "source": [
    "### 🧪 Exercise 4 — train the agent (the crown jewel) 👑\n",
    "\n",
    "Everything comes together in one loop. For each episode we reset the pond, decay ε, then step until the episode ends — choosing an action and updating the table each step. **Two lines are yours to fill:**\n",
    "\n",
    "1. Pick the action with the `choose_action` function you wrote.\n",
    "2. Apply the **Q-learning update** from the section above.\n",
    "\n",
    "Watch the printed success rate climb from near-zero toward **100%**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8507348",
   "metadata": {},
   "outputs": [],
   "source": [
    "q = np.zeros((16, 4))          # start again from a blank cheat sheet\n",
    "successes = []                  # 1.0 if the episode reached the gift, else 0.0\n",
    "\n",
    "for episode in range(EPISODES):\n",
    "    state, _ = env.reset()\n",
    "    eps = get_epsilon(episode, EPISODES)\n",
    "    done = False\n",
    "    while not done:\n",
    "        # ==================== YOUR CODE HERE ====================\n",
    "        ### HINT: reuse the choose_action function you wrote in Exercise 2\n",
    "        ...\n",
    "        next_state, reward, terminated, truncated, _ = env.step(action)\n",
    "        # ==================== YOUR CODE HERE ====================\n",
    "        ### HINT: new = old + ALPHA * (reward + GAMMA * best_next - old)\n",
    "        ...\n",
    "        state = next_state\n",
    "        done = terminated or truncated\n",
    "    successes.append(1.0 if reward > 0 else 0.0)\n",
    "\n",
    "    if (episode + 1) % max(1, EPISODES // 10) == 0:\n",
    "        recent = np.mean(successes[-100:])\n",
    "        print(f\"  episode {episode+1:>5}/{EPISODES} | eps={eps:4.2f} | recent success = {recent:5.1%}\")\n",
    "\n",
    "print(\"🏁 Training done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6fa77ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "window = 100\n",
    "rolling = np.convolve(successes, np.ones(window) / window, mode=\"valid\")\n",
    "plt.figure(figsize=(9, 4))\n",
    "plt.plot(range(window, window + len(rolling)), rolling)\n",
    "plt.xlabel(\"episode\")\n",
    "plt.ylabel(f\"success rate (rolling mean over {window} episodes)\")\n",
    "plt.title(\"The agent learns to cross the ice\")\n",
    "plt.ylim(-0.05, 1.05)\n",
    "plt.grid(alpha=0.3)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8823e0bb",
   "metadata": {},
   "source": [
    "### 🧪 Exercise 5 — how good did it get?\n",
    "\n",
    "Measure the agent's skill at the *end* of training: the mean success over the **last 200 episodes**. For a non-slippery 4×4 pond a trained agent should be essentially perfect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b704ad7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ==================== YOUR CODE HERE ====================\n",
    "### HINT: slice the last 200 with successes[-200:], then np.mean(...)\n",
    "...\n",
    "print(f\"🎯 Success over the last 200 training episodes: {final_success:.1%}\")\n",
    "if not SMOKE:\n",
    "    assert final_success >= 0.9, \"Below 90% — double-check your Q-update line!\"\n",
    "    print(\"✅ The agent has essentially mastered the pond.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43c010d6",
   "metadata": {},
   "source": [
    "## 5 · Look inside the brain\n",
    "\n",
    "Those 64 numbers *are* the agent's mind. From them we can read out its complete strategy — a **policy**: in each state, the single best move."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15476c83",
   "metadata": {},
   "source": [
    "### 🧪 Exercise 6 — read out the policy\n",
    "\n",
    "Collapse each row of the table into two summaries:\n",
    "\n",
    "- `best_action` — for every state, the action with the largest Q-value.\n",
    "- `best_value` — for every state, how large that best value is.\n",
    "\n",
    "Both are one call to a NumPy function along **axis 1** (across the 4 action-columns)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b81aea7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ==================== YOUR CODE HERE ====================\n",
    "### HINT: axis=1 collapses the 4 action-columns, leaving one number per state\n",
    "...\n",
    "print(\"Best action per state (0=L 1=D 2=R 3=U), laid out on the 4x4 pond:\")\n",
    "print(best_action.reshape(4, 4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f777ff50",
   "metadata": {},
   "outputs": [],
   "source": [
    "ARROWS = {0: \"←\", 1: \"↓\", 2: \"→\", 3: \"↑\"}   # ← ↓ → ↑\n",
    "desc = env.unwrapped.desc.astype(str)      # 4x4 grid of letters\n",
    "\n",
    "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))\n",
    "\n",
    "# Left: the learned policy as arrows\n",
    "ax1.imshow(np.zeros((4, 4)), cmap=\"Blues\", vmin=0, vmax=1)\n",
    "for s in range(16):\n",
    "    r, c = divmod(s, 4)\n",
    "    tile = desc[r, c]\n",
    "    if tile == \"H\":\n",
    "        ax1.text(c, r, \"H\", ha=\"center\", va=\"center\", fontsize=18, color=\"red\", weight=\"bold\")\n",
    "    elif tile == \"G\":\n",
    "        ax1.text(c, r, \"G\", ha=\"center\", va=\"center\", fontsize=18, color=\"green\", weight=\"bold\")\n",
    "    else:\n",
    "        ax1.text(c, r, ARROWS[int(best_action[s])], ha=\"center\", va=\"center\", fontsize=26)\n",
    "ax1.set_title(\"The learned policy (best move per tile)\")\n",
    "ax1.set_xticks([]); ax1.set_yticks([])\n",
    "\n",
    "# Right: how valuable each tile is (max Q)\n",
    "im = ax2.imshow(best_value.reshape(4, 4), cmap=\"viridis\")\n",
    "for s in range(16):\n",
    "    r, c = divmod(s, 4)\n",
    "    ax2.text(c, r, f\"{best_value[s]:.2f}\", ha=\"center\", va=\"center\",\n",
    "             color=\"white\", fontsize=10)\n",
    "ax2.set_title(\"Value of each tile (max Q)\")\n",
    "ax2.set_xticks([]); ax2.set_yticks([])\n",
    "fig.colorbar(im, ax=ax2, fraction=0.046, pad=0.04)\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "print(\"64 numbers = a complete skater's instinct for this pond.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "watchlearn-md",
   "metadata": {},
   "source": [
    "## 6 · Watch it learn 🎥\n",
    "\n",
    "You've dissected the finished brain — now watch it being **born**. This dashboard re-runs training from a **blank Q-table** (your Exercise-4 table `q` stays untouched), powered by the very functions *you* wrote: `choose_action` picks the moves, `get_epsilon` turns the exploration dial, and your Q-update line does the learning — live, on camera.\n",
    "\n",
    "**How to read the dashboard:**\n",
    "- **Left — the pond:** the elf actually skating (bright during 🎥 LIVE episodes, dimmed while ⏩ fast-forwarding).\n",
    "- **Right — the cheat sheet as a map:** each tile shows its current best move, and **the greener a tile, the more reward the agent expects from it**. The orange box marks where the elf is standing. Hover any tile to see its four raw Q-values.\n",
    "- **Top:** the episode counter, the exploration dial **ε** falling from 100% (pure dice) to 1% (pure cheat sheet), total wins, and the recent success rate.\n",
    "- **Bottom:** the learning curve **building itself** — one bar per chunk of episodes.\n",
    "\n",
    "We can't slow-play thousands of episodes, so the camera ⏩ **fast-forwards** through training and drops into 🎥 **slow-motion LIVE** at landmark episodes: **1, 2, 3** (pure random flailing — most end in 💧), then **10, 30, 100, 300, 1000**, and the **final** one (a confident beeline to the 🎁).\n",
    "\n",
    "👀 **The thing to watch:** the green doesn't spread out from the start — it **grows backwards from the gift**. The tile beside the 🎁 learns first, then its neighbour, then the whole path lights up. That's your Q-update in action: every win pushes a little value one step further back along the route."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "watchlearn-code",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ══════════════════════════════════════════════════════════════════════\n",
    "#  WATCH IT LEARN 🎥 — the training run, live on camera.\n",
    "#  A fresh, blank Q-table (`q_watch`) trains before your eyes — your\n",
    "#  Exercise-4 table `q` is left untouched. Fast-forward through the\n",
    "#  boring parts, slow-motion 🎥 LIVE at landmark episodes.\n",
    "# ══════════════════════════════════════════════════════════════════════\n",
    "import io, time, base64\n",
    "from PIL import Image as PILImage\n",
    "from IPython.display import HTML, display\n",
    "\n",
    "LANDMARKS = {1, 2, 3, 10, 30, 100, 300, 1000, EPISODES}   # episodes shown LIVE\n",
    "FF_EVERY  = 25                        # fast-forward: redraw dashboard every N episodes\n",
    "CHUNK     = max(10, EPISODES // 30)   # one learning-curve bar per CHUNK episodes\n",
    "TILES     = env.unwrapped.desc.astype(str)   # the pond map: S, F, H, G\n",
    "\n",
    "def frame_b64(frame):\n",
    "    \"\"\"Rendered numpy image → base64 PNG we can embed in live HTML.\"\"\"\n",
    "    buf = io.BytesIO()\n",
    "    PILImage.fromarray(frame).save(buf, format=\"PNG\")\n",
    "    return base64.b64encode(buf.getvalue()).decode()\n",
    "\n",
    "def cheatsheet_html(qt, agent_state=None):\n",
    "    \"\"\"The Q-table drawn as a map: best move per tile, greener = more promising.\n",
    "    Hover any tile to see its four raw Q-values.\"\"\"\n",
    "    rows = \"\"\n",
    "    for r in range(4):\n",
    "        cells = \"\"\n",
    "        for c in range(4):\n",
    "            s, tile = 4 * r + c, TILES[r, c]\n",
    "            hover = \"  \".join(f\"{ARROWS[a]} {qt[s, a]:.2f}\" for a in range(4))\n",
    "            style = \"width:54px;height:54px;text-align:center;font-size:22px;border:1px solid #9fc3de;\"\n",
    "            if tile == \"H\":\n",
    "                inner, style = \"💧\", style + \"background:#16324a;\"\n",
    "            elif tile == \"G\":\n",
    "                inner, style = \"🎁\", style + \"background:#ffe9a8;\"\n",
    "            else:\n",
    "                inner = ARROWS[int(np.argmax(qt[s]))] if qt[s].max() > 0 else \"·\"\n",
    "                style += f\"background:rgba(46,160,67,{qt[s].max():.2f});\"   # greener = higher Q\n",
    "            if s == agent_state:\n",
    "                style += \"outline:3px solid #ff6d00;outline-offset:-3px;\"   # the elf is HERE\n",
    "            cells += f\"<td title='{hover}' style='{style}'>{inner}</td>\"\n",
    "        rows += f\"<tr>{cells}</tr>\"\n",
    "    return f\"<table style='border-collapse:collapse'>{rows}</table>\"\n",
    "\n",
    "def curve_html():\n",
    "    \"\"\"The learning curve, building itself one bar at a time.\"\"\"\n",
    "    if not rate_history:\n",
    "        return \"\"\n",
    "    bars = \"\".join(\n",
    "        f\"<div title='{v:.0f}%' style='width:11px;height:{max(2, v * 0.42):.0f}px;\"\n",
    "        f\"background:#4c72b0;border-radius:2px 2px 0 0'></div>\" for v in rate_history)\n",
    "    return (f\"<div style='display:flex;align-items:flex-end;gap:2px;height:46px;margin-top:8px'>{bars}</div>\"\n",
    "            f\"<div style='color:#777'>the learning curve, building itself — one bar = % wins per {CHUNK} episodes</div>\")\n",
    "\n",
    "def draw_dash(banner, ep, eps, agent_state=None, live=False, note=\"\"):\n",
    "    \"\"\"Redraw the whole dashboard in place.\"\"\"\n",
    "    pct = 100 * (ep + 1) / EPISODES\n",
    "    rate = 100 * np.mean(recent) if recent else 0.0\n",
    "    dim = \"\" if live else \"opacity:0.45;\"          # pond dims while fast-forwarding\n",
    "    dash.update(HTML(\n",
    "        f\"<div style='font-family:monospace;font-size:13px;line-height:1.6'>\"\n",
    "        f\"<div style='font-size:15px;white-space:nowrap'><b>{banner}</b></div>\"\n",
    "        f\"<div style='white-space:nowrap'>episode <b style='font-size:16px'>{ep + 1:,}</b> / {EPISODES:,}\"\n",
    "        f\" · 🎲 exploration ε = <b>{100 * eps:.0f}%</b>\"\n",
    "        f\" · 🏆 wins <b>{wins}</b> · 📈 success rate (last 100) <b>{rate:.0f}%</b></div>\"\n",
    "        f\"<div style='background:#e0e0e0;width:470px;height:8px;border-radius:4px;margin:3px 0'>\"\n",
    "        f\"<div style='background:#4c72b0;width:{pct:.1f}%;height:8px;border-radius:4px'></div></div>\"\n",
    "        f\"<div style='display:flex;gap:14px;margin-top:5px;align-items:flex-start'>\"\n",
    "        f\"<img src='data:image/png;base64,{board_b64}' width='224' style='{dim}border-radius:4px'/>\"\n",
    "        f\"{cheatsheet_html(q_watch, agent_state)}</div>\"\n",
    "        f\"<div style='min-height:22px;font-size:14px'><b>{note}</b></div>\"\n",
    "        f\"{curve_html()}</div>\"))\n",
    "\n",
    "# ── train a fresh cheat sheet, on camera ───────────────────────────────\n",
    "q_watch = np.zeros((16, 4))               # blank: 64 zeros, zero knowledge\n",
    "wins, chunk_wins, recent, rate_history = 0, 0, [], []\n",
    "dash = display(HTML(\"<b>🎥 Setting up the training camera…</b>\"), display_id=True)\n",
    "env.reset()\n",
    "board_b64 = frame_b64(env.render())\n",
    "\n",
    "for ep in range(EPISODES):                             # the same loop as Exercise 4...\n",
    "    eps = get_epsilon(ep, EPISODES)                    # ...run with YOUR functions\n",
    "    state, _ = env.reset()\n",
    "    live = (ep + 1) in LANDMARKS\n",
    "    if live:\n",
    "        board_b64 = frame_b64(env.render())\n",
    "        style = (\"the dice decide almost every move!\" if eps > 0.5\n",
    "                 else \"the cheat sheet decides almost every move\")\n",
    "        draw_dash(f\"🎥 LIVE — episode {ep + 1:,} begins\", ep, eps, agent_state=state,\n",
    "                  live=True, note=f\"ε = {100 * eps:.0f}% → {style}\")\n",
    "        time.sleep(0 if SMOKE else (1.0 if ep < 3 else 0.5))\n",
    "    done, step = False, 0\n",
    "    while not done:\n",
    "        step += 1\n",
    "        action = choose_action(state, q_watch, eps)    # YOUR Exercise 2\n",
    "        next_state, reward, terminated, truncated, _ = env.step(action)\n",
    "        q_watch[state, action] += ALPHA * (reward + GAMMA * np.max(q_watch[next_state])\n",
    "                                           - q_watch[state, action])   # YOUR Exercise 4 line\n",
    "        state = next_state\n",
    "        done = terminated or truncated\n",
    "        if live:\n",
    "            board_b64 = frame_b64(env.render())\n",
    "            draw_dash(f\"🎥 LIVE — episode {ep + 1:,} · step {step}\", ep, eps,\n",
    "                      agent_state=state, live=True)\n",
    "            time.sleep(0 if SMOKE else (0.28 if ep < 3 and step <= 25 else 0.11))\n",
    "    win = reward > 0\n",
    "    wins += win\n",
    "    chunk_wins += win\n",
    "    recent = (recent + [win])[-100:]\n",
    "    if (ep + 1) % CHUNK == 0:\n",
    "        rate_history.append(100 * chunk_wins / CHUNK)\n",
    "        chunk_wins = 0\n",
    "    if live:\n",
    "        note = (\"🎁 GOT THE GIFT!\" if win else\n",
    "                \"💧 splash — fell in a hole\" if terminated else\n",
    "                \"⏰ time's up — wandered too long\")\n",
    "        draw_dash(f\"🎥 episode {ep + 1:,} finished\", ep, eps, live=True, note=note)\n",
    "        time.sleep(0 if SMOKE else (1.3 if ep < 3 else 0.8))\n",
    "    elif (ep + 1) % FF_EVERY == 0:\n",
    "        draw_dash(\"⏩ fast-forwarding through training…\", ep, eps,\n",
    "                  note=\"watch the map turn green — knowledge spreading backwards from the gift 🎁\")\n",
    "        time.sleep(0 if SMOKE else 0.02)\n",
    "\n",
    "final_rate = 100 * np.mean(recent)\n",
    "draw_dash(\"🏁 TRAINING COMPLETE — the cheat sheet wrote itself\", EPISODES - 1, eps,\n",
    "          note=f\"ε ended at {100 * eps:.0f}% · success over the last 100 episodes: {final_rate:.0f}%\")\n",
    "print(f\"🏁 Watched {EPISODES:,} episodes: {wins:,} wins · last-100 success {final_rate:.0f}% \"\n",
    "      f\"(this demo trained its own `q_watch` — your Exercise-4 table `q` is untouched)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d91e27a8",
   "metadata": {},
   "source": [
    "## 7 · Watch it play 🎬\n",
    "\n",
    "Numbers are convincing; a replay is delightful. We run **one greedy episode** (no exploration — pure exploitation of the table), capture every frame, and stitch them into a GIF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c41ea00",
   "metadata": {},
   "outputs": [],
   "source": [
    "import imageio.v2 as imageio\n",
    "from IPython.display import Image as IPImage, display\n",
    "\n",
    "state, _ = env.reset(seed=SEED)\n",
    "frames = [env.render()]\n",
    "for _ in range(30):\n",
    "    action = int(np.argmax(q[state]))          # greedy: always the best move\n",
    "    state, reward, terminated, truncated, _ = env.step(action)\n",
    "    frames.append(env.render())\n",
    "    if terminated or truncated:\n",
    "        break\n",
    "env.close()\n",
    "\n",
    "gif_path = REPO / \"data\" / \"frozenlake_agent.gif\"\n",
    "imageio.mimsave(str(gif_path), frames, fps=3, loop=0)\n",
    "print(f\"Saved {len(frames)} frames → {gif_path}\")\n",
    "display(IPImage(filename=str(gif_path)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b67f7d54",
   "metadata": {},
   "source": [
    "## 8 · CartPole — when the cheat sheet explodes 🎪\n",
    "\n",
    "The Q-table trick has a fatal limit. It works because FrozenLake has exactly **16 states** — you can list them in a table. Now meet **CartPole**: balance a pole upright by sliding a cart left or right. Its state is **four continuous numbers**:\n",
    "\n",
    "`[cart position, cart velocity, pole angle, pole angular velocity]`\n",
    "\n",
    "There are infinitely many possible states — you can't make a table with a row for every real number. So we **replace the table with a function that estimates Q-values**, and the best function approximator we know is a **neural network**. Feed it the 4 numbers, it outputs a Q-value for each action. That's a **Deep Q-Network (DQN)**.\n",
    "\n",
    "Notice what just happened: **Day 5's neural networks have met today's rewards.** The course loops closed — the brains you built to *see* can also learn to *act*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63ee2912",
   "metadata": {},
   "outputs": [],
   "source": [
    "env2 = gym.make(\"CartPole-v1\", render_mode=\"rgb_array\")\n",
    "obs, _ = env2.reset(seed=SEED)\n",
    "print(\"A CartPole state is 4 continuous numbers:\")\n",
    "print(\"  [cart position, cart velocity, pole angle, pole angular velocity]\")\n",
    "print(\"  example:\", np.round(obs, 3))\n",
    "print(\"Actions: 0 = push cart LEFT, 1 = push cart RIGHT\")\n",
    "print(\"Reward: +1 for every step the pole stays up (500 = a perfect episode)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddff587a",
   "metadata": {},
   "source": [
    "### 🧪 Exercise 7 — the CartPole baseline\n",
    "\n",
    "Same idea as Exercise 1: measure how a **random** agent does before we train anything. Run 20 episodes of random actions and report the **average number of steps** the pole stays up.\n",
    "\n",
    "Expected outcome: only about **~20–25 steps** — a random cart drops the pole almost immediately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b525a0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "N_CARTPOLE = 20\n",
    "baseline_env = gym.make(\"CartPole-v1\")\n",
    "baseline_env.action_space.seed(SEED)\n",
    "lengths = []\n",
    "for _ in range(N_CARTPOLE):\n",
    "    obs, _ = baseline_env.reset()\n",
    "    done, steps = False, 0\n",
    "    # ==================== YOUR CODE HERE ====================\n",
    "    ### HINT: count steps until the episode is terminated or truncated\n",
    "    ...\n",
    "    lengths.append(steps)\n",
    "baseline_env.close()\n",
    "print(f\"🎲 Random agent balanced the pole for {np.mean(lengths):.1f} steps on average (500 = perfect)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d955cf52",
   "metadata": {},
   "source": [
    "## 9 · Train a Deep Q-Network 🧠\n",
    "\n",
    "Writing a solid DQN from scratch (replay buffers, target networks, the training loop) is a lot of careful code. Real labs don't reinvent it — they use a trusted library. We'll use **stable-baselines3** with a recipe tuned by the community for CartPole.\n",
    "\n",
    "We train in **4 chunks** and evaluate after each one, so you can *watch* the mean reward climb from ~10 toward the maximum of 500. On a GPU the full run takes a few minutes; in smoke-test mode it's just a quick sanity pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f9e1c90",
   "metadata": {},
   "outputs": [],
   "source": [
    "from stable_baselines3 import DQN\n",
    "\n",
    "eval_env = gym.make(\"CartPole-v1\")\n",
    "\n",
    "def evaluate(model, n_episodes=5):\n",
    "    \"\"\"Average total reward over a few *greedy* (deterministic) episodes.\"\"\"\n",
    "    returns = []\n",
    "    for _ in range(n_episodes):\n",
    "        obs, _ = eval_env.reset()\n",
    "        done, total = False, 0.0\n",
    "        while not done:\n",
    "            action, _ = model.predict(obs, deterministic=True)\n",
    "            obs, reward, terminated, truncated, _ = eval_env.step(int(action))\n",
    "            total += reward\n",
    "            done = terminated or truncated\n",
    "        returns.append(total)\n",
    "    return float(np.mean(returns))\n",
    "\n",
    "# Hyperparameters from the stable-baselines3 rl-zoo recipe for CartPole:\n",
    "model = DQN(\"MlpPolicy\", env2, learning_rate=2.3e-3, batch_size=64,\n",
    "            buffer_size=100_000, learning_starts=1_000, gamma=0.99,\n",
    "            target_update_interval=10, train_freq=256, gradient_steps=128,\n",
    "            exploration_fraction=0.16, exploration_final_eps=0.04,\n",
    "            policy_kwargs=dict(net_arch=[256, 256]), seed=SEED,\n",
    "            device=DEVICE, verbose=0)\n",
    "\n",
    "chunk = max(1, TIMESTEPS // 4)\n",
    "print(f\"Training the DQN for ~{TIMESTEPS} steps in 4 chunks (watch the reward climb)...\")\n",
    "for i in range(4):\n",
    "    model.learn(total_timesteps=chunk, reset_num_timesteps=False, progress_bar=False)\n",
    "    mean_reward = evaluate(model)\n",
    "    print(f\"  chunk {i+1}/4  (~{(i+1)*chunk:>6} steps)  mean reward = {mean_reward:6.1f}  (500 = perfect)\")\n",
    "\n",
    "print(\"🏁 DQN training complete!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a19776a",
   "metadata": {},
   "source": [
    "## 10 · Before vs. after — the wow moment ✨\n",
    "\n",
    "Time for a **live head-to-head match**: same game, two players, **side by side**. On the left, an untrained agent mashing random buttons; on the right, **your trained DQN**. They play 3 episodes **at the same time, from the exact same starting state**, so you can compare every single decision in real time:\n",
    "\n",
    "- while **both** are alive the match runs in 🐌 **slow motion** — compare the ⬅️/➡️ choices each brain makes and watch each pole angle (it turns red near the fatal **±12°**);\n",
    "- the moment one loses, its panel **freezes on the crash** with a verdict (💥 *and why* — pole past ±12°, or cart off the edge) while the survivor plays on in ⏩ **fast-forward**;\n",
    "- between episodes there's a short countdown, and a **results table** fills in as the match unfolds, ending with a final scoreboard.\n",
    "\n",
    "At the end you get a **🔁 Replay button** to run the whole match again — the random agent plays different moves every time. (Each agent's last episode is also quietly saved as a GIF in `data/`.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4eb9f4ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ══════════════════════════════════════════════════════════════════════\n",
    "#  THE MATCH 🥊 — both agents play the SAME episodes side by side, live.\n",
    "#  Slow motion while both survive, freeze + verdict when one crashes,\n",
    "#  a results table that fills in, and a 🔁 Replay button at the end.\n",
    "# ══════════════════════════════════════════════════════════════════════\n",
    "import io, time, base64\n",
    "import imageio.v2 as imageio\n",
    "from PIL import Image as PILImage\n",
    "from IPython.display import HTML, display\n",
    "\n",
    "N_EPISODES = 1 if SMOKE else 3        # episodes in the match\n",
    "PAUSE      = 0.0 if SMOKE else 1.0    # base pause for freezes & countdowns (s)\n",
    "MAX_STEPS  = 500                      # CartPole's own limit — 500 = a perfect game\n",
    "SLOW = (1, 0.08)   # (draw every Nth frame, delay) while BOTH are still alive\n",
    "FAST = (2, 0.02)   # ...and once somebody has crashed (fast-forward)\n",
    "\n",
    "# CartPole's actual losing conditions — watch for them in the live panels!\n",
    "POLE_LIMIT_DEG = 12.0    # game over if the pole tips more than ±12° ...\n",
    "CART_LIMIT_X   = 2.4     # ... or the cart slides off the screen edge (±2.4)\n",
    "\n",
    "def frame_b64(frame):\n",
    "    \"\"\"Rendered numpy image → base64 PNG we can embed in live HTML.\"\"\"\n",
    "    buf = io.BytesIO()\n",
    "    PILImage.fromarray(frame).save(buf, format=\"PNG\")\n",
    "    return base64.b64encode(buf.getvalue()).decode()\n",
    "\n",
    "def verdicts(terminated, obs, steps):\n",
    "    \"\"\"(long text for the player card, short text for the results table).\"\"\"\n",
    "    if not terminated:\n",
    "        return (f\"🏆 PERFECT — survived all {steps} steps, the pole never fell!\",\n",
    "                f\"🏆 {steps} steps\")\n",
    "    if abs(np.degrees(obs[2])) > POLE_LIMIT_DEG:\n",
    "        return (f\"💥 GAME OVER — pole tipped past ±{POLE_LIMIT_DEG:.0f}° after {steps} steps\",\n",
    "                f\"💥 pole fell · {steps} steps\")\n",
    "    return (f\"💥 GAME OVER — cart slid off the screen after {steps} steps\",\n",
    "            f\"💥 cart off edge · {steps} steps\")\n",
    "\n",
    "class Player:\n",
    "    \"\"\"One contestant: its brain, its own copy of the game, and its live stats.\"\"\"\n",
    "    def __init__(self, name, brain, color):\n",
    "        self.name, self.brain, self.color = name, brain, color\n",
    "        self.env = gym.make(\"CartPole-v1\", render_mode=\"rgb_array\")\n",
    "        self.env.action_space.seed(SEED)\n",
    "        self.scores = []\n",
    "\n",
    "    def begin_episode(self, ep):\n",
    "        self.obs, _ = self.env.reset(seed=SEED + ep)   # both players get the SAME start!\n",
    "        self.done, self.steps, self.action = False, 0, 1\n",
    "        self.verdict = self.short = \"\"\n",
    "        self.frames = []                               # the GIF keeps the last episode\n",
    "        self.snap()\n",
    "\n",
    "    def snap(self):\n",
    "        \"\"\"Capture the current screen for the live panel + the GIF.\"\"\"\n",
    "        frame = self.env.render()\n",
    "        self.frames.append(frame)\n",
    "        self.b64 = frame_b64(frame)\n",
    "\n",
    "    def play_step(self):\n",
    "        if self.done:\n",
    "            return                                        # stays frozen on its crash frame\n",
    "        if self.brain is None:\n",
    "            self.action = self.env.action_space.sample()  # random button-mashing\n",
    "        else:\n",
    "            a, _ = self.brain.predict(self.obs, deterministic=True)\n",
    "            self.action = int(a)\n",
    "        self.obs, _r, terminated, truncated, _ = self.env.step(self.action)\n",
    "        self.steps += 1\n",
    "        if terminated or truncated:\n",
    "            self.done = True\n",
    "            self.scores.append(self.steps)\n",
    "            self.verdict, self.short = verdicts(terminated, self.obs, self.steps)\n",
    "            self.snap()                                   # keep the crash frame on screen\n",
    "\n",
    "def card_html(p):\n",
    "    \"\"\"One player's panel: name, live stats (or verdict), progress bar, frame.\"\"\"\n",
    "    if p.done:\n",
    "        col = \"#2ca02c\" if p.short.startswith(\"🏆\") else \"#d62728\"\n",
    "        status = f\"<b style='color:{col}'>{p.verdict}</b>\"\n",
    "    else:\n",
    "        deg = np.degrees(p.obs[2])\n",
    "        col = \"#d62728\" if abs(deg) > 0.75 * POLE_LIMIT_DEG else \"#2ca02c\"\n",
    "        push = \"⬅️ pushing LEFT\" if p.action == 0 else \"➡️ pushing RIGHT\"\n",
    "        status = (f\"<div style='white-space:nowrap'>step <b>{p.steps:3d}</b>/{MAX_STEPS} · {push}</div>\"\n",
    "                  f\"<div style='white-space:nowrap'>pole angle <b style='color:{col}'>{deg:+5.1f}°</b> (±{POLE_LIMIT_DEG:.0f}° = fall) · \"\n",
    "                  f\"cart x <b>{p.obs[0]:+.2f}</b> (±{CART_LIMIT_X} = edge)</div>\")\n",
    "    bar = min(100.0, 100.0 * p.steps / MAX_STEPS)\n",
    "    return (f\"<div style='flex:1;min-width:490px;max-width:560px'>\"\n",
    "            f\"<div style='font-size:15px'><b>{p.name}</b></div>\"\n",
    "            f\"<div style='min-height:46px'>{status}</div>\"\n",
    "            f\"<div style='background:#e0e0e0;height:9px;border-radius:5px'>\"\n",
    "            f\"<div style='background:{p.color};width:{bar:.1f}%;height:9px;border-radius:5px'></div></div>\"\n",
    "            f\"<img src='data:image/png;base64,{p.b64}' style='width:100%;margin-top:5px;border:1px solid #ccc;border-radius:4px'/>\"\n",
    "            f\"</div>\")\n",
    "\n",
    "def history_html(history):\n",
    "    \"\"\"The episode-by-episode results table (fills in as the match unfolds).\"\"\"\n",
    "    if not history:\n",
    "        return \"\"\n",
    "    rows = \"\".join(\n",
    "        f\"<tr><td style='padding:1px 12px 1px 0'><b>episode {ep}</b></td>\"\n",
    "        f\"<td style='padding:1px 12px'>{a}</td><td style='padding:1px 12px'>{b}</td></tr>\"\n",
    "        for ep, a, b in history)\n",
    "    return (f\"<table style='margin-top:8px;border-collapse:collapse'>\"\n",
    "            f\"<tr><td></td><td style='padding:1px 12px'><b>🎲 Random</b></td>\"\n",
    "            f\"<td style='padding:1px 12px'><b>🧠 DQN</b></td></tr>{rows}</table>\")\n",
    "\n",
    "def draw(banner, history, footer=\"\"):\n",
    "    \"\"\"Redraw the whole match panel in place (banner, both cards, results, footer).\"\"\"\n",
    "    cards = \"\".join(card_html(p) for p in players)\n",
    "    screen.update(HTML(\n",
    "        f\"<div style='font-family:monospace;font-size:13px;line-height:1.5'>\"\n",
    "        f\"<div style='font-size:15px'><b>{banner}</b></div>\"\n",
    "        f\"<div style='display:flex;gap:10px;margin-top:6px;overflow-x:auto'>{cards}</div>\"\n",
    "        f\"{history_html(history)}{footer}</div>\"))\n",
    "\n",
    "def run_match():\n",
    "    \"\"\"Play the full match (all episodes) — also re-run by the 🔁 Replay button.\"\"\"\n",
    "    history = []\n",
    "    for p in players:\n",
    "        p.scores = []                                     # fresh scoreboard each match\n",
    "\n",
    "    for ep in range(1, N_EPISODES + 1):\n",
    "        for p in players:\n",
    "            p.begin_episode(ep)\n",
    "        draw(f\"🥊 Episode {ep} of {N_EPISODES} — same start, two brains. GO!\", history)\n",
    "        time.sleep(PAUSE * 0.8)\n",
    "\n",
    "        for step in range(1, MAX_STEPS + 1):\n",
    "            for p in players:\n",
    "                p.play_step()                             # both act on the SAME tick\n",
    "            everyone_done = all(p.done for p in players)\n",
    "            both_alive = all(not p.done for p in players)\n",
    "            fe, dl = SLOW if both_alive else FAST         # slow-mo → fast-forward\n",
    "            if step % fe == 0 or everyone_done:\n",
    "                for p in players:\n",
    "                    if not p.done:\n",
    "                        p.snap()\n",
    "                phase = \"🐌 slow motion — both alive!\" if both_alive else \"⏩ fast-forward\"\n",
    "                draw(f\"🥊 Episode {ep} of {N_EPISODES} · {phase}\", history)\n",
    "                time.sleep(0.0 if SMOKE else dl)\n",
    "            if everyone_done:\n",
    "                break\n",
    "\n",
    "        for p in players:                                 # safety net (never hit at 500)\n",
    "            if not p.done:\n",
    "                p.done = True\n",
    "                p.scores.append(p.steps)\n",
    "                p.verdict, p.short = verdicts(False, p.obs, p.steps)\n",
    "        history.append((ep, players[0].short, players[1].short))\n",
    "        draw(f\"🏁 Episode {ep} finished — read the verdicts!\", history)\n",
    "        time.sleep(PAUSE * 1.6)                           # let the result sink in\n",
    "        if ep < N_EPISODES:\n",
    "            for t in (3, 2, 1):\n",
    "                draw(f\"⏳ New episode starting in {t}…\", history)\n",
    "                time.sleep(PAUSE * 0.7)\n",
    "\n",
    "    ravg, davg = (float(np.mean(p.scores)) for p in players)\n",
    "    footer = (f\"<div style='margin-top:10px;border-top:2px solid #888;padding-top:6px'>\"\n",
    "              f\"<b>🏆 FINAL SCOREBOARD — steps the pole stayed up ({MAX_STEPS} = perfect)</b><br>\"\n",
    "              f\"🎲 Random agent : best <b>{max(players[0].scores)}</b> · average <b>{ravg:.1f}</b><br>\"\n",
    "              f\"🧠 Trained DQN : best <b>{max(players[1].scores)}</b> · average <b>{davg:.1f}</b><br>\"\n",
    "              f\"→ the trained brain lasted <b>{davg / ravg:.0f}×</b> longer — \"\n",
    "              f\"same game, same physics, only the brain changed.</div>\")\n",
    "    draw(\"🏆 MATCH OVER — final results:\", history, footer)\n",
    "    imageio.mimsave(str(RAND_GIF), players[0].frames, fps=25, loop=0)\n",
    "    imageio.mimsave(str(DQN_GIF),  players[1].frames, fps=25, loop=0)\n",
    "\n",
    "# ── set up the two contestants and play the match ──────────────────────\n",
    "players = [Player(\"🎲 Random agent\", None,  \"#ff7f0e\"),\n",
    "           Player(\"🧠 Trained DQN\",  model, \"#4c72b0\")]\n",
    "RAND_GIF = REPO / \"data\" / \"cartpole_random.gif\"   # each agent's last episode is\n",
    "DQN_GIF  = REPO / \"data\" / \"cartpole_trained.gif\"  # quietly saved as a GIF file\n",
    "\n",
    "screen = display(HTML(\"<b>🥊 The match is about to begin…</b>\"), display_id=True)\n",
    "run_match()\n",
    "\n",
    "# ── the 🔁 Replay button (falls back to a hint if widgets are missing) ─\n",
    "try:\n",
    "    import ipywidgets as widgets\n",
    "    replay_btn = widgets.Button(description=\"🔁 REPLAY the match!\", button_style=\"success\",\n",
    "                                layout=widgets.Layout(width=\"240px\", height=\"36px\"))\n",
    "    def on_replay(_):\n",
    "        replay_btn.disabled = True          # no double-clicks mid-match\n",
    "        try:\n",
    "            run_match()                     # re-drives the live panel up above ☝️\n",
    "        finally:\n",
    "            replay_btn.disabled = False\n",
    "    replay_btn.on_click(on_replay)\n",
    "    display(replay_btn)\n",
    "    print(\"☝️ Click to replay the match above — the random agent plays new moves every time!\")\n",
    "except Exception:\n",
    "    print(\"🔁 To watch the match again, just re-run this cell.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dc91fbb",
   "metadata": {},
   "source": [
    "## 🔓 Unlock your Studio module\n",
    "\n",
    "The 🎮 **Game Master** needs two files: the **Q-table** (`.npy`) and the **trained DQN** (`.zip`). Save both into your Studio's models folder to light up the tab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ba6ac40",
   "metadata": {},
   "outputs": [],
   "source": [
    "REQUIRED_FILES = [\"game_master_qtable.npy\", \"game_master_dqn.zip\"]\n",
    "\n",
    "np.save(MODELS_DIR / \"game_master_qtable.npy\", q)\n",
    "model.save(str(MODELS_DIR / \"game_master_dqn.zip\"))   # sb3 keeps the \".zip\" name exactly as given\n",
    "\n",
    "missing = [f for f in REQUIRED_FILES if not (MODELS_DIR / f).exists()]\n",
    "assert not missing, f\"Something didn't save: {missing}\"\n",
    "\n",
    "print(\"🔓 UNLOCKED: 🎮 Game Master!\")\n",
    "for f in REQUIRED_FILES:\n",
    "    kb = (MODELS_DIR / f).stat().st_size / 1024\n",
    "    print(f\"   ✅ {f}  ({kb:.1f} KB)\")\n",
    "print(\"\\nRun your Studio to play against both agents you trained:\")\n",
    "print(\"   python ai_studio/app.py   (from the repo root)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d143b6dd",
   "metadata": {},
   "source": [
    "## 🏁 Checkpoint\n",
    "\n",
    "Today you crossed from **perceiving** to **acting**. You:\n",
    "\n",
    "- Learned the **reinforcement-learning loop** — act, receive a reward and a new state, update your beliefs, repeat.\n",
    "- Trained a **Q-table** on FrozenLake: 64 numbers that grew from all-zeros into a flawless instinct for the pond.\n",
    "- Wrote the **Q-learning update** — the single line that turns raw experience into knowledge.\n",
    "- Trained a **Deep Q-Network** on CartPole, where a neural network *replaces* the table — Day 5's brains meeting today's rewards.\n",
    "- Unlocked the 🎮 **Game Master** in your Studio.\n",
    "\n",
    "That's **seven modules built**. And here's the quiet truth about every one of them — the penguin classifier, the digit reader, the language models, today's agents: none were *programmed* with the answer. Each started knowing nothing and got better, from data or from rewards. Every capability in your Studio was **trained**.\n",
    "\n",
    "**Tomorrow is Demo Day.** You'll assemble the entire AI Studio, rehearse your demo, and then run the **Grand Benchmark** — where we uncover the single mathematical operation humming underneath all seven modules, and discover why making it *fast* is a whole adventure of its own. See you there. 🚀"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2ca4cdf",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div align=\"center\" style=\"background:#14161E;border-radius:12px;padding:18px 12px 14px;margin-top:6px\">\n",
    "  <a href=\"https://ahalimi.com\" target=\"_blank\">\n",
    "    <img src=\"data:image/webp;base64,UklGRoIqAABXRUJQVlA4WAoAAAAQAAAA5gAAWQAAQUxQSKYTAAAB/yckSPD/eGtEpO4TjtpIkqSI2v1F8Sc8N4OI/k8At3/I//6Tj6f3BqhueU5ce/Oq9iC/k11117+ixBmggCowS0wzwOxAAFWHaKfZAwNzmOJmctNzUHdXrZiivM7bEA+hDdu2ZXNa7Tiv637H44K7U1ghxd3d3akbXku3NtToqru3q7i7FHcLaXFL0KQ4xSU6M+/73NeP53neSYBv8v2MiAng/6EytWjXycESqBzQytuvMaFLg+8++3wCXEseBrbtzx8aCGqf/POeveBLGIKuz02PiP6Zz740TyOWXnm1iHjy66ORLUmoYOcHI+ZdePhqHYOUtdR2pz4T8dxnCC05iN5dN4zXf7wmgAmIDIz6/GMRV67AEuT4TxJ/XgHMTUFZuEPv19+PSSstOYyZyEvXgBtDlMOkaUwYi5YEAnp45WncWIRKdJ8e/Xvhwz/jvUEWvoNYxA6/ig82xoZ7sv++wLJdBItcxnkxe4LZMM+5hWWXodYM8OTuKsndPVnf/XEWPrwzrTln9GRUs4jFOu/knfDhHeexJ7Vi8lrQ2P+oww7//MpILPPZA/c/6KBO58S4120450we3H3Nwiokf/ke3A9eGHHmBJM08scRcZSbdT0du+LDujPi2yEqnY1z/1ok/tR6vQ+jfG9xPcI5Lq7Fhm9i/Ft5a3lV4hcRU3Gb44MNBeDxrs03R4x7a3ANbNjm7BtPJKlCdDwZ+RFvZccVAJGdwME4P44jDdsSP82/IlHp7DD7F1FsWZCQqA5yKenYfDk2bDOuiwPxqsRpF42aH3+DBm0GUXI2jZd60DBNpKdiTayK9OyhXByvTSBBSinAco0Y9WZrNWzYNvqdd0agCmfHN0Zrxxz7qkHzjVaT8iCUEP+KzfBhmrH04EzaOP9adXQ/no8OY8wZv/9dQYgNsQrjithlGLdsMR1RFt1vHgmcEpst9WakDbbYojBn/giocM6MfYZtYIoaZ9eYNf2Mg/ZdEF/ZLmYb1dftkalOiOF7q9lHtXP+TdtMuWDmO5EfH/nf17plEJ5BFcFE5g/bggXzJ3QhQHS/shfABudFTP7pnAYKyNmpVPbleJsYpsH8NyZOxADXPm+PsIaLvtfyb9Z6oQcBQRdRwfiV57w6bAsvZvh6eIlLLseBBmfGy71/Wr6mk0qxTu9DbzNsF9NiGxKInjf2Vcm1d8SBq4+vAKvbKqZHag3XMtO0V0PCddg7vQgQfc/na6hUSESp0J66leF7MOPpT2wlx7njeoyy67T8wSpmAPJGIPC86Sbv/otiGCNPVZEXCc534jzcWav4MamKP0ZMwQFj+fdjGsg5O87GGR67LQpjmXcXrIOtPDMeWUUCjH0HI+YdiJCNviIW5l90GuvMKzYfzogVv/TF6OixWPDcdLBFgHNKXIutdchmO69KSWy995ab7Lw9IMbuMnmDrffpMa6LyzCGsda1b9bM07p71t5s5F9WwxaBbOSTcTyL2Tkx5q2HD2eA27e5+XdAzzfjnX0xQCDV4ezYWrAVneaqMXdzq5CbWSdbz4/v4AxrzW7yEcmLWPDDr8W8Sbg5Em07P4gXVyexyBPrvhw3JdcwQG2oQhVuJW4iWl5gskfjOgEJJlo7crsqZq5KY1El1p4ds5bFqDVrL9qKmmgrPmwCBAhQVUAAQbA4jRsJgJyYmheu2rnx1OvHXTdwjUk1SKPuiue3xGxRmLPlS/HK+hgfXYEAAWojSlEK1Nbilrl7crPU6Gh4SuYdHfg791GpEsG9W3WuM/uz42edddMTV5to0xh1Xcw/FtyGYg5fmMs8ELXG5geFFc1mq2gVrWazOdgabBVFqyiKVlEUuSiazVZRFDlyGQg+zI3uzq6uRmdnZ3d3b193Z2fq6Oju6elqePKUGt5w95RcZp7MTFQ+Pgm1995IX+nB30xtfouG0RZGx18irtoAlExRISXBOudHvPQO7SZ+dAqLN2cZ5YiIXOSistVqNRf2Dww0m62iaBWDzebChQP9A4P9/QsXDg4MtqrUNbK3u6e7u7e7u6e7q6u3u6O3u7OrM3V0JDczc3OXycxYpFEBnUFL3fH+SGPIBkf+Jwb/tiGADIgMrPObubHg1Ddp2/n+D+oico4cBBGRyZEjRw6XyUzmbgKUc7SK3BpslZvFYHOw2WwO9A9EkQMZ0WpFVCxWl5l7Ssk8eaPRSKnRSJ5SZ6PxMkQ7FqvBi5Be7sU0FGRM/OnciLu/ucmIuZS71jru+oGIyzaEaEsst0nDilaz1SyKVtEqiogcubLIkXOOCCAg+CgKAWorCIIPZ64gNov5s8n2Sh8CGwI4rPmbNyLiledem1P0TVx9jYjihl3Bg4+4SgJUJ4iKKAVEG4tcbahGbSiKqihZ9G7PdRB6uQchhi6HiYee9nRQ/f6tUyeDGUOWtRNDiEUTUfr4l2Wyp5ZafIc3fqcwXuiRYHnQEMAc6Fp/rWW7vfnOC08DcobJKhib5xcD0P0jXjyRVPQ3noHElA/+vghA7rSpZAyXZaOODTbbt3vkeruM+9UyuKUTI/YenbgiHpM0NEAmlSIYRosVD987Onq8/9+fewgMeg474ojPrAcrTVkfY8nUxRKs3KsiqPTkLlCyJYsPsSR9jEj6KEltySEA4TUyQSCIyG2YR42rwtxdQ7JkAJZMVSZQldzM2hJqywQBqB13ym5tGIpcJSNyyRQlKQ9JRFvtq6J9WVElPoxyoGfsKKdeQb1YvAqGbAaMW34MYKpZ3FJ74m8/RzWK/Xfot75GYuC5TUEg9tmlSWdnLprvz7zveRBgbHrOFdZIEOZXrowZnzz34ouu+N+xUhsGq33/zn9Pe+DJO6eu3wFY3vzkzlbnb8dI0tgfX3vV1VduilU5B90/ElUpdjhW9HRkCH++D4HB5mc/fO8d9z15xsZggHTU/hR+zpWWoee4jQbiN49ZHLV/qzGiw/Tff90BasdYI5rLy6qw2ybtzP8+2z1i8ylx0/IY+O0TD+aaOzqK71/+RtxxOBhkf+D2/Xnglx2Furbab1WU/bHphxz6yX+8T9QZE/5U3Hf0KmOX2+aXH8S5iGz33bTvgVsfLQV675LJ++5z9cOWq8ymxRdIVeF3z/oWt379ayee9PXbYkUMY6UL4vodxnYtdeiMOGciBuLya7bn9Ju8APrPumKrvWdY2OUXHMxZj3aO3O1Xcd4IqQ3npxHfI1WFzTnrqHx+zJ378KzXY3qniTTvwks44yqA3pMjblk+G0S+9u0x91xKeco4IBcXvVlc9qIFtcYaT8aPqR5732yAyBe9WlyHFxDp+cuLWec0g0pnh2g96qqC1hUFn6by75tgxrovxneo1Hnx2LIY2Px//orTFziAvfWPKdcCWnDuxZx2L3DS7+JcrE50z5yb/9MnVQAJjX7Xwh/8dSt2xnEiMTqlVnLYYW48PgYjaOA9nloo/n4/GbCWI+rNxs+M8/AkyTpY+d4uBPigpyIoK7zooNY596Vm7IjX4M7XU1fyhlZaF7dRT8TlJJeU6Ho8bk0mIDu1oQXzLYJAztLJc+v3j7aKdbGapKNv/2rEkaQ6gxaE/PNRfJaEkyFarRaog6/m+H4WgBFFUSDmvEgAGBFtOKfFOyvIqTSOGUulY4UqwrGoESs9uvX8uKgdg2+QQAKcU2NwEznlxGdzfJoEGKiKaEUWCAdveeFxf8SeeI0x7Ut9H+R7pDrABBo4tRVbYjiAUSnrfqH49LKFqDQ3FUjUinrjE4PFOTi1QgQgIaO6gVST+NFZ3JPnrITVAN9UQwZCjHy1eECiUpr4br5fAgRRQxRUBjig3J2LDbEq07r/GcdZ0doMrwkxl0xz+Vfi/2RUqYrEX2PUNjjkTCsPwFIpqIxoK/HdiM8pVamSsuinKLVoQo3UMWMzHR0xlVST4XiAiROQs0vE3/AqxF0xsDYGTrsRVYAnz83ereJeF9XOaX+GrXOc3YZgd0uNdXd66wRcJYNUpy9GTKYsVtq+d3DcN76NVcjwNpyrIrbDq9oNWGW9zgzZB1fHqHYOvx3rfi7P7pGqAr6/1HIT1r1rEyxxcsQUUk3i7xH74SBQnVGO0nstGHfGKq/+D1Ylxj69mnd0PJzfWxarSsThE2T9vPapMdTVJw4NjQFhiUlf6+5f+RNzqDbRrnFr5MlY1egxHd0jlhWAsd7+zZxJ2Se3dfU+9NipEYeQqpx4/cGHpr8fa5ROjTihrZ9HPppEOeoAURkHPtq74h5LX7oaRnXi2FkTgC9HfLMd+/wT4Js9Fv/ZBMNROyAYoFzwzxOBP4ytEURbN0TeuEpa9+pZs577LAIyt1J7OUWVseZ7O3fDsnPzPWZVhqaOXXr82jdPLn0r4htt/TriIBysLQWAgHz2z9+K+DYYtcYdN1x55Q/23PjtPLNRqALoM28V945/IZ7qlYaiifBsSaJhqShO6SYvCudPOQ5VKmGs14y/YAEEhZIkNdQgosL587QL7jnrc2tcl4tN8BpOAFh1GZQ4JOJP8hrnoig2LWXwukZRKseOrPZm67XVrc71yVneud0p1z44mGPPsIoMOedQgz8XsQvutCBqgm0i30uAIHIuNAdQiXYkZ7fIv6dK6nm+OEGiUjhAIqgWo2evxMpH/u2OVyP+WhPwNTVkAMZKC/LdqIbGM3l2F4IWjKqSxgyikogDEl+KuNVcNZzxRwwYf9CCfDVRUUAGIvspUXya5GTqfXCFtbjrUSsAYUAIRZQC5SqB1PlYvDTCrIJRL8Y3abOmICoSJ9xCAtI6D+Y3xkmlAk4kgQQY18bc1cwqXNs141Sc4DVYXSoRa71RFcRedOj6iK+TKqSxL6yNWzK4NAbWDy+1CITk/KWIHXAnwMwMaYDfMTAlQxAZmQwsll0XgwjLVkqstiqJHXL8mIaVbNTzrSlVmVCVCqpNjxwol7txeMRJeAn4SqnSmLwwfkOHAEtcHzNHmQjuH8w7hyzwzP5TCQhE7E5Dy71ezFsbLyW+dh0OkGznweLzoZLnPD8K8sDSL8X0DuFEwcI8CBFdf2Tulx+0DDSj6M9NoMWvJyCUvZhDi/LZqyHjuIijwNyNrtfjGxXZiiZVNIvcFDjbz0gSYDZiVnOGFQK8KD7bBsYxOe8EnhxOijcnYZDt+Qttx92bQIs/x2XKIChaO+LOQZGnGQZi+QV/JZXExP44fnLLyOxmaYNpFn2bvRUPrIrh7Oisu04j9y697YHc/GM8g7HNOP5nw9yMjhVuO+BBssV2sE5hhaVd734EgXPwK/HXFQHWvzGuWUEFYpsVfKtRCETfTr7WxiEV42ZNd6Ns3BnxvSjt6n6evA7ngFfiuA5g3O/ivsk4QOiE27qv/fZl/+3ZdMonth20DOH7eJyKk/hbxBVjEHzi9Buv2AGVGl+fdve/p24YFset9GAcuyDo6jjtkE6Esfum/2LrP0H/a//80WngBSivd+RdmZ8RdPQt+9v5Up70qXvyEV9vWRR3P/4VHHAmTn3+ndvPOOfu2ZdvB4RirS/dfuPLvxktSSO++87Nd3xjUjD+x7dcdwwCjG2vu+rym44xK7bY4OZb7/5yUh3OxB8+P+PCv142+4kvNXAqmLvHyTN+wrvevGHSy8qg/LkVb7v91t2RrPsfd939j/VQdCXopFJdQOoBRgjwlARgAJ20LaOyQwy1s8nQHZh8zLFf2mMsSAANAV0OYB2Ad4ILGlQ3AHoECaB3QbSBF6SN1+qc88QMsFwBClh9tb63H5mDZcpd/UAPtR1dlE3US2KISi6qpQrzZLQpiCoFZUkVkqpQotqdWjNRKxOVJuplLsoyBUOUFVR6DuplBWWPTLUQlTIZi1M1H5MyT+7iY1ISRGaIkiAH/39ci0k1qtGwTWjxoAoxTFedcBavoRKmYYfKFdIQAkTjzgNx4x8n4iXnsy/vjpecb56HVYjO23fAEcs8tiuOs+GjG+EY608fjwBVf3QkDUkQH7LF3dhyXEpRrLcyAqSRz915XzIBYrVJ1GuziYBYI14ZbWb2l0l9gBixWVfFcFQSEFlRynP+2+qHD1qUEye+1PfeERjlBQvbiHlBeY135v6dDo5de143EAzODYCOnpACQRAEhIgAEQQRQY4gSgFBLLa9fzDgwkqBSgpQhUpSSYBqVIoQmBAySREMdpz2C8+o+fn7e2hO+ltJarz2U/74DFER1gbJK1aZ/+k3dmWp753wywaVBqTWCT9qJlEZERBAREiKKOXIOSJCRClyBAGBRAQRRFTEYOeZf3A+fSaLOoIcKIpciiACAgiBRI4AAolAQNEYSzlGLjW6mJdUco6Y082Y+XtnL+XcTu1yscFxr+rSf10wor9CovzZnw6kEASIshQKEVRKSFIgKiMllZFAQog2V0Z0r0hkiIooBSITQECIIQcCI6KCEhWhDygEPb+/Crh9RCnrvrjrvnvibqIU2cyIKEXVMgM7MePGNz7ZF4MVgaCF92SGKCCoFwhRqSh5jdUJCMzU+QYFH8MWo5PnaATgxY75M8d88fAvxxbZAUu5RW3d/F3YMk5+dYVMpYKP40BafPFhCkAjopVy7u4ARePZKynf+VcPIG14Sxro/P1VnlGPVywb28MGnRrs6QRE6vMKfSS0qCL4+A0W7P0oBRz/LgXZj1jGG558hZ9ZpuCvh/Q2Cn+aHAzs8RSZTMduozHgnb0fJJN5ao/XySwpBkM1MQz3HIBHLiVVWKqwFIIiAFxVLipdFfKPJ1ZQOCC2FgAAkFMAnQEq5wBaAD4xFolDIiEhEzo+mCADBKCHABk75Kvzyx/P/l30T3X0Wh0NzV/0vWr/hvtm+QH6M/5PuD/pp/r+oJ5g/5N/Qf+D/c/eD/1v61+4j+xftJ7gH60dYV/ZP+Z7AH80/rvpi/83/jfA5+2X7f/AT/KP7L/xPz6+QD0AP/j6gHYS/yD8Mfcz4M/lPyg/Zz11/HPn/8V+VPLu6i7Xb7DzZ/4Xhz8f/8L1AvXH+Y8Tva1al/lvQC9XPnv+s/v37r+ebqcd9vYA/l/8x/4X9f9tP8H+qPlsfav8l7An8g/qP+6/z35K/Hd/o/4792P837aPzj/Ff93/OfAT/Iv57/sv7x/lP/h/m///9V3sj/bL2Lv1l/TD95UPYEa4GYd/abbNVX6niLpKnvoETsrjiPHCI0/Wc8PrAjg7n8EqVXtHR+RLc2i7lY7+CTVqwOIzc5BHJlkYxvvrp1PLDFN2XvLePuLoZt0rLW7ny3aM7NhScU6XHruudQM3RIc4U0UqF9LZxJyznP7T1gMLBOkT+AcPhqBYnFJqI/taFKX0yroNPsy2Fpkkxv/aMYxdfZZmBZ1BHZ81obGX7T4vWQVvip+jZg9X3Hd29F/udLL20QewFOqp2t/VxG97A568m8ySl+5zsLa5ZcvMOPEoMbiC+r8Kt+W5bx4Cf7ebDhNyICv+u9b3XVNsJoOhSG9kWk6vm2WU4AiGw1gKXdgV/ei3YOTHaCXb4eEuy4UTvPGZ/8w3a13r8dumfz235RMJYcjXg94Pl2C3tP18XbJ7q8YhHgBuV+1zWAmGDcC8XoznK0aVGP/4qMS8ryT6NZ81gsdEHi/AfD69XH2MCPMYd6k/rS+QfPfTS1H0/AwQhZlvoCrvH37Zz0/36f97PWf4+2X5SAAA/vATgBDX98Iw+ABTcICW5/6API/Qk0kJwFpjSafS/Q4+b3nCKiUfLEieBTx8OyypKGtKaLklX6k60AYToicN2+4K1YTQFcbHVuQx3Kgcjjp8Hh9nPMvwN03aLtADfGNtJV8t4irAxRcuILD/z12f5R6NlESmcD22h2lMiLggyioErnZL1i12Wouql2i0pNCbXOcScIfntHmaa7l92hnQDo02eXoO4Drs1mPAMuSivzcgRN4iG/g8T8X0Cu4mVFVyl2ZGHkQ0wPO+Fcjyqc/WrV5a22RDK8oFFD7Y1gfHJI4An5FjehKfXjTd5FbXuRpL6FoEq6BE+6lHd2ScUiip1BqaF6j7o0PTgUulHIV6uyngPo+03zz8hpN8waRusKJX3AOC/3GHy9T8/svkpMuGotcE7w32wZ6DD3KkX+jdC8j9CkOGSiBH38cpRihEEC9CpjploWb4H2iwJ9UnP+rnAHByuvNcc+U/LiBjSEWgpcyFUM37YX5Iqg95iXjQy0dYazh+Al1Jzny3p3iVMmmDeUm2W8jKhRlf10ZPtCAAAVcyxHoE6kSP2J/L2fxdqnT7MOBfjlEVQ0sJBxAxQJFRy/ACI+WPEcAODz/eHtYVwDSymCIn3pswZa8pEHl716Xx1a28jcMOpvG9ruF6JHhrwtIwjX98nb564WtsH5S4NaWJbxBL4KJVPHSPi3WZlDUi+JmJOhGvwUrDVZ+VFW3e369BnVBifU+7gpMh2SZwH4n/pBqxeeqtQa+7DfHK8PSVsK8mN7dOPZVrmXEDnNwQxhmAtfFUn/T4x03v7wEHNL+M71Qn+dlr/nDSfB7db9qR0QWyLcdPorN5qv5VQDvysftQTP+cnT77NE8chxPYrrGIHfmsp8NwSViVln0PnImSVSWZIX4Mk5h9KgcZpH2lVr8pKKRozBL5VHHbsFWhZ/c2iqPUBKb8RDytqaZ7OJiMrBRSiX00t++6goH74wNKiZT4ePyP+Yh/V7LO1rrZFbte/il1GsPPLv5MsqANiy5j9H750prvo8z4beZ0NuHxfxLROMIItU5fIwFw0c9wmnGbqmWnAho71i61UxteJIIcU/e4D74XxVnNe2+7X4lYMqjIi+YUTN36cvpiVcYrinaT2JIWr0r6Vw705Y+1FrT+5BxIsU99YqcFjaeQtp5xR1OWTBoP0gpOy0DkNMtPZnNU+Yzm0VxG75/+RtOAcVJf+KYtDwAdHC0GUS0Lwa1UAdq1Xvh8dEQzNgDs1eCmDg7qQM6w/xp4WjTM+8RR6IPqE0+9c1Xslh7TC7l4GUYpUgS5Mn3PedZVrx8yeBlroZWPah7/CcfyN7gqzkAAG9nFyi0n9XLfeyIrDCmgvKqwqTGiZSXPsgZLxoZL/3NxkBwPKpbak2uVoapL8ZTDcsHlZgt59DaMjg2xiDgyj1vS0Vn8fQ420OyDMglyFFDo2ZLIzgrU2AgzOS0r9ExRG2LaBnavLmZbvNfMqmp4S94JHRHs5lBsamuNH6iwrFJIa0R5AVZvlLqhHBBlbFeBVbIajI7/3lYOAFuvrL22+YqmSm44PVCTq1lvi7U90LIkm0OPfqzUwgwLv8SiHgO63VO0jPd4J92kxLUfA4lJn4Art8iSYnuvh++8/lH6SPy5XL+FG1bW/1p9h8gpBfNohHuAZUq0AVRC4qyGQ1pcepXEdfj/u32jiwgfsOH+CjRkSjWnK5s6fXAWjNd6KBaO/5JM2BCFp/y/JgjXTvb9Mwg0iiIQY/yu8+/B54a2fdv0NhBaJMH7oSedFS27NcrExUXsSz6pARo+4xC/wGig1XsXDoUjpkJReOxNOCPTZ7eSutjXqanB3flcO8Kb73Qgftv3KOnhbRRmgI4FNyyzWLGp3mI9epIwwTREyUd7vgbysyfFPhIO7cJPd+WjI69LuxDiA3uwxvDzaB+oYxwvJOHTCCrhgweLo1C/ClZJeTRDGsErqzKN2GZNQXMFaZOk3OtW9zlgjvgOSh1XdT3Qs91yKAhJ7eYOta+XvNcQTBDQCV6lY9mQWksgXI7CT0mqt+GiMuaR8TgmKtGP1oLIVLciA/7RMWtkyfrctlQjPBUG13mgUdRpiGxva67jfd5fXcUvHp9e8+B7G8As9LHMxHT9dhFm5f3PMj3YL4zzQozi3XnPEJKnnoA/r+OavX+MzdmMfw4HSN0VSXa4mIsy2sf4/W4AwwCH7XH7RSb85mSZMFWOQcL/0BH8HtwZ6mixuMFKSWjk0tjj8kbuZn5FxY38tVHlx1aVDqjd79MEwbh62AjaWPzOZclh7gCxlscmEOL0bhrellqZUtddDw/PGpCIcXDVi41IbkURJgQz7c+Oxkg+dDTakjypzXNz/pQfiIUhWnfJN28yv3/1GLA1Z5owSSrsm+yrkpoZyrwcXDhqQLPLLat341qbP/OIudJ4B3NXV4zcjIKTQfrl8xe7QL0pNXDxta9GbrVr1gAzDjqJEu10+mrYeVxIS2vxV6WSbNG7t9Q/tdxFUu86nj5KQFbTx5yqJA8KPN3utK/B7R1o8E+JMMEUxt2xXmyRMuQDLpu7MuWso4RbLCGhUgJf2zf8D4bOPqjr4k8AyMaW69uyqtL8E3/UEL+FnyUn6QTyUH3VuKuj5I51CREkS/3es/v5P54Fw8+4JT5Fx3U1njx5Skex1dQUrheamL7wylm/gv6Y19zKDz84xXU614rk4GWQItNgQWaXkHNpLCCwn41ZxCbHXYZ9IaL0XE+9Cs5pS9Rh8eR8dRRspECRQFEw756c6qHnu2UxXOh1XpP2kf8EbBib/LFXwLDEaTkdmHLVHWlQygB6hrs9QvWqncCugDInRsPW4rr5d59adLifU5hAQXwm0I/KihsWipg9BgWX8Q6CGhCqlLH5K3QuotivIvxTMnatMGiBBqU0xbkRcJC2wYTeyb+3yetRS/Rv0SbjuJ/H9/68if/oYM79K+jiwSUfHOIDb712dZe8klrm1m/9zISAE6S7XcfZgk8GhomIF/w5fFWGFY09BmfJMcKxjlGuccgIGWfRhlg4Vt2WtVKNXpnQ3AOh40whOVMV8NRBs/+hWOYdc9m53n3khRGTiAff1AGWQ04nNaIhI5/6YwvXVpMawMu7THRWD9Z+Ub5dXPJDVl3G+JnIumQwPSGKfndinL9xZ099ULrFzdpwF1qPQf+26rPy2/JLVs30/ED6fg+W5sZDH4wuq7pPbZ1dF4Bejz+NRMJrCb4PER4o7y0KJ/SMbsOc8TuJOR27bpV9T0foFXSmviDA7ymNLnowSejpY1PWMB9jTnMVYuznkzAKSwNabkzzH8aMQ4m3xyis2D+c75TVBA4zvWjiHApa0AoZZL/WGfXaCD/OhVAftRdJnZ6NyjINc+NraHciIpNLMBj7Q4Tjl6JQp16Aq1SxVxsq+kJkxktMEQnPfGMrBGvP/3lmIkA4ERgENCQE4UJ0G2Oi0jpjsToifs6NeEKC5ccMSSs4TMTnemXl6mxbx5iAWRs4r9T6r1OTz7WELHlJW3e7nh5Zg4vYzH2hU9PX41tRl3hAhf9njzIZ/uJClipzVmFL+bfOg7gyHExSzuYX4L8fq10yS3A1+gDSAcJywdPdLj417Ds7Z45oTRLU3K5yYmjOHv97IrYpH8KA45br/KlyDjOT010aulVc5p+m/OAR2uIpVRRQ/WiG22C/X/3xi2rmgnXc3/K705yRk2zJ6ZX6wb738u8x3wNuKH48JKu3449b74ImUPzEXiiKhPGCuFNCZcoEj2G5kfrm+6ENkyk4HCvBxE8l9IBJEPu6Cbj0tYNGdm+LuQzQ+sH22nBtwF9pXG6H1Q4P3YuSKtKuw/kDvgMGfYuSy9XGLVR/cUMRGJ9BjKLGbiJPdkvduilVnipBzPIk+e+Ru6uLzsVBJPa0LpgTwBF7C5YTuQqaCJElDJcSoE8KJJyiQC+wfetr4EMA+4JPIi1PhK+CfSGMrQHIHohG84FBlsqebrYiF0ocamWlSDbfKmcCTp8cHOyqjLuSrjJM9J6oQ/BLyKntP9kc9kekIa2EsaOwtzsfCb8+NQTfJQ5Z9oSlYZE1Mgf2whgoG8WXfMY+X9HLh8DkubqY7fQhyyJYuuvhki1IC4Hdhu3Gf3IE7gWNpWJLexNEVwuB5F0lDGFpGTV7U7nqmvzBhuf1fwy8R9M+pcK9e13yY4J6f1599ifl0InlQ8fZXUv+KN5r9kRXU3uYFvCAQXfvv8G5Ur3yJ0t9QbQ/246I4N59ifaRUOoN1vb7NaxsrDnDqealFiAkRWtB7I1sqRKXonprWOwc+0pGDgV67CiP6szbbjwQxMaJf9qQcz0a3En43c1Y1JylqC2wrUsnpyCEypZHe4a0ZI1Sbrgy8jjzPHEl9kwm6RxV2aG2kKSlLmnJzFHUvL/gh7sGfx/Y0VczBL62x1fUcu1qmOlHTUS2H1DdNrERXaINUo3D1X7vpvQHAssKG3M66KB2ofvgOYm+dbM7kudnddqlowpMsq1dXQkZ9Plvk59papxyAHOoaPdueivZD/GDphJTP6TYbwe6WMO+F2sKcsbiHUTOv0XdDr/U1vJyOxUMbsK0rOgbF8D9gSYzWeNXOQYioNqIGabtgk7nTDVTn5yQIfTQW5HYKEyKwa8R/BKhPTN2IaugjqVZWtWDo53xM1qTDN7guQgQyuW5Px7d9cB+up9fzmnjC4aYDk/zCJeRAk+dF/LHASrWSz9NxUYpjh+pFEnukKQ5SkmdiJ/8jRUQV25pBq1fWWAqQTthpU4KCBdLIBiUzX30eIsI6AHb44YkJZNAwg8T8gO1qoB/ErXCLJjJsUCyE5qCPcaK979riGE49FhTdClzf0p6AP7HwaNw29U8Zi1aLImmYoKFOKAjnFZBhjezAskTEzY01KnOrT0n70VBO8Z/p/wo+k/tqSAnqgu6ltoGjyC4RIHv8OW+wB/hH5cfzBqv4SRzTsgjIYo9/C0Az8Z+5AUaBfRmBStj+/deInNA1uHxdw4MW/EjHC4uQjSuk8HdsdqadJNlFR779p8slkZ6khz4tcf/HmsTo0z8tWTL9LA9HmNlUgKaZ3Or7IWoVlLGSZIkXhXcNvToaUJl2eJv3QRFB+bR7vRrgMcla+EZoQH2DvFlEyUevB5HUhV/extLE2UgH+sdylBpUbiNCDZJ1jVe1dF+iYhaQ04AGKH+udCsT4AeQZ2D+iowqLjEcVjk7XX3fTMBm/HJHgGWYSQTNsrI5cOVPY9I0Ux8Hos7YbMTZSttQpgJ1oTg61z7KlYGV6DNgqYwx5X19yKE4ooYiZX0622QR0Qp+bgOgEF19u16f+1jS/WAPuH7ipaIwyTtggq5jRugyd1+bsnyu71SrqtFZDDviJcVShzIO8hU0v8XxKfBjBcHlhMxphJ27iXYID88Bxo/Q5Of0VS3RJXLHIk/Ragm67uo5hpnMFMHSbIK6E/Jugqbp3oboYWAcIX7ZO1eS+FFhD+9s7CMwJY8V75NQE/sEtf6N2Pg0tFlz5znSMHrKgLxtVobmhnu5ie0Q2JzUDE3V8TfePpVpn1zyHZh8qY+luf3pk/Hj5uVGvvKPT7pCuQoDJyILc3P+Gab9t8JUaZObF7J3lGpheFQCioxq2JAuQt/hw89/0ha5df7zjkzx2VCLi1m8sDx6qvIg5IBNWf/WcDltqnB/4zzEHCRQfSu291BlhTlm0CRYphysQx/kMZAlKyEcvDvhBiWP6MQal63KZr4+h7Keac0SYcAXNY8qg/4LqT253kCm0M3OVI0btHYq3C3l4qObplGEjoq5zJyXDOl7b9MvDhOUa1ZpNvBosWFVHCH1AihE9nnyirEt0sDANKVtpsqyPIG0y2bK7RwSQiYJUrygN13RQ73LDq/umtHFr2eQXC0GydMV9jpSm+vKWY5aoE4zRXwyf+kYDI5NX27awP+76F6DC4XhU92joZlOA2l+1aTdV02SEWAByTyItZy2jlrDKDKd7tpTQdsgwowKFJrMbt17cmHt55XlU12xkvLJVtxkJGIEhEvpoZrbmmaHp3xaUKWQK1+AWYo7w0XljKSjVZPJF3v+mRMCTyFmGs2L85gna1f7dThR6NkGC8XOXjljWYuLReJx5TkL6dsiWyJnSxPwIfgNHUJWY4vx7OtXHrmE4WDVufuQMeuDI2md33yJhOGl2KfqccmeYK1eFE1WlNv/YSqXclxJsjeI26zgxTn0i9Ocoi843ftYDfsZzHdkDNJcEYAUNSNuJirSvpl3806LzrY4deTMbavVK+Sx/V7IztCW4tfS2B4D/fW14h3upRkDlh6Jxaa2f6IUM4pBrJIMKTKVYvTZnBD6p8NCqej5Ag6nyKhz1mq9FSBI//kCpyc3Med35j5yks4fHZNwHtQrYYuG0n9cTxi/9v7mL1/2RMDTMh2wB/35vGGY8w+akNn8b/y0UJ+ARD1fbElHWmX+uSaXEeYmfY5IvXwjOfgcfMKeHjfSA8oaXh4Een3AhS8HlMMBsWhq86BWwYAA0NgwOY3E+d6sv8/ZPbY7Qn94gPNEqTvLKPKy7Z+HvVP84M/LZKzR+Lzo28Jcfanex/VEV3VtSSWISJCQ2IKzcMfJ1O4x3lwDmXW+XBvsNQ9nACKNwWXE0CgkAn9F4yvI+NekwAoOLFdjHg1uO87D3e5y1Zi7Zg64yZLLx871+65A+c6/8JBW/PAfOk/p6KocHyRhI5mcIICMS+lzY1u2Usfh9/CWwvnfX4r299vlIhdvjfO6U4I6uhpwiz/zg5NLobAkM4DaPf0ZmMlHMDLjNL1OpVtwp/YWCILRhVroXjMHGalACHoHTL+WC5D0y7WejtU1sOL4GJ+76EJUbLQAAAA\" alt=\"Dr. Abdelghafour HALIMI\" width=\"260\"/>\n",
    "  </a>\n",
    "</div>\n",
    "\n",
    "<div align=\"center\" style=\"text-align:center;margin-top:8px\"><sub>The Lab · AI Studio — Intro to AI · © 2026 <b>Dr. Abdelghafour HALIMI</b> · <a href=\"https://ahalimi.com\">ahalimi.com</a> · All rights reserved.</sub></div>"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "Dr. Abdelghafour HALIMI"
   }
  ],
  "copyright": "© 2026 Dr. Abdelghafour HALIMI · ahalimi.com · All rights reserved",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}