{
"cells": [
{
"cell_type": "markdown",
"id": "1d195ca8",
"metadata": {},
"source": [
"# Day 1 โ The Data Detective ๐\n",
"\n",
"> **Mission briefing:** Welcome to the Lab, recruit. Before we let you anywhere near an AI model, you have to pass the entrance exam โ and it isn't about code. Every powerful AI system is built on top of *data*, and a researcher who can't **read** their data ships broken models. This morning a crate of penguin measurements arrived from a field station in Antarctica. Interrogate it.\n",
"\n",
"**Today you will:**\n",
"- Load a real dataset with **pandas** and size it up (`.head`, `.info`, `.describe`)\n",
"- Hunt down **missing values** and clean the data\n",
"- **Filter, group, and count** to answer real questions\n",
"- **Plot** the data โ and spot a pattern a machine could learn tomorrow\n",
"\n",
"*No neural networks today. Just you, the data, and a magnifying glass. Every researcher here will tell you the same thing: **AI is 80% understanding your data, and 20% everything else.***\n",
"\n",
"A precise reading
This course focuses on learning from data. AI also includes rule-based and search systems. Tables are a useful starting point, but images, audio and text need representations suited to their structure.
\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ab94664c",
"metadata": {},
"outputs": [],
"source": [
"# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"# โ SETUP โ run me first! (identical in every Lab notebook) โ\n",
"# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"import os, sys, pathlib\n",
"\n",
"IN_COLAB = \"google.colab\" in sys.modules\n",
"SMOKE = os.environ.get(\"SMOKE_TEST\", \"0\") == \"1\" # tiny fast run for automated tests\n",
"\n",
"REPO_URL = \"https://github.com/A-Halimi/ai-studio-internship.git\" # instructor: set once\n",
"\n",
"if IN_COLAB:\n",
" if not pathlib.Path(\"/content/ai-studio-internship\").exists():\n",
" !git clone -q {REPO_URL} /content/ai-studio-internship\n",
" %cd /content/ai-studio-internship/notebooks/day01\n",
"\n",
"HERE = pathlib.Path.cwd()\n",
"REPO = HERE.parents[1] # .../notebooks/dayNN -> repo root\n",
"DATA_DIR = pathlib.Path(os.environ.get(\"COURSE_DATA_DIR\", REPO / \"data\"))\n",
"SAMPLES_DIR = REPO / \"data_samples\" # small datasets shipped with the repo\n",
"MODELS_DIR = REPO / \"ai_studio\" / \"models\" # where your Studio modules unlock\n",
"for p in (REPO / \"data\", MODELS_DIR):\n",
" p.mkdir(parents=True, exist_ok=True)\n",
"\n",
"import random\n",
"import numpy as np\n",
"SEED = 42\n",
"random.seed(SEED); np.random.seed(SEED)\n",
"print(f\"โ Setup done | Colab: {IN_COLAB} | Smoke test: {SMOKE}\")\n",
"print(f\" data: {DATA_DIR}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "929e8249",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"\n",
"pd.set_option(\"display.max_columns\", None) # never hide a column when we print\n",
"print(\"Detective toolkit loaded ๐ต๏ธ\")"
]
},
{
"cell_type": "markdown",
"id": "8415d2ab",
"metadata": {},
"source": [
"## ๐ง Meet the suspects\n",
"\n",
"Our data comes from **Palmer Station, Antarctica**, where researchers measured hundreds of penguins from **three species**:\n",
"\n",
"| Species | How to spot it |\n",
"|---|---|\n",
"| **Adelie** | The classic tuxedo penguin, found all over the peninsula |\n",
"| **Chinstrap** | Named for the thin black line under its chin |\n",
"| **Gentoo** | The big one, with a bright orange bill |\n",
"\n",
"For each penguin they recorded four **body measurements**, plus which **island** it lived on and its **sex**:\n",
"\n",
"- `bill_length_mm`, `bill_depth_mm` โ the beak\n",
"- `flipper_length_mm` โ the \"wing\"\n",
"- `body_mass_g` โ weight in grams\n",
"\n",
"One **row** is one penguin. One **column** is one thing we measured. Rows are examples, columns are features โ this is *exactly* how almost all machine-learning data is organized. Get comfortable with the shape."
]
},
{
"cell_type": "markdown",
"id": "f0a543c5",
"metadata": {},
"source": [
"### Step 1 โ Load the evidence\n",
"\n",
"`pandas` is the tool every data scientist reaches for first. It reads a spreadsheet-like file into a **DataFrame** โ think of it as a programmable Excel sheet. `.head()` shows the first few rows so we can eyeball what we are dealing with."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c5d43565",
"metadata": {},
"outputs": [],
"source": [
"penguins = pd.read_csv(SAMPLES_DIR / \"penguins.csv\")\n",
"print(f\"Loaded {len(penguins)} penguins, {penguins.shape[1]} columns each.\")\n",
"penguins.head()"
]
},
{
"cell_type": "markdown",
"id": "d8e450a7",
"metadata": {},
"source": [
"### Step 2 โ Size it up\n",
"\n",
"Two commands you will run on *every* new dataset for the rest of your career:\n",
"\n",
"- **`.info()`** โ how many rows, the type of each column, and how many values are actually *present* (non-null). The place to spot **missing** data.\n",
"- **`.describe()`** โ count, mean, min, max, and quartiles for every number column. The place to spot **weird** values (a penguin weighing 0 g? a typo?)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "810f6e17",
"metadata": {},
"outputs": [],
"source": [
"penguins.info()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c83c308a",
"metadata": {},
"outputs": [],
"source": [
"penguins.describe()"
]
},
{
"cell_type": "markdown",
"id": "1b41d8e1",
"metadata": {},
"source": [
"### ๐งช Exercise 1 โ Count the gaps\n",
"\n",
"Real data is messy. Some penguins are missing measurements โ maybe a pen ran dry, maybe a bird wriggled free before the scale settled.\n",
"\n",
"- Compute how many values are **missing** in each column and store it in `missing_per_column`.\n",
"- **Expected:** a couple of missing body measurements, and about a dozen missing `sex` values."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "57939a6d",
"metadata": {},
"outputs": [],
"source": [
"# How many values are missing in each column?\n",
"# ==================== YOUR CODE HERE ====================\n",
"### HINT: penguins.isna() marks every cell True (missing) or False\n",
"### HINT: chain .sum() on that to count the missing values per column\n",
"...\n",
"print(missing_per_column)\n",
"\n",
"assert missing_per_column.sum() == 19, \"Expected 19 missing values in total across all columns.\""
]
},
{
"cell_type": "markdown",
"id": "1b9e8535",
"metadata": {},
"source": [
"### Step 3 โ Clean it up\n",
"\n",
"For today's investigation we want only penguins with a **complete** record. `dropna()` removes any row that has at least one missing value. We keep the original around and work on a clean copy called `df`.\n",
"\n",
"> โ ๏ธ Dropping rows is the quick fix, not always the *right* one โ throwing away data can bias your results. Later in the Lab you will learn gentler repairs (filling a gap with a sensible guess). For 11 rows out of 344, dropping is fine."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9a891ce5",
"metadata": {},
"outputs": [],
"source": [
"df = penguins.dropna().reset_index(drop=True)\n",
"print(f\"Before: {len(penguins)} penguins\")\n",
"print(f\"After : {len(df)} penguins ({len(penguins) - len(df)} dropped)\")\n",
"assert len(df) == 333, \"A clean dataset should have 333 complete records.\"\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"id": "14383426",
"metadata": {},
"source": [
"## Ask questions, get answers\n",
"\n",
"A DataFrame is not just for looking at โ it is for **interrogating**. Three moves answer almost any question about a dataset:\n",
"\n",
"- **Filter** โ keep only the rows that match a condition\n",
"- **Group** โ split rows into buckets (e.g. by species) and summarize each bucket\n",
"- **Count** โ tally how often each value appears\n",
"\n",
"Let's use all three."
]
},
{
"cell_type": "markdown",
"id": "c343eeeb",
"metadata": {},
"source": [
"### ๐งช Exercise 2 โ The heavyweights\n",
"\n",
"Find every penguin heavier than **5000 grams** (that's 5 kg โ a chunky bird).\n",
"\n",
"- Build a filtered DataFrame called `heavy`.\n",
"- **Expected:** around 60 penguins โ and if you peek at their species, they will almost all be the *same* one. Remember which; it comes up again."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c5dd6410",
"metadata": {},
"outputs": [],
"source": [
"# ==================== YOUR CODE HERE ====================\n",
"### HINT: a condition like df[\"body_mass_g\"] > 5000 gives True/False for every row\n",
"### HINT: put that condition inside df[ ... ] to keep only the True rows\n",
"...\n",
"print(f\"{len(heavy)} penguins over 5000 g\")\n",
"print(heavy[\"species\"].value_counts())\n",
"\n",
"assert len(heavy) > 40, \"That is fewer heavyweights than expected โ check your condition.\""
]
},
{
"cell_type": "markdown",
"id": "d34ddda4",
"metadata": {},
"source": [
"### ๐งช Exercise 3 โ The average penguin, by species\n",
"\n",
"Split the penguins by species and compute the **mean** of every numeric column for each group.\n",
"\n",
"- Produce a table called `species_means`.\n",
"- **Hint:** `.groupby(\"species\")` makes the buckets; `.mean(numeric_only=True)` averages the numbers (the `numeric_only=True` skips text columns like `island`).\n",
"- **Expected:** one row per species, with Gentoo standing out as the biggest on nearly every measurement."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5fdcf63d",
"metadata": {},
"outputs": [],
"source": [
"# ==================== YOUR CODE HERE ====================\n",
"...\n",
"species_means"
]
},
{
"cell_type": "markdown",
"id": "e68891d3",
"metadata": {},
"source": [
"### ๐งช Exercise 4 โ Who lives where?\n",
"\n",
"Count how many penguins were recorded on each **island**.\n",
"\n",
"- Store the result in `island_counts`.\n",
"- **Expected:** three islands (Biscoe, Dream, Torgersen), with Biscoe hosting the most."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "26dec1c0",
"metadata": {},
"outputs": [],
"source": [
"# ==================== YOUR CODE HERE ====================\n",
"### HINT: .value_counts() on a single column tallies each distinct value\n",
"...\n",
"print(island_counts)"
]
},
{
"cell_type": "markdown",
"id": "a2a83620",
"metadata": {},
"source": [
"## See it to believe it\n",
"\n",
"Tables are precise but slow to read. A good plot shows a pattern in a glance. Start with a **histogram**: it slices a numeric column into bins and shows how many penguins land in each โ the *shape* of the data.\n",
"\n",
"Here is one we made for you: body mass, drawn separately for each species so you can compare. Run it."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5bb7c4a8",
"metadata": {},
"outputs": [],
"source": [
"plt.figure(figsize=(8, 5))\n",
"for species, group in df.groupby(\"species\"):\n",
" plt.hist(group[\"body_mass_g\"], bins=20, alpha=0.6, label=species)\n",
"plt.xlabel(\"Body mass (g)\")\n",
"plt.ylabel(\"Number of penguins\")\n",
"plt.title(\"Body mass by species\")\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "fadb4d04",
"metadata": {},
"source": [
"See how **Gentoo** (heavier) sits clearly to the right, while **Adelie** and **Chinstrap** overlap almost completely? Body mass alone can pick out a Gentoo โ but it can't tell an Adelie from a Chinstrap. We need a smarter view."
]
},
{
"cell_type": "markdown",
"id": "fee6b6c5",
"metadata": {},
"source": [
"### ๐งช Exercise 5 โ The scatter plot that changes everything\n",
"\n",
"A **scatter plot** puts one measurement on the x-axis and another on the y-axis, one dot per penguin. The magic move: **color each dot by species**. If the species land in separate regions, we have found structure a machine could learn.\n",
"\n",
"Plot **bill length** (x) against **bill depth** (y), one color per species.\n",
"\n",
"- Loop over the species; for each one, call `plt.scatter(...)` on just that group's rows with a `label=`.\n",
"- The scaffold handles the axis labels and legend โ you write the scatter loop.\n",
"- **Expected:** three fairly distinct blobs of color."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2919246c",
"metadata": {},
"outputs": [],
"source": [
"plt.figure(figsize=(8, 6))\n",
"# ==================== YOUR CODE HERE ====================\n",
"### HINT: for species, group in df.groupby(\"species\"):\n",
"### HINT: plt.scatter(group[\"bill_length_mm\"], group[\"bill_depth_mm\"], label=species, alpha=0.7)\n",
"...\n",
"plt.xlabel(\"Bill length (mm)\")\n",
"plt.ylabel(\"Bill depth (mm)\")\n",
"plt.title(\"Bill shape separates the species\")\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "36754286",
"metadata": {},
"source": [
"### ๐ The revelation\n",
"\n",
"Look at that. Each species forms its **own cluster** โ a region of the plot where its penguins live. You could almost draw the dividing lines with a ruler.\n",
"\n",
"That is the whole idea behind tomorrow's mission. **If *you* can see the boundary, a machine can *learn* it.** Hand a computer these dots and their colors, and it can work out the dividing lines on its own โ then color in a brand-new, unlabeled dot for you. That's called **classification**, and it's the first AI superpower you'll build. ๐ง\n",
"\n",
"> **Why bill *length* and *depth*?** Because together they capture *shape*, not just size. Size (mass, flipper) mostly separates Gentoo from the rest; shape is what finally splits Adelie from Chinstrap."
]
},
{
"cell_type": "markdown",
"id": "958a791b",
"metadata": {},
"source": [
"### ๐งช Exercise 6 โ What moves together?\n",
"\n",
"When one measurement goes up, do the others go up too? **Correlation** measures exactly that, from **+1** (perfect lockstep), through **0** (unrelated), to **โ1** (perfect opposites).\n",
"\n",
"- Compute the correlation matrix of the four numeric columns and store it in `corr`.\n",
"- We'll draw it as a **heatmap** so the strong pairs jump out.\n",
"- **Hint:** select the four columns, then call `.corr()`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6725cbba",
"metadata": {},
"outputs": [],
"source": [
"numeric_cols = [\"bill_length_mm\", \"bill_depth_mm\", \"flipper_length_mm\", \"body_mass_g\"]\n",
"# ==================== YOUR CODE HERE ====================\n",
"...\n",
"\n",
"plt.figure(figsize=(6, 5))\n",
"sns.heatmap(corr, annot=True, cmap=\"coolwarm\", vmin=-1, vmax=1, fmt=\".2f\")\n",
"plt.title(\"Correlation between measurements\")\n",
"plt.show()\n",
"corr"
]
},
{
"cell_type": "markdown",
"id": "b8488ccd",
"metadata": {},
"source": [
"**Read the map.** The brightest off-diagonal square is **flipper length โ body mass** at about **+0.87** โ long-flippered penguins are heavy penguins. That makes sense: both grow with the overall size of the bird.\n",
"\n",
"Notice that `bill_length` and `bill_depth` are only weakly (even negatively) correlated โ which is *exactly* why they made such a good scatter plot. Two measurements that *don't* move together carry **independent** information, and independent information is what pulls the species apart."
]
},
{
"cell_type": "markdown",
"id": "9e7b8bb2",
"metadata": {},
"source": [
"## ๐๏ธ The Detective's Report\n",
"\n",
"Time to prove you can work solo.\n",
"\n",
"### ๐งช Exercise 7 โ Close the case\n",
"\n",
"Answer three questions with **one line of pandas each** โ no plots, just the value. Everything you need you have already used above.\n",
"\n",
"1. Which species is **heaviest on average**?\n",
"2. Which **island** hosts the most **Gentoo** penguins?\n",
"3. What is the **mean flipper length** of **Chinstrap** penguins?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d10478da",
"metadata": {},
"outputs": [],
"source": [
"# 1) Heaviest species on average\n",
"# ==================== YOUR CODE HERE ====================\n",
"### HINT: group by species, take the mean body_mass_g, then .idxmax() names the winner\n",
"...\n",
"\n",
"# 2) Island with the most Gentoo penguins\n",
"# ==================== YOUR CODE HERE ====================\n",
"### HINT: keep only Gentoo rows first, then value_counts the island column, then .idxmax()\n",
"...\n",
"\n",
"# 3) Mean flipper length of Chinstrap penguins\n",
"# ==================== YOUR CODE HERE ====================\n",
"...\n",
"\n",
"print(f\"1. Heaviest species on average : {heaviest_species}\")\n",
"print(f\"2. Island with most Gentoo : {top_gentoo_island}\")\n",
"print(f\"3. Chinstrap mean flipper (mm) : {chinstrap_flipper:.1f}\")\n",
"\n",
"assert heaviest_species == \"Gentoo\"\n",
"assert top_gentoo_island == \"Biscoe\"\n",
"assert round(chinstrap_flipper, 1) == 195.8\n",
"print(\"\\nโ Case closed โ all three answers check out.\")"
]
},
{
"cell_type": "markdown",
"id": "7cc1ab41",
"metadata": {},
"source": [
"## ๐ Checkpoint\n",
"\n",
"Case closed, Detective. You just did what every AI project starts with โ long before a single model is trained:\n",
"\n",
"- โ Loaded and inspected a real dataset (`.head`, `.info`, `.describe`)\n",
"- โ Found and removed missing data\n",
"- โ Filtered, grouped, and counted to answer real questions\n",
"- โ Visualized the data โ and spotted **species clusters** with your own eyes\n",
"- โ Measured which features carry independent information\n",
"\n",
"You proved the Lab's motto: **understand your data before you model it.**\n",
"\n",
"**Next up โ Notebook 1b: \"Meet the Machines.\" ๐ค** You've earned a reward: a hands-on tour of four finished AI systems โ vision, mood-reading, story-writing, and a sorting hat. Play with them today; you will *build* every one over the next two weeks.\n",
"\n",
"> **Tomorrow (Day 2):** those penguin clusters you found? You'll hand them to a machine, train it to identify a brand-new penguin on its own, and unlock the very first tab of your **AI Studio**. ๐"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "91997f6b-e71e-4e71-9308-0ed8c6639041",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "330f9ff7-f27e-4a07-9bb8-4cc4e1dd7df6",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "f9c486a6",
"metadata": {},
"source": [
"---\n",
"\n",
"