Lab 1 — Build a Training Dataset

Feature

An input measurement the model uses, e.g. word count, pixel brightness, time of day.

Label

The correct answer attached to a training example.

Class imbalance

When one label appears far more often than others, skewing training.

Train/validation/test split

Separating data so the model is judged on examples it has never seen.

Data leakage

When information from the test set sneaks into training, giving a fake high score.

You have 30 minutes and a class of 25 students. Plan the data.

1. 11. Define labels: Two classes: scam and legit. Write a one-sentence rule so every labeller agrees, e.g. 'scam = asks for money, credentials or urgent action from an unknown sender'.
2. 22. Decide features: Message text, sender known/unknown, contains link, urgency words, spelling errors.
3. 33. Collect evenly: Target 100 scam and 100 legit. Ask each student for 4 of each so no single inbox dominates.
4. 44. Double label: Two students label each message independently. Disagreements go to a third — this measures how noisy your labels are.
5. 55. Split and freeze: Shuffle, then take 140 train / 30 validation / 30 test. Seal the test set until the very end.

pythonimport random
from collections import Counter

data = [("win a free phone now", "scam"), ("maths homework page 42", "legit")] * 50
print(Counter(label for _, label in data))

random.seed(42)          # same shuffle every run = reproducible
random.shuffle(data)
n = len(data)
train = data[: int(0.7 * n)]
val   = data[int(0.7 * n) : int(0.85 * n)]
test  = data[int(0.85 * n) :]
print(len(train), len(val), len(test))

Setting a random seed means your split is reproducible — a teammate running the same code gets the same rows.