Table of Contents
Most Kaggle Titanic tutorials open with import pandas as pd and immediately start running .describe(). I did something different: I read about the actual disaster first.
The result was better feature engineering, more informed imputation strategies, and a clearer picture of what was noise versus signal. The model barely mattered — the features did all the work.
Why Research Before Code
The Titanic dataset has 891 rows and 11 features. You could get ~77% accuracy by predicting “all women survive, all men die.” Getting past 77% requires understanding why people survived beyond gender. That understanding comes from the history, not the data.
Here’s what I learned before touching a single CSV:
The evacuation was physically stratified. First-class cabins were on upper decks A-C, steps from the lifeboats on the Boat Deck. Third-class passengers were on decks F-G with restricted access routes and, in some cases, locked gates. Passenger class isn’t just wealth — it’s physical proximity to survival.
“Women and children first” was enforced unevenly. Officers on different sides of the ship interpreted the protocol differently. Second Officer Lightoller interpreted it strictly (women and children only), while First Officer Murdoch interpreted it as priority (women first, but men could board if space remained). This creates variance that pure data can’t fully capture.
Family dynamics affected decisions. Solo travelers had no one to help or wait for. Small families evacuated together efficiently. Large families often refused to separate, which meant some members who could have survived didn’t board lifeboats.
Research-Driven Feature Engineering
Every feature I engineered had a documented “why” from the research:
Title Extraction
The Name field contains titles: “Mr. John Smith”, “Mrs. Margaret Brown”, “Master. William Carter”. “Master” specifically means a male child in 1912 terminology. This is crucial — when Age is missing (20% of rows), the title tells us whether someone was a child or an adult.
df['Title'] = df['Name'].str.extract(r' ([A-Za-z]+)\.', expand=False)
# Master → child, Mr → adult male, Mrs/Miss → female
This isn’t just a nice feature — it’s the foundation for informed age imputation. Instead of filling missing ages with the global median (28), I imputed by title group:
- Master: median age ~5
- Miss: median age ~22
- Mr: median age ~30
This preserves the child/adult distinction that drives survival.
FamilySize and the Non-Linear Effect
df['FamilySize'] = df['SibSp'] + df['Parch'] + 1
df['IsAlone'] = (df['FamilySize'] == 1).astype(int)
Research predicted a non-linear relationship: solo travelers do poorly (no one to help during chaos), small families do best (mutual support), large families do worst (can’t evacuate together). The data confirmed exactly this pattern.
HasCabin as a Wealth Signal
77% of Cabin values are missing. Most tutorials either drop the column or try to impute deck letters from sparse data. But research suggests the missingness itself is the signal — cabin numbers were more consistently recorded for upper-class passengers with documented reservations.
df['HasCabin'] = df['Cabin'].notna().astype(int)
A simple binary flag that captures the wealth/documentation proxy without overfitting to sparse data.
The Overfitting Discovery
Here’s the most important thing I learned, and it wasn’t from any tutorial:
On this dataset, higher CV scores mean worse leaderboard scores past a threshold.
CV 0.86 → LB 0.758 (worst submission)
CV 0.83 → LB 0.773 (best submission)
CV 0.82 → LB 0.773 (tied best)
With only 891 training rows, any model complex enough to score above ~83% in cross-validation is memorizing noise. The optimal zone was CV 0.82-0.83 with standard deviation under 0.012.
This inverted relationship only happens on small datasets. On the Spaceship Titanic competition (8700 rows), CV and LB tracked almost perfectly. Understanding this required running multiple experiments and trusting the LB over my CV scores — counterintuitive for most ML practitioners.
The Conservative Model That Won
My best submission used a deliberately conservative Gradient Boosting classifier:
model = GradientBoostingClassifier(
n_estimators=50, # few trees
max_depth=3, # shallow
min_samples_leaf=10, # large leaves
subsample=0.8, # randomization
learning_rate=0.1,
random_state=42
)
Only 8 features: Pclass, Age, SibSp, Parch, Fare, FamilySize, IsAlone, IsFemale.
This scored 0.773 on the leaderboard — modest by competition standards but achieved through understanding rather than brute force. More importantly, I know exactly why it works and can transfer those insights to future problems.
What I’d Do Differently
Submit earlier. I ran a complex arena comparison of 21 model configurations before submitting once. Should have submitted the simplest model first to establish the CV-LB gap, then used that gap to calibrate everything else.
Trust the gap, not the CV. High CV on small datasets is a trap. The only reliable signal is the gap between CV and LB — smaller gap = more trustworthy model.
Stop sooner. The ceiling on Titanic with standard approaches is ~0.773-0.78. Chasing higher requires tricks (target encoding with careful out-of-fold computation, stacking at risk of overfitting) that produce diminishing returns. The value was in the methodology, not the extra half-point.
The Takeaway
Domain research isn’t a luxury — it’s the highest-leverage activity in the ML pipeline. Thirty minutes reading about the Titanic disaster told me more about what features would work than three hours of automated feature selection ever could.
The full experiment log, research documents, and code are in the Kaggle ML Toolkit repository.