Research First: Beating Kaggle's Titanic with History Books

Most Kaggle Titanic tutorials open with import pandas as pd and immediately start running .describe(). I did something different: I read about the actual disaster first.

The result was better feature engineering, more informed imputation strategies, and a clearer picture of what was noise versus signal. The model barely mattered. The features did all the work.

Why Research Before Code

The Titanic dataset has 891 rows and 11 features. You could get ~77% accuracy by predicting “all women survive, all men die.” Getting past 77% requires understanding why people survived beyond gender. That understanding comes from the history, not the data.

Here’s what I learned before touching a single CSV:

The evacuation was physically stratified. First-class cabins were on upper decks A-C, steps from the lifeboats on the Boat Deck. Third-class passengers were on decks F-G with restricted access routes and, in some cases, locked gates. Passenger class isn’t just wealth; it’s physical proximity to survival.

“Women and children first” was enforced unevenly. Officers on different sides of the ship interpreted the protocol differently. Second Officer Lightoller interpreted it strictly (women and children only), while First Officer Murdoch interpreted it as priority (women first, but men could board if space remained). This creates variance that pure data can’t fully capture.

Family dynamics affected decisions. Solo travelers had no one to help or wait for. Small families evacuated together efficiently. Large families often refused to separate, which meant some members who could have survived didn’t board lifeboats.

Research-Driven Feature Engineering

Every feature I engineered had a documented “why” from the research:

Title Extraction

The Name field contains titles: “Mr. John Smith”, “Mrs. Margaret Brown”, “Master. William Carter”. “Master” specifically means a male child in 1912 terminology. This is crucial: when Age is missing (20% of rows), the title tells us whether someone was a child or an adult.

df['Title'] = df['Name'].str.extract(r' ([A-Za-z]+)\.', expand=False)
# Master → child, Mr → adult male, Mrs/Miss → female

This isn’t just a nice feature; it’s the foundation for informed age imputation. Instead of filling missing ages with the global median (28), I imputed by title group:

• Master: median age ~5
• Miss: median age ~22
• Mr: median age ~30

This preserves the child/adult distinction that drives survival.

FamilySize and the Non-Linear Effect

df['FamilySize'] = df['SibSp'] + df['Parch'] + 1
df['IsAlone'] = (df['FamilySize'] == 1).astype(int)

Research predicted a non-linear relationship: solo travelers do poorly (no one to help during chaos), small families do best (mutual support), large families do worst (can’t evacuate together). The data confirmed exactly this pattern.

HasCabin as a Wealth Signal

77% of Cabin values are missing. Most tutorials either drop the column or try to impute deck letters from sparse data. But research suggests the missingness itself is the signal, since cabin numbers were more consistently recorded for upper-class passengers with documented reservations.

df['HasCabin'] = df['Cabin'].notna().astype(int)

A simple binary flag that captures the wealth/documentation proxy without overfitting to sparse data.

Targeted EDA

With research hypotheses in hand, exploratory analysis became confirmation rather than fishing. The missing-value picture framed the whole imputation strategy:

And the univariate predictors lined up exactly with the history:

Choosing a Model Family

Before tuning anything, I compared five model families with stratified 5-fold cross-validation, just to see which deserved attention:

Tree-based methods clearly led. But as the next section shows, “best CV” turned out to be a trap on a dataset this small.

The Overfitting Discovery

Here’s the most important thing I learned, and it wasn’t from any tutorial:

On this dataset, higher CV scores mean worse leaderboard scores past a threshold.

CV 0.86 → LB 0.758 (worst submission)
CV 0.83 → LB 0.773 (best submission)
CV 0.82 → LB 0.773 (tied best)

With only 891 training rows, any model complex enough to score above ~83% in cross-validation is memorizing noise. The optimal zone was CV 0.82-0.83 with standard deviation under 0.012.

This inverted relationship only happens on small datasets. On the Spaceship Titanic competition (8700 rows), CV and LB tracked almost perfectly. Understanding this required running multiple experiments and trusting the LB over my CV scores, which is counterintuitive for most ML practitioners.

The Conservative Model That Won

My best submission used a deliberately conservative Gradient Boosting classifier:

model = GradientBoostingClassifier(
    n_estimators=50,    # few trees
    max_depth=3,        # shallow
    min_samples_leaf=10, # large leaves
    subsample=0.8,      # randomization
    learning_rate=0.1,
    random_state=42
)

Only 8 features: Pclass, Age, SibSp, Parch, Fare, FamilySize, IsAlone, IsFemale.

This scored 0.773 on the leaderboard, modest by competition standards but achieved through understanding rather than brute force. More importantly, I know exactly why it works and can transfer those insights to future problems.

What I’d Do Differently

1. Submit earlier. I ran a complex arena comparison of 21 model configurations before submitting once. Should have submitted the simplest model first to establish the CV-LB gap, then used that gap to calibrate everything else.

2. Trust the gap, not the CV. High CV on small datasets is a trap. The only reliable signal is the gap between CV and LB, where a smaller gap means a more trustworthy model.

3. Stop sooner. The ceiling on Titanic with standard approaches is ~0.773-0.78. Chasing higher requires tricks (target encoding with careful out-of-fold computation, stacking at risk of overfitting) that produce diminishing returns. The value was in the methodology, not the extra half-point.

What Didn’t Work

Worth recording the dead ends, since they shaped the final simple model:

• Deck extraction from Cabin — 77% missing left too little signal; the HasCabin flag captured what was there without overfitting.
• Complex interactions (Age × Pclass, fare-per-person) — marginal at best; the simple features already encoded them.
• SVM — theoretically appealing, consistently beaten by trees on this mixed-type data.
• Kitchen-sink feature engineering — more features meant more noise. The research-backed subset beat the everything-included version.

The Takeaway

Domain research isn’t a luxury. It’s the highest-leverage activity in the ML pipeline. Thirty minutes reading about the Titanic disaster told me more about what features would work than three hours of automated feature selection ever could.

The full experiment log, research documents, and code are in the Kaggle ML Toolkit repository.