Our first Kaggle entry: a baseline with receipts

We have started entering competitions. The first is on Kaggle: Playground Series S6E10, predicting airline passenger satisfaction. This post is about what we submitted and, more to the point, what that number does and does not tell you.

What a Playground round is

Kaggle's Playground Series are practice rounds. They are real competitions with a real leaderboard, but they award no points and no medals. We chose one for that reason. A first entry should teach you the platform's mechanics (data download, submission format, how the public score behaves) without anyone pretending the score is an achievement.

So the claim here is small. We have a baseline, and we can show how we got it.

The task

The data is a table of airline passengers. Each row has 21 features: age, flight distance, two delay columns, thirteen service ratings (wifi, seat comfort, cleanliness and so on), and four categorical columns (gender, customer type, type of travel, class). The target is whether the passenger was satisfied. The training file has 699,635 rows and the test file has 299,844. The metric is ROC AUC, so the submission is a probability per row, not a label.

The baseline

One script, one model, no feature engineering. The four categorical columns are marked as categories, and the test file is given the training file's category list so both agree. We train LightGBM with five-fold stratified cross-validation, a fixed seed, and early stopping on each held-out fold:

params = dict(objective="binary", learning_rate=0.05, num_leaves=127,
              min_child_samples=50, feature_fraction=0.8,
              bagging_fraction=0.8, bagging_freq=1)
folds = StratifiedKFold(5, shuffle=True, random_state=42)
# per fold: up to 3000 rounds, stop after 100 without improvement

Each fold's model scores its own held-out rows. Put those scores together and every training row has a prediction from a model that never saw it. The ROC AUC of that pooled set is the cross-validation score: 0.95887. For the test file, we average the five fold models and write submission.csv.

Kaggle scored that file at 0.95844 on the public leaderboard.

What the two numbers tell you

They sit 0.0004 apart. That is what you hope to see: the cross-validation estimate, which we computed ourselves on data we have, agrees with a score Kaggle computed on data we cannot see. If they had disagreed by a lot, we would have a leakage bug or a split problem to find before doing anything clever.

They do not tell you where we rank, and we are not claiming a rank. They also do not tell you the model is good in any absolute sense. A plain gradient-boosted baseline is what everyone else will start from, so the number is a reference point for our own later changes, not a result to point at.

The receipts

The lab's rule, from our prove-it-done work, is that a result is not a result until someone else can rerun the check. For this entry that means:

• A single script that takes the data directory and an output path, prints each fold's AUC as it goes, and prints the pooled out-of-fold AUC at the end.
• A fixed seed for the fold split, so a rerun builds the same folds.
• The parameters in the open, as shown above, with nothing hidden in the script.
• Two scores with two origins: one we computed, one Kaggle computed. We report both.

Before publishing this we re-ran the script from scratch. It printed the same 0.95887 and wrote a submission file byte-for-byte identical to the one we sent.

The script is not public yet.

What we would try next

Nothing here is done yet, so these are plans, not results. The natural next steps are sum and ratio features across the rating columns, a second model type to blend with, and a look at the two delay columns, which may overlap. We have not checked. The discipline stays the same. Change one thing, rerun the same five folds, and keep a change only if the pooled out-of-fold AUC moves by more than the noise between folds.

The pattern, if you are starting too

• Pick a round that doesn't pay in points. It lets you spend the first submission on learning the pipeline.
• Compute your own out-of-fold score before you look at the leaderboard. When the two agree, you can trust the loop. When they don't, fix that first.
• Fix the seed and print per-fold numbers. A single pooled number hides a bad fold.
• Report the baseline plainly. A later improvement only means something against a baseline you can reproduce.

Independent, personal project. Kaggle is a trademark of its owner; this post is not affiliated with or endorsed by Kaggle.