Round 24

Handling Missing Data in the Churn Model

A mid-size subscription analytics team is finalizing a customer churn prediction model before next quarter's retention campaign. Exploratory analysis revealed that ~18% of records have missing values across several key features (payment history, support ticket counts, usage logs), with missingness patterns that appear non-random (newer customers and certain acquisition channels are disproportionately affected). The team must decide on a single primary strategy for handling this missing data before model training. Constraints: the campaign launch date is fixed in five weeks, the model must be interpretable enough for the retention team to trust its outputs, and the chosen method must be maintainable by a small analytics team without dedicated ML infrastructure engineers. Tradeoffs include bias risk if missingness correlates with churn itself, added model complexity versus transparency, computational and pipeline overhead, and the danger of silently distorting the training distribution. The team must pick one dominant approach to standardize on, acknowledging reasonable disagreement about which balance of accuracy, interpretability, and speed is best.

Dashboard

Status

DECIDED

Winner

F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn.

Majority

F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn.

Duration

6716 ms

Topic

Options

F Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn.
3.00 of 3 PPV winner
A Delay model deployment by two weeks to run a targeted data-quality audit and backfill missing fields at the source before any imputation decision is made.
0.00 of 3
B Use a model class that natively handles missing values (e.g., gradient-boosted trees with built-in missing-value splits), avoiding explicit imputation altogether.
0.00 of 3
C Drop all rows with missing values in key features, accepting a smaller but fully observed training set, prioritizing simplicity and avoiding any imputation bias.
0.00 of 3
D Use multiple imputation (e.g., MICE) to generate several completed datasets, pool model results, and accept the added computational and explanatory complexity for statistically principled uncertainty handling.
0.00 of 3
E Build a separate small model to predict missingness patterns and use it to inform stratified imputation by customer segment, trading extra engineering effort for more context-aware estimates.
0.00 of 3

Checks

Baselines

PPV no gold answer
Majority no gold answer
Comparison PPV matched majority
k samples8
Agents3
Parsed samples 24 / 24
Quality unanimous

PPV

Top Influences

Claude Sonnet 5 · delegate 1.000
GPT 5.4 mini · delegate 1.000
Gemini 3 Flash Preview · delegate 1.000

Samples

Answer Distribution

Claude Sonnet 5 F: 8
GPT 5.4 mini F: 8
Gemini 3 Flash Preview F: 8

PPV

Influences

Label Role Score
Claude Sonnet 5 delegate 1.000
GPT 5.4 mini delegate 1.000
Gemini 3 Flash Preview delegate 1.000

Voting matrix

Constructed Delegation Matrix

To / From Claude Sonnet 5 GPT 5.4 mini Gemini 3 Flash Preview
Claude Sonnet 5 0.0000.0000.000
GPT 5.4 mini 0.0000.0000.000
Gemini 3 Flash Preview 0.0000.0000.000
A 0.0000.0000.000
B 0.0000.0000.000
C 0.0000.0000.000
D 0.0000.0000.000
E 0.0000.0000.000
F 1.0001.0001.000

Embeddings

Sample Embeddings

24 samples
Claude Sonnet 5 sample #0 · F · 1536 dims F Claude Sonnet 5 sample #1 · F · 1536 dims F Claude Sonnet 5 sample #2 · F · 1536 dims F Claude Sonnet 5 sample #3 · F · 1536 dims F Claude Sonnet 5 sample #4 · F · 1536 dims F Claude Sonnet 5 sample #5 · F · 1536 dims F Claude Sonnet 5 sample #6 · F · 1536 dims F Claude Sonnet 5 sample #7 · F · 1536 dims F GPT 5.4 mini sample #0 · F · 1536 dims F GPT 5.4 mini sample #1 · F · 1536 dims F GPT 5.4 mini sample #2 · F · 1536 dims F GPT 5.4 mini sample #3 · F · 1536 dims F GPT 5.4 mini sample #4 · F · 1536 dims F GPT 5.4 mini sample #5 · F · 1536 dims F GPT 5.4 mini sample #6 · F · 1536 dims F GPT 5.4 mini sample #7 · F · 1536 dims F Gemini 3 Flash Preview sample #0 · F · 1536 dims F Gemini 3 Flash Preview sample #1 · F · 1536 dims F Gemini 3 Flash Preview sample #2 · F · 1536 dims F Gemini 3 Flash Preview sample #3 · F · 1536 dims F Gemini 3 Flash Preview sample #4 · F · 1536 dims F Gemini 3 Flash Preview sample #5 · F · 1536 dims F Gemini 3 Flash Preview sample #6 · F · 1536 dims F Gemini 3 Flash Preview sample #7 · F · 1536 dims F
Claude Sonnet 5 GPT 5.4 mini Gemini 3 Flash Preview

Samples

Agent Responses

Claude Sonnet 5 openai_compatible · anthropic/claude-sonnet-5
#0 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

Given the tight five-week timeline and the need for a small team to maintain an interpretable model, options requiring delays (A), complex statistical machinery (D), or extra sub-models (E) impose too much overhead or risk missing the deadline. Since missingne...

#1 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

Given non-random missingness that likely correlates with churn (newer customers, specific acquisition channels), dropping rows (C) risks severe bias and loses signal, while MICE (D) or auxiliary missingness models (E) add complexity and timeline risk that a sm...

#2 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

Given non-random missingness that likely correlates with churn (newer customers, certain channels), dropping rows (C) risks severe bias and loses signal, while heavy approaches like MICE (D) or segment-based models (E) add complexity and timeline risk beyond t...

#3 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

Given the non-random missingness (likely predictive of churn itself) and the constraints of a fixed 5-week deadline, need for interpretability, and a small team without ML infra support, the best approach must capture the signal in missingness while remaining ...

#4 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

Since missingness is non-random and likely correlated with churn (newer customers, certain acquisition channels), dropping rows (C) risks losing signal and biasing the dataset, while full MICE (D) or segment-based modeling (E) add complexity beyond what a smal...

#5 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

Given the non-random missingness (which likely correlates with churn) and the constraints (5-week deadline, need for interpretability, small team without ML infra), option F is preferable because it directly captures whether missingness itself signals churn ri...

#6 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

Given the tight five-week deadline, non-random missingness that may itself signal churn risk, and a small team without ML infra support, the best balance is an approach that is simple to maintain, interpretable, and explicitly captures the informative nature o...

#7 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

Given the non-random missingness (informative in itself), the tight 5-week deadline, and the need for interpretability with a small team, the best approach is to explicitly flag missingness rather than hide it. Option F preserves signal from missingness patter...

GPT 5.4 mini openai_compatible · openai/gpt-5.4-mini
#0 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

The missingness is non-random and likely informative, so dropping rows would risk bias and losing too much data, while multiple imputation or a separate missingness model adds complexity that is hard to maintain and explain. A simple mean/median imputation wit...

#1 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

The best choice is to use simple imputation plus missingness indicators, because it is practical for a small team, easy to implement before the fixed launch date, and keeps the model interpretable for stakeholders. Since missingness is non-random and may itsel...

#2 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

Because the missingness is non-random, simply dropping rows could bias the training set, and multiple imputation or a separate missingness model adds complexity that is harder to maintain and explain. A native-missing-value model could work, but for a small te...

#3 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

Given the non-random missingness, dropping rows risks bias and wasting too much data, while multiple imputation and separate missingness models add complexity that is hard to maintain with a small team. A native-missing model is attractive, but the team also n...

#4 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

The missingness is non-random, so dropping rows risks introducing serious bias, and multiple imputation or a separate missingness model would add complexity that is hard to maintain within five weeks. A model with built-in missing handling can work well, but f...

#5 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

The missingness is non-random, so dropping rows would likely bias the training data, and multiple imputation or a separate missingness model adds complexity that is hard to maintain for a small team. A model with native missing-value handling can work well, bu...

#6 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

The missingness is non-random, so simply dropping rows would likely bias the training set, and multiple imputation or a separate missingness model adds complexity that may be hard to maintain on the timeline. A model with built-in missing handling is appealing...

#7 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

The missingness is non-random, so dropping rows would likely bias the training set, and multiple imputation or separate missingness models add more complexity than this team can maintain under a tight launch timeline. A simple imputation plus explicit missingn...

Gemini 3 Flash Preview openai_compatible · google/gemini-3-flash-preview
#0 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

To address the non-random missingness (MNAR/MAR) while meeting strict deadlines and transparency requirements, the team needs a method that captures the signal in the missing data without heavy infrastructure. Option (F) is the most practical choice because ad...

#1 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

This approach addresses the non-random missingness (MNAR/MAR) by capturing the information value of the missing data itself without introducing the heavy overhead of MICE or source-level audits. It maintains high model interpretability and transparency, allowi...

#2 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

The team faces non-random missing data (MNAR/MAR) on a tight deadline, requiring a balance between predictive power, interpretability, and ease of maintenance. Option (F) is the most practical choice because it captures the predictive information inherent in t...

#3 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

This decision requires balancing a fixed five-week deadline and a small team's capacity with the need for transparency and handling non-random missingness. Option (F) is the most pragmatic choice because it captures the predictive power of missing patterns (cr...

#4 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

This decision requires balancing a tight timeline (five weeks), the need for interpretability for non-technical teams, and the non-random nature of the missing data. Option (F) is the most practical choice because it captures the predictive information inheren...

#5 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

Missing data that is non-random (MNAR/MAR) often contains predictive value, and simple imputation without indicators risks losing this information or distorting the distribution. Since the project lacks dedicated ML infrastructure but requires transparency and...

#6 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

The team faces a scenario where missingness is non-random (MNAR/MAR) and predictive of certain segments, meaning dropping data (C) would introduce significant selection bias. Option (B) and (F) are the most practical given the five-week deadline and small team...

#7 F · Add explicit 'missingness' indicator features alongside simple mean/median imputation, letting the model learn whether missingness itself is predictive of churn. 1536 dims

This problem requires balancing a fixed five-week deadline and limited engineering resources with the fact that missingness is non-random and potentially predictive of churn. Option (F) is the most pragmatic choice because it captures the information contained...

Timings

Stages

Sampling6072 ms
Embedding632 ms
Signals11 ms
PPV1 ms

Errors

Agent and Round Errors

No errors recorded.