Algorithm Showdown
Knowing how each algorithm works is necessary but not sufficient. The test of real understanding is being able to look at a scenario — a dataset, a prediction task, a set of constraints — and reason through which approaches will work, which will fail, and precisely why. This lesson is a structured practice session doing exactly that. For each scenario, the correct answer is less important than the reasoning process. Think carefully, use the frameworks from the previous lessons, and be willing to argue for your position.
How to Think Through a Scenario
Before jumping to an answer, work through a mental checklist: 1. Task type: Classification (binary/multiclass) or Regression? 2. Dataset profile: What is N? What is D? Are features numerical, categorical, or mixed? 3. Constraints: Is there an interpretability requirement? A latency budget? A labeling budget? 4. Data quality signals: Is there class imbalance? Likely noise? Missing values? 5. Evaluation metric: What does success actually mean here — accuracy, RMSE, recall, precision? Once you have these answers, match them against each algorithm's known strengths: - Logistic/Linear Regression: fast, interpretable, linear boundaries, needs feature scaling, regularization helps - Decision Tree: interpretable, handles mixed features, overfits easily without pruning - Random Forest: robust, handles high D well, needs no feature scaling, slow at large N - Gradient Boosting: state-of-the-art for tabular data, sensitive to tuning, opaque - k-NN: no training cost, lazy, degrades in high D, needs feature scaling, slow at prediction time With these frameworks in mind, work through the five scenarios below.
In practice, two competent data scientists analyzing the same problem often choose different algorithms and both perform well. What matters is coherent reasoning: you can explain what properties of the algorithm match the problem, you have a plan to validate the choice empirically, and you know what would change your recommendation.
Scenario A: Detecting wildfires from satellite imagery meta-features. Dataset: 15,000 images, each represented by 12 extracted features (smoke density index, temperature anomaly, humidity ratio, wind speed, etc.). Target: fire detected (1) or not (0). Class distribution: 8% positive. Requirement: government agency must justify every positive alert to a deployment commander in plain English. Analysis: - Binary classification. - N=15,000, D=12. Medium N, low D. - 8% positive class — moderately imbalanced. Optimize recall (missing a fire is catastrophic). - Interpretability required — must justify every alert. - Decision tree (depth 4-6) or logistic regression. A shallow decision tree produces explicit if-then rules. Logistic regression provides feature coefficients. Random Forest loses the per-prediction explanation chain. - Recommendation: shallow decision tree tuned for recall (lower the classification threshold or apply class weighting). Backup: logistic regression with L2 regularization. Scenario B: Estimating delivery time for an e-commerce platform. Dataset: 2 million past orders, 28 features (origin, destination region, package weight, carrier, time of year, whether it's a holiday). Target: actual delivery days (continuous). No interpretability requirement. Prediction must complete in < 20ms. Analysis: - Regression. - N=2M, D=28. Large N. - No interpretability requirement. - 20ms inference budget. Gradient boosting inference is fast (traverse a set of trees). k-NN would be O(N) per prediction — 2 million comparisons per query is too slow. - Recommendation: Gradient Boosting (LightGBM) — handles large N efficiently, top accuracy on tabular regression, fast at inference. Feature engineering (encode region, carrier as embeddings or one-hot) is worthwhile. Scenario C: Classifying handwritten exam answers as correct or incorrect. Dataset: 400 labeled exam papers, each answer represented by 6 features (length, keyword count, coherence score, grammar score, topic match, formatting score). Binary target. No latency constraint. Teacher wants to understand what the model is doing. Analysis: - Binary classification. - N=400, D=6. Small N. - Small N means complex models will overfit. Logistic regression and shallow decision trees are appropriate. - Teacher interpretability desired — logistic regression coefficients are meaningful ('each additional keyword adds X to the log-odds of correctness'). - Recommendation: logistic regression with L2 regularization. Cross-validate with 5 folds given small N.
It is tempting to reach for gradient boosting or deep learning simply because they win benchmarks. But a benchmark is not your problem. On small datasets, a well-tuned logistic regression frequently equals or beats XGBoost. On problems with 5 features and 300 examples, a 1,000-tree Random Forest is overkill. Always test the simple baseline first — the complexity premium must pay for itself in measured performance on your specific data.
Match each problem profile to the algorithm most likely to perform best.
Terms
Definitions
Drag terms onto their definitions, or click a term then click a definition to match.
A model trained to predict customer churn achieves 91% accuracy. The dataset has 91% non-churners and 9% churners. A colleague argues this is excellent performance. What is the most important follow-up analysis?
You are choosing between a Random Forest and k-NN for a dataset with D=600 features and N=50,000 examples. Which is more likely to succeed, and what is the primary reason?
Scenario Design and Showdown
- Work in groups of three. Each group will design one original scenario AND solve one from another group.
- Part 1 — Design (15 minutes):
- Create a realistic supervised learning scenario. Specify:
- a. The prediction task and target variable (be precise)
- b. Approximate N and D
- c. One meaningful constraint (interpretability, latency, label cost, etc.)
- d. The metric you would use to evaluate success and why
- Write a brief 'intended solution' — which algorithm you think is best and why.
- Part 2 — Exchange:
- Pass your scenario to another group. Solve the scenario you received using the five-dimension checklist. Do not see the intended solution.
- Part 3 — Debrief:
- Compare your solution to the intended one. Where do you agree? Where do you disagree? If you disagreed, which argument is more defensible — and why?
- The goal is not to get the 'right' answer. The goal is to produce well-reasoned arguments that could be defended in a professional review.