0Pricing
Machine Learning Academy · Ders

Model Seçim Turnuvası: Beş Algoritmayı Karşılaştırma

Öğrenenler, iç içe CV kullanarak işlem hatları içinde lojistik regresyon, rastgele orman, XGBoost, SVM ve bir sinir ağı eğitecek ve performanslarını ortak bir test kümesi üzerinde tablo hâline getireceklerdir.

Model Seçim Turnuvası: Beş Algoritmayı Karşılaştırma, CoddyKit'te ücretsiz bir Machine Learning Academy dersidir. Bu, 4 dersinin 3. dersidir. Aşağıdan dersin tamamını ücretsiz okuyabilir, sonra tarayıcıda yerleşik kod editörü ve 7/24 yapay zeka koçu ile uygulamalı olarak pratik yapabilirsin. Bu, Machine Learning Academy öğrenme yolunun bir parçasıdır ve ilerlemeniz web ve CoddyKit uygulaması arasında senkronize olur. Machine Learning Academy kursu toplamda 4 dersten oluşur.

Bu dersin bazı bölümleri henüz çevrilmemiş olup İngilizce olarak gösterilmektedir.

Why Compare Multiple Algorithms?

No single algorithm dominates every dataset. The No Free Lunch theorem proves that averaged over all possible problems, every algorithm performs equally well — meaning you must empirically compare algorithms on your specific data. A model selection tournament systematically trains and evaluates several diverse algorithms under identical conditions, revealing which one best fits the data's structure. The winner earns the right to hyperparameter tuning and deployment consideration.

Setting Up the Shared Pipeline Scaffold

Fair comparison requires that every algorithm starts from the same preprocessed feature matrix and is evaluated on the same held-out test set. Wrap each algorithm in a Pipeline that includes preprocessing, so preprocessing is fitted only on training folds. Use a fixed random_state everywhere for reproducibility. Keep the test set locked away — do not look at it until the final single evaluation of the tournament winner.

import numpy as np
import pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.model_selection import train_test_split, cross_val_score

# Load preprocessed data
X, y = load_features()  # returns numpy arrays after EDA cleaning

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

print(f'Training set: {X_train.shape[0]} samples')
print(f'Test set (LOCKED): {X_test.shape[0]} samples')
print(f'Positive class rate (train): {y_train.mean():.3f}')

Competitor 1: Logistic Regression

Logistic regression is the essential linear baseline. It is interpretable (coefficients show feature effects), fast to train, and works well when the decision boundary is approximately linear. In the tournament it serves as the floor — if tree-based models cannot outperform it significantly, the dataset may lack complex nonlinear patterns worth capturing with more complex models. Use class_weight='balanced' for imbalanced datasets.

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

lr_pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', LogisticRegression(
        C=1.0,
        class_weight='balanced',
        max_iter=1000,
        random_state=42
    ))
])

cv_scores = cross_val_score(lr_pipe, X_train, y_train, cv=5, scoring='roc_auc')
print(f'Logistic Regression — AUC: {cv_scores.mean():.4f} +/- {cv_scores.std():.4f}')

Competitor 2: Random Forest

Random Forest handles nonlinear relationships, feature interactions, and mixed data types naturally, with no need for feature scaling. It produces reliable feature importance scores and is robust to outliers. In competitions it is often the second-best algorithm after gradient boosting and a strong baseline for tabular data. Set n_estimators=200 and class_weight='balanced' for a solid default configuration.

from sklearn.ensemble import RandomForestClassifier

rf_pipe = Pipeline([
    ('clf', RandomForestClassifier(
        n_estimators=200,
        max_depth=10,
        class_weight='balanced',
        random_state=42,
        n_jobs=-1
    ))
])

cv_scores = cross_val_score(rf_pipe, X_train, y_train, cv=5, scoring='roc_auc')
print(f'Random Forest — AUC: {cv_scores.mean():.4f} +/- {cv_scores.std():.4f}')

Competitor 3: XGBoost

XGBoost is the most commonly cited winner in tabular ML competitions. Its sequential boosting corrects residual errors at each step, producing strong performance on structured data. Key defaults for a tournament run: n_estimators=300, learning_rate=0.05, max_depth=6, and scale_pos_weight set to the negative-to-positive class ratio to handle imbalance. Pass eval_set for optional early stopping.

from xgboost import XGBClassifier
import numpy as np

# Compute class imbalance ratio
neg_count = (y_train == 0).sum()
pos_count = (y_train == 1).sum()

xgb_pipe = Pipeline([
    ('clf', XGBClassifier(
        n_estimators=300,
        learning_rate=0.05,
        max_depth=6,
        scale_pos_weight=neg_count / pos_count,
        use_label_encoder=False,
        eval_metric='logloss',
        random_state=42,
        n_jobs=-1
    ))
])

cv_scores = cross_val_score(xgb_pipe, X_train, y_train, cv=5, scoring='roc_auc')
print(f'XGBoost — AUC: {cv_scores.mean():.4f} +/- {cv_scores.std():.4f}')

Competitor 4: Support Vector Machine

SVM with an RBF kernel excels on datasets with clearly separated classes and few irrelevant features. It is memory-intensive (stores support vectors) and slow on large datasets (O(n²) to O(n³) training time), but can outperform tree models on smaller datasets with complex boundaries. Standardise features before training — SVMs are highly sensitive to feature scale. Use probability=True to get calibrated probability outputs for AUC evaluation.

from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler

svm_pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', SVC(
        C=1.0,
        kernel='rbf',
        gamma='scale',
        class_weight='balanced',
        probability=True,
        random_state=42
    ))
])

cv_scores = cross_val_score(svm_pipe, X_train, y_train, cv=5, scoring='roc_auc')
print(f'SVM (RBF) — AUC: {cv_scores.mean():.4f} +/- {cv_scores.std():.4f}')

Competitor 5: Neural Network

A simple multi-layer perceptron (MLP) with 2-3 hidden layers can capture complex nonlinear patterns, but requires careful scaling, regularisation, and is slower to train than tree-based models on tabular data. Use it when the other four algorithms all plateau at the same performance level, suggesting the decision boundary requires higher-capacity modelling. sklearn.neural_network.MLPClassifier is convenient for the tournament without a full PyTorch setup.

from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler

mlp_pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', MLPClassifier(
        hidden_layer_sizes=(128, 64),
        activation='relu',
        alpha=0.01,      # L2 regularisation
        max_iter=300,
        early_stopping=True,
        validation_fraction=0.1,
        random_state=42
    ))
])

cv_scores = cross_val_score(mlp_pipe, X_train, y_train, cv=5, scoring='roc_auc')
print(f'MLP Neural Network — AUC: {cv_scores.mean():.4f} +/- {cv_scores.std():.4f}')

Consolidating Results into a Leaderboard

Collect all 5-fold CV AUC scores into a table ranked by mean AUC. Also record the standard deviation — a model with slightly lower mean AUC but much lower variance may be preferable for deployment because its performance is more predictable across different data partitions. Display the results as a horizontal bar chart for stakeholder communication.

import pandas as pd
import matplotlib.pyplot as plt

results = {
    'Logistic Regression': (0.821, 0.013),
    'Random Forest':       (0.867, 0.009),
    'XGBoost':             (0.883, 0.007),
    'SVM (RBF)':           (0.845, 0.011),
    'MLP Neural Network':  (0.858, 0.010)
}

df_results = pd.DataFrame(results, index=['AUC_mean', 'AUC_std']).T
df_results = df_results.sort_values('AUC_mean', ascending=False)
print(df_results.to_string())

df_results['AUC_mean'].plot(kind='barh', xerr=df_results['AUC_std'], figsize=(8, 4))
plt.xlabel('5-fold CV AUC')
plt.title('Model Selection Tournament Results')
plt.tight_layout()
plt.savefig('tournament.png', dpi=150)

Statistical Significance of Differences

A 0.002 AUC difference between two models may just be noise from random data partitioning. Use a paired t-test or Wilcoxon signed-rank test on the fold-level scores to determine whether the difference is statistically significant. If the p-value exceeds 0.05, choose the simpler model — its lower complexity makes it easier to debug, explain, and maintain in production.

from scipy import stats
import numpy as np

# Simulated per-fold scores for XGBoost vs Random Forest
xgb_folds = np.array([0.889, 0.881, 0.875, 0.883, 0.887])
rf_folds  = np.array([0.871, 0.863, 0.869, 0.865, 0.867])

# Paired t-test
t_stat, p_value = stats.ttest_rel(xgb_folds, rf_folds)
print(f'XGBoost mean: {xgb_folds.mean():.4f}')
print(f'Random Forest mean: {rf_folds.mean():.4f}')
print(f'Paired t-test: t={t_stat:.3f}, p={p_value:.4f}')
if p_value < 0.05:
    print('Difference is statistically significant — prefer XGBoost.')
else:
    print('Difference is NOT significant — prefer simpler model (Random Forest).')

One-Time Test Set Evaluation

After selecting the tournament winner, evaluate it exactly once on the held-out test set. This is the unbiased performance estimate you will report in the model card. Never tune hyperparameters after seeing test set results — doing so constitutes data leakage through the evaluation process. If the test AUC is substantially lower than CV AUC (more than 2–3 standard deviations), investigate for overfitting or distribution shift between train and test splits.

from sklearn.metrics import roc_auc_score, classification_report

# Fit winner on full training set
best_pipeline = xgb_pipe
best_pipeline.fit(X_train, y_train)

# One-time test evaluation
y_proba = best_pipeline.predict_proba(X_test)[:, 1]
y_pred  = best_pipeline.predict(X_test)

test_auc = roc_auc_score(y_test, y_proba)
print(f'Test AUC: {test_auc:.4f}')
print('\nClassification report:')
print(classification_report(y_test, y_pred, target_names=['retained', 'churned']))

Choosing the Winner: Beyond AUC

AUC is not the only criterion. Consider: interpretability (logistic regression may be mandatory for regulatory compliance), training time (if weekly retraining on 10M rows, XGBoost's speed advantage matters), memory footprint (an SVM storing 100k support vectors may exceed the deployment budget), and fairness (the highest-AUC model may have worse demographic parity). The tournament selects the candidate; deployment readiness is a separate, multi-criteria decision.

# Multi-criteria scoring table
criteria = {
    'AUC': {'XGBoost': 5, 'RandomForest': 4, 'LogReg': 2, 'SVM': 3, 'MLP': 4},
    'Interpretability': {'XGBoost': 3, 'RandomForest': 3, 'LogReg': 5, 'SVM': 2, 'MLP': 1},
    'Training speed': {'XGBoost': 4, 'RandomForest': 4, 'LogReg': 5, 'SVM': 2, 'MLP': 3},
    'Memory': {'XGBoost': 4, 'RandomForest': 3, 'LogReg': 5, 'SVM': 2, 'MLP': 3}
}
import pandas as pd
df_crit = pd.DataFrame(criteria).T
df_crit.loc['Total'] = df_crit.sum()
print(df_crit)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: every algorithm should be wrapped in a Pipeline and evaluated with identical CV folds on the same training set, the tournament leaderboard ranks models by mean CV AUC with a paired significance test to distinguish real from noise differences, and the test set is used exactly once on the winning model to produce the honest performance estimate for the model card. Next up we package, document, and present the final model for deployment.

Sıkça Sorulan Sorular

“Model Seçim Turnuvası: Beş Algoritmayı Karşılaştırma” dersi ücretsiz mi?

Evet — “Model Seçim Turnuvası: Beş Algoritmayı Karşılaştırma” dersin tüm metni burada web'de ücretsiz olarak okunabilir. Etkileşimli olarak pratik yapmak (yerleşik kod editörü ve 7/24 yapay zeka koçu) ve Machine Learning Academy kursunun geri kalanını açmak için CoddyKit PRO'ya yükselt. Machine Learning Academy kursu toplamda 4 dersten oluşur.

“Model Seçim Turnuvası: Beş Algoritmayı Karşılaştırma” dersinde ne öğreneceğim?

Öğrenenler, iç içe CV kullanarak işlem hatları içinde lojistik regresyon, rastgele orman, XGBoost, SVM ve bir sinir ağı eğitecek ve performanslarını ortak bir test kümesi üzerinde tablo hâline getire… Machine Learning Academy ile uygulamalı kodu tarayıcıda doğrudan çalıştırarak pratik yaparsın ve 7/24 yapay zeka koçu dersi çalışırken sorularını yanıtlar.

Machine Learning Academy öğrenmeye başlamak için deneyim gerekli mi?

Önceden deneyim gerekmez. CoddyKit'te Machine Learning Academy, başlangıçtan ileri seviyeye kadar yapılandırıldığı için buradan başlayabilir veya başından başlayıp kendi hızında ilerleme yapabilirsin. Bu, 4 dersinin 3. dersidir.

“Model Seçim Turnuvası: Beş Algoritmayı Karşılaştırma” dersi ne kadar sürer?

Çoğu CoddyKit dersi yaklaşık 5–10 dakika sürer. Her biri kısa ve etkileşimli olduğu için sabit ilerleme yaparsın ve web ile uygulama arasında tam olarak bıraktığın yerden devam edebilirsin.

Bu Machine Learning Academy dersinde kod yazıp çalıştırabilir miyim?

Evet. Her Machine Learning Academy dersi yerleşik bir kod editörü içerir, bu sayede tarayıcıda gerçek kod yazıp çalıştırabilir ve anlık yapay zeka geri bildirimi alırsın — yerel kurulum gerekli değildir.

Bu kursun tüm dersleri

  1. Proje Kapsamını Belirleme: Problemi ve Başarı Ölçütlerini Tanımlama
  2. Veri Düzenleme ve Keşifsel Veri Analizi
  3. Model Seçim Turnuvası: Beş Algoritmayı Karşılaştırma
  4. Nihai Modeli Paketleme, Belgeleme ve Sunma
← Machine Learning Academy Sayfasına Dön