⚙️ Week 8 – Advanced ML · दिन 45

दिन 45: Pipelines, Cross-Validation और Tuning

⏱ 2.5 hours📶 Advanced🎯 100% self-study

दिन 45 / 60

✅ आज आप सीखेंगे

Pipelines: preprocessing + model एक object में
एक train/test split क्यों काफ़ी नहीं
k-fold और stratified cross-validation
Grid और randomised hyper-parameter search
Train → validate → final test, सही क्रम

🏢 असली दुनिया में

आपका loan model एक split पर 0.83 accuracy और दूसरे पर 0.79 देता है। सच कौन सा? और production server पर ले जाते समय कोई inputs scale करना भूल जाता है और हर prediction ग़लत हो जाती है। Pipelines और cross-validation दोनों समस्याएँ हल करते हैं – यही notebook experiment और production model का फ़र्क है।

Pipelines

एक object सब करता है – एक बार fit, कहीं भी इस्तेमाल

Python

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline, make_pipeline
l = pd.read_csv("loan_applications.csv")
l["lti"] = l["loan_amount"] / (l["monthly_income"] * 12)
num = ["monthly_income", "credit_score", "loan_amount", "years_employed", "lti"]
cat = ["employment"]
X, y = l[num + cat], l["approved"]
pre = ColumnTransformer([("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num),
                         ("cat", OneHotEncoder(handle_unknown="ignore"), cat)])
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
model = Pipeline([("prep", pre), ("clf", LogisticRegression(max_iter=2000))])
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
model.fit(X_tr, y_tr)
print("test accuracy:", round(model.score(X_te, y_te), 3))
new = pd.DataFrame([{"monthly_income": 52000, "credit_score": 745, "loan_amount": 600000,
                     "years_employed": 4, "lti": 600000 / (52000 * 12), "employment": "Salaried"}])
print("P(approve) for a raw new row:", round(model.predict_proba(new)[0, 1], 3))

test accuracy: 0.829 P(approve) for a raw new row: 0.861

💡 Pipelines क्यों?

Scaler और encoder हर बार सिर्फ़ training हिस्से पर fit होते हैं – कोई leakage नहीं। नया raw data सीधे अंदर जाता है। और पूरा object deployment (Day 50) के लिए एक file में save होता है।

एक split क़िस्मत है

Python

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline, make_pipeline
l = pd.read_csv("loan_applications.csv")
l["lti"] = l["loan_amount"] / (l["monthly_income"] * 12)
num = ["monthly_income", "credit_score", "loan_amount", "years_employed", "lti"]
cat = ["employment"]
X, y = l[num + cat], l["approved"]
pre = ColumnTransformer([("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num),
                         ("cat", OneHotEncoder(handle_unknown="ignore"), cat)])
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
scores = []
for seed in range(10):
    a, b, c, d = train_test_split(X, y, test_size=0.2, random_state=seed, stratify=y)
    scores.append(Pipeline([("prep", pre), ("clf", LogisticRegression(max_iter=2000))]).fit(a, c).score(b, d))
print("10 different splits:", [round(s, 3) for s in scores])
print(f"range {min(scores):.3f} – {max(scores):.3f}")

10 different splits: [0.817, 0.842, 0.808, 0.783, 0.825, 0.8, 0.817, 0.858, 0.85, 0.846] range 0.783 – 0.858

k-fold cross-validation

Python

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline, make_pipeline
l = pd.read_csv("loan_applications.csv")
l["lti"] = l["loan_amount"] / (l["monthly_income"] * 12)
num = ["monthly_income", "credit_score", "loan_amount", "years_employed", "lti"]
cat = ["employment"]
X, y = l[num + cat], l["approved"]
pre = ColumnTransformer([("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num),
                         ("cat", OneHotEncoder(handle_unknown="ignore"), cat)])
from sklearn.model_selection import cross_val_score, StratifiedKFold, cross_validate
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
models = {"logistic": LogisticRegression(max_iter=2000),
          "random forest": RandomForestClassifier(n_estimators=300, min_samples_leaf=3, random_state=0, n_jobs=-1),
          "boosting": HistGradientBoostingClassifier(learning_rate=0.05, max_iter=200, random_state=0)}
for name, clf in models.items():
    res = cross_validate(Pipeline([("prep", pre), ("clf", clf)]), X, y, cv=cv, scoring=["accuracy", "roc_auc", "f1"])
    print(f"{name:14s} acc {res['test_accuracy'].mean():.3f} ± {res['test_accuracy'].std():.3f} | "
          f"AUC {res['test_roc_auc'].mean():.3f} | F1 {res['test_f1'].mean():.3f}")

logistic acc 0.809 ± 0.016 | AUC 0.905 | F1 0.793 random forest acc 0.820 ± 0.026 | AUC 0.886 | F1 0.803 boosting acc 0.803 ± 0.025 | AUC 0.877 | F1 0.785

🧠 mean ± std बताएँ

0.83 ± 0.02 जैसा CV score business को अपेक्षित performance और उसकी स्थिरता दोनों बताता है। जिन दो models की ranges overlap करें वे लगभग बराबर हैं – सरल वाला चुनें।

Hyper-parameter tuning

Python

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline, make_pipeline
l = pd.read_csv("loan_applications.csv")
l["lti"] = l["loan_amount"] / (l["monthly_income"] * 12)
num = ["monthly_income", "credit_score", "loan_amount", "years_employed", "lti"]
cat = ["employment"]
X, y = l[num + cat], l["approved"]
pre = ColumnTransformer([("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num),
                         ("cat", OneHotEncoder(handle_unknown="ignore"), cat)])
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
pipe = Pipeline([("prep", pre), ("clf", RandomForestClassifier(random_state=0, n_jobs=-1))])
grid = {"clf__n_estimators": [200], "clf__max_depth": [4, 8, None], "clf__min_samples_leaf": [1, 5, 20]}
gs = GridSearchCV(pipe, grid, cv=StratifiedKFold(5, shuffle=True, random_state=0), scoring="roc_auc", n_jobs=-1).fit(X, y)
print("best params:", gs.best_params_)
print("best CV AUC:", round(gs.best_score_, 3))
res = pd.DataFrame(gs.cv_results_)[["param_clf__max_depth", "param_clf__min_samples_leaf", "mean_test_score"]]
res["param_clf__max_depth"] = res["param_clf__max_depth"].astype(object).where(res["param_clf__max_depth"].notna(), "None").astype(str)
print(res.pivot(index="param_clf__max_depth", columns="param_clf__min_samples_leaf", values="mean_test_score").round(3))

best params: {'clf__max_depth': 8, 'clf__min_samples_leaf': 20, 'clf__n_estimators': 200} best CV AUC: 0.891 param_clf__min_samples_leaf 1 5 20 param_clf__max_depth 4 0.890 0.891 0.890 8 0.885 0.889 0.891 None 0.882 0.888 0.891

💡 Double underscore

Pipeline के अंदर parameters का नाम step__parameter होता है: clf__max_depth मतलब “clf नाम के step का max_depth”। Preprocessing भी tune कर सकते हैं: prep__num__simpleimputer__strategy।

Python

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline, make_pipeline
l = pd.read_csv("loan_applications.csv")
l["lti"] = l["loan_amount"] / (l["monthly_income"] * 12)
num = ["monthly_income", "credit_score", "loan_amount", "years_employed", "lti"]
cat = ["employment"]
X, y = l[num + cat], l["approved"]
pre = ColumnTransformer([("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num),
                         ("cat", OneHotEncoder(handle_unknown="ignore"), cat)])
from scipy.stats import loguniform, randint
from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold
from sklearn.ensemble import HistGradientBoostingClassifier
pipe = Pipeline([("prep", pre), ("clf", HistGradientBoostingClassifier(random_state=0))])
space = {"clf__learning_rate": loguniform(0.01, 0.3), "clf__max_leaf_nodes": randint(8, 64),
         "clf__min_samples_leaf": randint(5, 60), "clf__max_iter": randint(100, 500)}
rs = RandomizedSearchCV(pipe, space, n_iter=25, cv=StratifiedKFold(5, shuffle=True, random_state=0),
                        scoring="roc_auc", random_state=0, n_jobs=-1).fit(X, y)
print("best CV AUC:", round(rs.best_score_, 3))
print({k.replace("clf__", ""): (float(round(v, 3)) if isinstance(v, float) else int(v)) for k, v in rs.best_params_.items()})

best CV AUC: 0.892 {'learning_rate': 0.014, 'max_iter': 157, 'max_leaf_nodes': 43, 'min_samples_leaf': 43}

GridSearchCV

हर combination आज़माता है
2–3 parameters, कम values के लिए अच्छा
Cost गुणा में बढ़ती है

RandomizedSearchCV

n_iter random combinations लेता है
कई parameters / ranges के लिए बेहतर
अच्छी settings कहीं जल्दी ढूँढता है

Professional protocol

1आख़िरी test set (20%) अलग करके बंद रखें
2बचे 80% पर: CV से models की तुलना
3सबसे अच्छे 1–2 को CV search से tune
4चुनी pipeline को पूरे 80% पर फिर fit
5बंद test set पर एक बार score करके report

Test set पर कभी tune न करें

Python

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline, make_pipeline
l = pd.read_csv("loan_applications.csv")
l["lti"] = l["loan_amount"] / (l["monthly_income"] * 12)
num = ["monthly_income", "credit_score", "loan_amount", "years_employed", "lti"]
cat = ["employment"]
X, y = l[num + cat], l["approved"]
pre = ColumnTransformer([("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num),
                         ("cat", OneHotEncoder(handle_unknown="ignore"), cat)])
from sklearn.model_selection import train_test_split, GridSearchCV, StratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, classification_report
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=0.2, random_state=7, stratify=y)
pipe = Pipeline([("prep", pre), ("clf", LogisticRegression(max_iter=2000))])
gs = GridSearchCV(pipe, {"clf__C": [0.001, 0.01, 0.1, 1, 10]}, cv=StratifiedKFold(5, shuffle=True, random_state=0), scoring="roc_auc").fit(X_dev, y_dev)
final = gs.best_estimator_                                  # refit on all dev data automatically
p = final.predict_proba(X_test)[:, 1]
print("chosen C:", gs.best_params_["clf__C"], "| CV AUC:", round(gs.best_score_, 3), "| final TEST AUC:", round(roc_auc_score(y_test, p), 3))
print(classification_report(y_test, p > 0.5, digits=3))

chosen C: 0.01 | CV AUC: 0.905 | final TEST AUC: 0.918 precision recall f1-score support 0 0.793 0.922 0.853 129 1 0.889 0.721 0.796 111 accuracy 0.829 240 macro avg 0.841 0.822 0.825 240 weighted avg 0.838 0.829 0.827 240

Learning curve: क्या ज़्यादा data मदद करेगा?

Python

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline, make_pipeline
l = pd.read_csv("loan_applications.csv")
l["lti"] = l["loan_amount"] / (l["monthly_income"] * 12)
num = ["monthly_income", "credit_score", "loan_amount", "years_employed", "lti"]
cat = ["employment"]
X, y = l[num + cat], l["approved"]
pre = ColumnTransformer([("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num),
                         ("cat", OneHotEncoder(handle_unknown="ignore"), cat)])
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
pipe = Pipeline([("prep", pre), ("clf", RandomForestClassifier(n_estimators=200, min_samples_leaf=5, random_state=0, n_jobs=-1))])
sizes, tr, va = learning_curve(pipe, X, y, cv=StratifiedKFold(5, shuffle=True, random_state=0), scoring="roc_auc",
                               train_sizes=np.linspace(0.1, 1.0, 8), n_jobs=-1)
plt.figure(figsize=(8.5, 3.8))
plt.plot(sizes, tr.mean(1), marker="o", label="training", color="#94a3b8")
plt.plot(sizes, va.mean(1), marker="o", label="cross-validation", color="#4f46e5")
plt.fill_between(sizes, va.mean(1) - va.std(1), va.mean(1) + va.std(1), alpha=0.2, color="#4f46e5")
plt.xlabel("training rows"); plt.ylabel("AUC"); plt.legend(); plt.title("Validation AUC is still rising → more data would help a little")
plt.show()

Training size के साथ training और validation AUC का learning curve — बड़ा अंतर = overfitting (ज़्यादा data या सरल model मदद करेगा)। दोनों नीचे = underfitting (बेहतर features मदद करेंगे)।

🏅 आज के Job Skills

✔ ColumnTransformer के साथ Pipelines बनाना✔ Stratified k-fold CV से models evaluate करना✔ GridSearchCV और RandomizedSearchCV से tune करना✔ Leakage-free train/validate/test protocol अपनाना

Practice tasks

🏋️ Practice – ख़ुद करें

Task 1: House prices के लिए pipeline बनाएँ (numbers scale, city/furnishing/parking one-hot, Ridge) और 5-fold CV MAE बताएँ।

Solution देखें

Python

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score
h = pd.read_csv("house_prices.csv")
num = ["area_sqft", "bedrooms", "bathrooms", "age_years", "metro_km"]; cat = ["city", "furnishing", "parking"]
pipe = Pipeline([("prep", ColumnTransformer([("n", StandardScaler(), num), ("c", OneHotEncoder(handle_unknown="ignore"), cat)])), ("m", Ridge(alpha=1))])
s = -cross_val_score(pipe, h[num + cat], h["price_lakh"], cv=5, scoring="neg_mean_absolute_error")
print(f"CV MAE ₹{s.mean():.2f} ± {s.std():.2f} lakh")

CV MAE ₹4.77 ± 0.31 lakh

Task 2: Loan pipeline के अंदर KNN का n_neighbors (3–51) GridSearchCV से tune करें। सबसे अच्छा k?

Solution देखें

Python

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline, make_pipeline
l = pd.read_csv("loan_applications.csv")
l["lti"] = l["loan_amount"] / (l["monthly_income"] * 12)
num = ["monthly_income", "credit_score", "loan_amount", "years_employed", "lti"]
cat = ["employment"]
X, y = l[num + cat], l["approved"]
pre = ColumnTransformer([("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num),
                         ("cat", OneHotEncoder(handle_unknown="ignore"), cat)])
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV
gs = GridSearchCV(Pipeline([("prep", pre), ("knn", KNeighborsClassifier())]), {"knn__n_neighbors": list(range(5, 160, 10))}, cv=5, scoring="roc_auc").fit(X, y)
print(gs.best_params_, round(gs.best_score_, 3))

{'knn__n_neighbors': 95} 0.894

Task 3: दो lines में समझाएँ कि test set पर tune करने से score ज़्यादा आशावादी क्यों आता है।

Quick quiz

🧠 Quick Quiz – ख़ुद को जाँचें

1. Scaler को Pipeline में रखने का मुख्य फ़ायदा?

सिर्फ़ training folds पर fit।

2. 5-fold CV model को कितनी बार train करता है?

हर fold पर एक बार।

3. "clf" नाम के step के max_depth का parameter नाम?

Double underscore।

4. RandomizedSearchCV कब GridSearchCV से बेहतर?

बड़े spaces को कुशलता से cover करता है।

5. आख़िरी test set कब इस्तेमाल करें?

वरना वह अनदेखा नहीं रहता।

Interview corner

💼 Interview Corner

Cross-validation क्या है और क्यों?

Data को k folds में बाँटकर k−1 पर train और बचे एक पर validate, k बार घुमाकर। यह एक split से ज़्यादा भरोसेमंद अनुमान (mean ± std) देता है और सारा data training व validation दोनों में इस्तेमाल होता है।

Leakage के बिना hyper-parameters कैसे tune करेंगे?

पहले test set अलग करें; बचे data पर stratified CV के साथ Pipeline पर GridSearchCV/RandomizedSearchCV चलाएँ; सबसे अच्छी pipeline फिर fit करें; test set पर एक बार evaluate करें। Time series में TimeSeriesSplit लें।

सारांश

Pipeline + ColumnTransformer = leak-free, deploy करने लायक models।
Stratified k-fold CV mean ± std देता है, एक क़िस्मत वाला number नहीं।
छोटे grids के लिए GridSearchCV, बड़े spaces के लिए RandomizedSearchCV।
Test set बंद रखें, CV से tune करें, एक बार test करें।

➡️ आगे

Day 46: unsupervised learning – K-Means clustering से customer segments ढूँढें।

अक्सर पूछे जाने वाले प्रश्न (FAQ)

कितने folds?

5 या 10 आम हैं। छोटे data पर ज़्यादा, बहुत बड़े data या धीमे models पर कम।

Tuning में बहुत समय लगता है – सुझाव?

कम iterations वाला RandomizedSearchCV, n_jobs=-1, पहली बार छोटा sample, और धीमे models की जगह HistGradientBoosting लें।

← Data Science (60 Days)

दिन 45: Pipelines, Cross-Validation और Tuning

✅ आज आप सीखेंगे

🏢 असली दुनिया में

Pipelines

एक split क़िस्मत है

k-fold cross-validation

Hyper-parameter tuning

GridSearchCV

RandomizedSearchCV

Professional protocol

Learning curve: क्या ज़्यादा data मदद करेगा?

🏅 आज के Job Skills

Practice tasks

🏋️ Practice – ख़ुद करें

Quick quiz

🧠 Quick Quiz – ख़ुद को जाँचें

Interview corner

💼 Interview Corner

सारांश

अक्सर पूछे जाने वाले प्रश्न (FAQ)

यह lesson कैसा लगा?

पढ़ने वालों के reviews

Video से सीखें – आसान Hindi में