🤖 Week 7 – Machine Learning Core · Day 41
Day 41: Confusion Matrix, Precision, Recall, ROC
⏱ 2.5 hours📶 Intermediate🎯 100% self-study
Day 41 / 60
✅ Today you will learn
- Confusion matrix: TP, FP, FN, TN
- The accuracy paradox on imbalanced data
- Precision, recall and F1 – when each matters
- ROC-AUC and precision-recall curves
- Pick the threshold that minimises business cost
🏢 In the real world
Only 2 in 100 UPI transactions are fraud. A model that says “never fraud” is 98% accurate – and completely useless. The payments team needs to know: of the frauds, how many do we catch? Of our alerts, how many are real? Today you learn to answer both.
Python – create the practice data
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=20000, n_features=8, n_informative=5, weights=[0.98], class_sep=1.2, random_state=7)
cols = ["amount_z", "hour_z", "device_age", "geo_dist", "merchant_risk", "txn_last_hr", "card_age", "velocity"]
pd.DataFrame(X, columns=cols).assign(is_fraud=y).to_csv("upi_transactions.csv", index=False)
print("Saved upi_transactions.csv | fraud rate:", round(y.mean() * 100, 2), "%")Saved upi_transactions.csv | fraud rate: 2.55 %
The confusion matrix
| Predicted fraud | Predicted normal | |
|---|---|---|
| Actually fraud | TP – caught ✅ | FN – missed 💸 |
| Actually normal | FP – false alarm 😠 | TN – correctly ignored ✅ |
Python
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.dummy import DummyClassifier
from sklearn.metrics import confusion_matrix, accuracy_score
df = pd.read_csv("upi_transactions.csv")
X, y = df.drop(columns="is_fraud"), df["is_fraud"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
dummy = DummyClassifier(strategy="most_frequent").fit(X_tr, y_tr)
model = LogisticRegression(max_iter=2000).fit(X_tr, y_tr)
for name, m in [("always 'normal'", dummy), ("logistic regression", model)]:
p = m.predict(X_te)
tn, fp, fn, tp = confusion_matrix(y_te, p).ravel()
print(f"{name:20s} accuracy={accuracy_score(y_te, p):.3f} TP={tp} FN={fn} FP={fp} TN={tn}")always 'normal' accuracy=0.975 TP=0 FN=153 FP=0 TN=5847
logistic regression accuracy=0.978 TP=20 FN=133 FP=1 TN=5846
⚠️ The accuracy paradox
Both models have ~98% accuracy, yet the dummy catches zero frauds. On imbalanced data, accuracy hides everything that matters.
Precision, recall, F1
🎯
Precision = TP / (TP + FP)Of our alerts, how many are real fraud? Low precision = annoyed customers.🕸️
Recall = TP / (TP + FN)Of all frauds, how many did we catch? Low recall = money lost.⚖️
F1 = 2PR / (P + R)Harmonic mean – high only when both are high.Python
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, ConfusionMatrixDisplay
import matplotlib.pyplot as plt
df = pd.read_csv("upi_transactions.csv")
X, y = df.drop(columns="is_fraud"), df["is_fraud"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
m = LogisticRegression(max_iter=2000).fit(X_tr, y_tr)
print(classification_report(y_te, m.predict(X_te), target_names=["normal", "fraud"], digits=3))
ConfusionMatrixDisplay.from_estimator(m, X_te, y_te, display_labels=["normal", "fraud"], cmap="Blues", colorbar=False)
plt.title("Confusion matrix at threshold 0.5"); plt.show() precision recall f1-score support
normal 0.978 1.000 0.989 5847
fraud 0.952 0.131 0.230 153
accuracy 0.978 6000
macro avg 0.965 0.565 0.609 6000
weighted avg 0.977 0.978 0.969 6000
| Situation | Priority | Why |
|---|---|---|
| Fraud, cancer screening | Recall | Missing a case is very costly |
| Spam filter, legal notices | Precision | A false alarm hides an important email |
| Loan approval | Precision of “approve” | A bad loan costs lakhs |
| Churn campaign | Balance (F1) + cost | Calls cost money, lost customers cost more |
ROC curve and AUC
Python
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score, RocCurveDisplay, PrecisionRecallDisplay, average_precision_score
df = pd.read_csv("upi_transactions.csv")
X, y = df.drop(columns="is_fraud"), df["is_fraud"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
models = {"Logistic": LogisticRegression(max_iter=2000), "Random forest": RandomForestClassifier(n_estimators=200, random_state=0, n_jobs=-1)}
fig, ax = plt.subplots(1, 2, figsize=(11, 4.2))
for name, m in models.items():
m.fit(X_tr, y_tr); p = m.predict_proba(X_te)[:, 1]
RocCurveDisplay.from_predictions(y_te, p, name=f"{name} (AUC={roc_auc_score(y_te, p):.3f})", ax=ax[0])
PrecisionRecallDisplay.from_predictions(y_te, p, name=f"{name} (AP={average_precision_score(y_te, p):.3f})", ax=ax[1])
ax[0].plot([0, 1], [0, 1], "k:", label="random"); ax[0].set_title("ROC curve"); ax[0].legend(fontsize=8)
ax[1].set_title("Precision–recall curve (better for rare events)"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
🧠 Reading AUC
0.5 = random guessing, 0.7 = fair, 0.8 = good, 0.9+ = excellent. AUC does not depend on the threshold – it measures how well the model ranks cases.
Choose the threshold by cost
Python
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import precision_score, recall_score
df = pd.read_csv("upi_transactions.csv")
X, y = df.drop(columns="is_fraud"), df["is_fraud"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
p = RandomForestClassifier(n_estimators=200, random_state=0, n_jobs=-1).fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
COST_FN, COST_FP = 8000, 150 # missed fraud ₹8,000; OTP check / annoyed customer ₹150
print("thr precision recall missed alarms total cost")
best = None
for t in [0.02, 0.05, 0.1, 0.15, 0.2, 0.3, 0.5, 0.7]:
pred = (p >= t).astype(int)
fn = int(((pred == 0) & (y_te == 1)).sum()); fp = int(((pred == 1) & (y_te == 0)).sum())
cost = fn * COST_FN + fp * COST_FP
best = min(best or (cost, t), (cost, t))
print(f"{t:.2f} {precision_score(y_te, pred):.2f} {recall_score(y_te, pred):.2f} {fn:4d} {pred.sum():5d} ₹{cost:>9,}")
print(f"Cheapest threshold: {best[1]} (₹{best[0]:,})")thr precision recall missed alarms total cost
0.02 0.10 0.80 30 1288 ₹ 414,750
0.05 0.22 0.75 39 513 ₹ 371,850
0.10 0.41 0.69 47 261 ₹ 399,250
0.15 0.53 0.67 51 191 ₹ 421,350
0.20 0.62 0.61 60 150 ₹ 488,550
0.30 0.82 0.57 66 106 ₹ 530,850
0.50 0.93 0.44 85 73 ₹ 680,750
0.70 0.96 0.30 107 48 ₹ 856,300
Cheapest threshold: 0.05 (₹371,850)
💼 Business wins over F1
Because a missed fraud costs 50× more than a false alarm, the cheapest threshold is well below 0.5 – we accept more OTP checks to catch more fraud. Always translate errors into rupees.
Handling class imbalance
Python
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score, precision_score
df = pd.read_csv("upi_transactions.csv")
X, y = df.drop(columns="is_fraud"), df["is_fraud"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
for cw in [None, "balanced"]:
p = LogisticRegression(max_iter=2000, class_weight=cw).fit(X_tr, y_tr).predict(X_te)
print(f"class_weight={str(cw):9s} recall={recall_score(y_te, p):.2f} precision={precision_score(y_te, p):.2f}")class_weight=None recall=0.13 precision=0.95
class_weight=balanced recall=0.75 precision=0.08
- class_weight="balanced" – make mistakes on the rare class cost more during training.
- Threshold tuning – often the simplest and best fix.
- Resampling – undersample the majority or oversample the minority (imbalanced-learn’s SMOTE), only on training data.
- Use PR-AUC / recall / precision, never accuracy alone.
🏅 Job skills unlocked today
✔ Read a confusion matrix✔ Compute and choose precision, recall, F1✔ Compare models with ROC-AUC and PR curves✔ Pick thresholds with a cost matrix and handle imbalance
Practice tasks
🏋️ Practice – do it yourself
Task 1: For the loan model (Day 40), print the classification report and ROC-AUC.
Show solution
Python
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score
l = pd.read_csv("loan_applications.csv"); l["lti"] = l["loan_amount"] / (l["monthly_income"] * 12)
X, y = l[["credit_score", "lti", "years_employed"]], l["approved"]
a, b, c, d = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
m = LogisticRegression(max_iter=5000).fit(a, c)
print(classification_report(d, m.predict(b), digits=3)); print("AUC", round(roc_auc_score(d, m.predict_proba(b)[:, 1]), 3)) precision recall f1-score support
0 0.852 0.820 0.835 161
1 0.800 0.835 0.817 139
accuracy 0.827 300
macro avg 0.826 0.827 0.826 300
weighted avg 0.828 0.827 0.827 300
AUC 0.913
Task 2: A bank loses ₹2,00,000 on a bad approved loan and ₹15,000 of profit on a wrongly rejected good loan. Find the cheapest threshold.
Show solution
Python
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
l = pd.read_csv("loan_applications.csv"); l["lti"] = l["loan_amount"] / (l["monthly_income"] * 12)
X, y = l[["credit_score", "lti", "years_employed"]], l["approved"]
a, b, c, d = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
p = LogisticRegression(max_iter=5000).fit(a, c).predict_proba(b)[:, 1]
costs = {t: int(((p >= t) & (d == 0)).sum() * 200000 + ((p < t) & (d == 1)).sum() * 15000) for t in np.arange(0.3, 1.0, 0.025).round(3)}
print(min(costs, key=costs.get), "→ ₹", min(costs.values()))0.925 → ₹ 1455000
Task 3: Compute precision, recall and F1 by hand from TP=40, FP=60, FN=10.
Show solution
Python
tp, fp, fn = 40, 60, 10 p, r = tp / (tp + fp), tp / (tp + fn) print(round(p, 3), round(r, 3), round(2 * p * r / (p + r), 3))
0.4 0.8 0.533
Quick quiz
🧠 Quick Quiz – test yourself
1. Recall answers…
TP / (TP + FN).
2. A spam filter that sends important emails to spam has low…
Too many false positives.
3. AUC = 0.5 means…
Random ranking.
4. Lowering the threshold usually…
More cases flagged → fewer missed.
5. Best single metric for a 1%-positive problem?
Focuses on the rare class.
Interview corner
💼 Interview Corner
Precision vs recall – give an example where each matters more.
Recall matters more for fraud or disease screening – missing a case is expensive. Precision matters more for spam filtering or recommending legal action – false alarms are costly. Choose the threshold from the cost of each error.
Your model has 99% accuracy on fraud. Is it good?
Not necessarily. Check the base rate – if fraud is 1%, a dummy gets 99%. Look at recall, precision, PR-AUC and the confusion matrix, and compare with a baseline.
Summary
- Confusion matrix → TP, FP, FN, TN.
- Precision = trust in alerts, recall = coverage of positives, F1 balances.
- ROC-AUC ranks models; PR curves for rare events.
- Pick thresholds by rupee cost; handle imbalance with weights or thresholds.
➡️ Next
Day 42: KNN and decision trees – two intuitive classifiers and how they draw boundaries.
Frequently Asked Questions
Macro vs weighted average in the report?
Macro averages each class equally (good for imbalance); weighted averages by class size (dominated by the majority).
Should I balance the test set?
No – keep the test set realistic. Only rebalance training data.
