Support Vector Machine (SVM) in Python
In this lesson: Support Vector Machine explained — margin, kernels — with scikit-learn SVC code and accuracy output. In Hindi & English.
What is a Support Vector Machine?
A Support Vector Machine (SVM) is a classification algorithm that draws the boundary between two classes with the widest possible gap (margin). Think of a road between two villages: SVM builds the widest road that keeps every house on its own side. The houses that touch the road edges are the support vectors – only they decide where the road goes.
| Term | Meaning |
|---|---|
| Hyperplane | The decision boundary – a line in 2-D, a plane in 3-D |
| Margin | Distance from the boundary to the nearest points of each class |
| Support vectors | The training points closest to the boundary |
| C | Penalty for mistakes: small C = wider, softer margin; large C = fits training data tightly |
| Kernel | A trick to draw curved boundaries (RBF, polynomial) |
| gamma | For RBF: how far one point’s influence reaches; large gamma = wiggly boundary |
Step 1: a linear SVM you can see
Twelve students: hours studied and hours slept the night before, and whether they passed. A linear SVM finds the boundary and tells us which students are support vectors.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.svm import SVC
X = np.array([[1, 5], [2, 4], [2, 6], [3, 3], [3, 5], [4, 2], # failed
[5, 7], [6, 6], [6, 8], [7, 5], [8, 7], [8, 6]]) # passed
y = np.array([0] * 6 + [1] * 6)
model = SVC(kernel="linear", C=1.0).fit(X, y)
print("support vectors:\n", model.support_vectors_)
print("prediction for 4.5 h study, 6 h sleep:", ["fail", "pass"][model.predict([[4.5, 6]])[0]])
# draw points, boundary and margins
xx, yy = np.meshgrid(np.linspace(0, 9, 200), np.linspace(1, 9, 200))
zz = model.decision_function(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
fig, ax = plt.subplots(figsize=(7, 4.2), dpi=110)
ax.contour(xx, yy, zz, levels=[-1, 0, 1], linestyles=["--", "-", "--"], colors="#334155")
ax.scatter(X[:, 0], X[:, 1], c=np.where(y == 1, "#16a34a", "#dc2626"), s=60, edgecolor="k")
ax.scatter(*model.support_vectors_.T, s=220, facecolors="none", edgecolors="#2563eb", linewidths=2, label="support vectors")
ax.set_xlabel("hours studied"); ax.set_ylabel("hours slept"); ax.legend(loc="lower right")
ax.set_title("Linear SVM: solid = boundary, dashed = margin")
fig.tight_layout(); fig.savefig("svm_margin.png")
Only the circled points matter. Delete any other point and retrain – the boundary would not move.
Step 2: kernels for curved boundaries
Real data is rarely separable by a straight line. The RBF kernel lets SVM draw curves without you creating new features by hand.
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
X, y = make_moons(n_samples=300, noise=0.2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
fig, axes = plt.subplots(1, 2, figsize=(9, 3.6), dpi=110)
xx, yy = np.meshgrid(np.linspace(-1.5, 2.5, 300), np.linspace(-1, 1.5, 300))
for ax, kernel in zip(axes, ["linear", "rbf"]):
m = SVC(kernel=kernel, C=1, gamma="scale").fit(X_train, y_train)
print(f"{kernel:<6} test accuracy: {m.score(X_test, y_test):.2f}")
zz = m.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
ax.contourf(xx, yy, zz, alpha=0.25, cmap="RdYlGn")
ax.scatter(X[:, 0], X[:, 1], c=y, cmap="RdYlGn", s=12, edgecolor="k", linewidths=0.3)
ax.set_title(f"kernel = {kernel}")
fig.tight_layout(); fig.savefig("kernels.png")
Step 3: what C and gamma do
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
X, y = make_moons(n_samples=300, noise=0.25, random_state=1)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=1)
print(f"{'C':>6} {'gamma':>6} {'train':>6} {'test':>6} {'#SV':>5}")
for C, gamma in [(0.1, 0.5), (1, 0.5), (1, 5), (100, 50)]:
m = SVC(kernel="rbf", C=C, gamma=gamma).fit(X_tr, y_tr)
print(f"{C:>6} {gamma:>6} {m.score(X_tr, y_tr):>6.2f} {m.score(X_te, y_te):>6.2f} {len(m.support_):>5}")The last row scores almost 100% on training data but worse on test data – it memorised the noise (overfitting). Choose C and gamma with cross-validation, never by looking at training accuracy.
Step 4: a real medical dataset (with scaling)
scikit-learn includes the Wisconsin breast-cancer dataset: 569 tumour samples, 30 measurements each, labelled malignant or benign. SVM measures distances, so features with large numbers (area ≈ 1000) would dominate small ones (smoothness ≈ 0.1). Always scale features first – a Pipeline does it safely.
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
data.data, data.target, test_size=0.25, stratify=data.target, random_state=42)
unscaled = SVC().fit(X_train, y_train)
print("without scaling:", round(unscaled.score(X_test, y_test), 3))
pipe = make_pipeline(StandardScaler(), SVC())
grid = GridSearchCV(pipe, {"svc__C": [0.1, 1, 10], "svc__gamma": ["scale", 0.01, 0.1]}, cv=5)
grid.fit(X_train, y_train)
print("best params :", grid.best_params_)
print("with scaling :", round(grid.score(X_test, y_test), 3))
pred = grid.predict(X_test)
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, target_names=data.target_names, digits=3))Read the confusion matrix row by row: the first row is actual malignant tumours, the second actual benign. In medicine, missing a malignant tumour (top-right cell) is the costliest mistake, so look at malignant recall, not only accuracy.
When to use SVM
| Good for | Not ideal for |
|---|---|
| Small to medium datasets (hundreds to tens of thousands of rows) | Millions of rows – training becomes slow |
| Many features, e.g. text with thousands of words | When you need probabilities or easy explanations |
| Clear margin between classes | Very noisy, overlapping classes without tuning |
For a comparison with other classifiers on a real business problem, see the churn project in the Data Science course.
Practice tasks
- Change C to 0.01 and 100 in Step 1 and describe how the margin changes.
- Try
kernel="poly", degree=3on the moons data and compare test accuracy. - In Step 4, add
class_weight="balanced"and compare malignant recall. - Use
SVC(probability=True)and print the probability for the first 5 test samples.
Summary
- SVM finds the boundary with the widest margin; support vectors define it.
- C controls the trade-off between a wide margin and training mistakes.
- Kernels (RBF, poly) create curved boundaries; gamma controls how curvy.
- Always scale features (StandardScaler in a Pipeline) and tune C/gamma with cross-validation.
- Judge with a confusion matrix and recall, not accuracy alone.
