Supervised Machine Learning

In supervised learning, the model learns from labeled training data, making predictions based on known input-output pairs. This section explores several key algorithms with interactive demos running directly in your browser via ONNX Runtime Web (no backend, no API keys).

The bundled .onnx files are small tutorial models trained on synthetic data so the UI and math stay consistent with the lesson. Regenerate them anytime with python scripts/export_tutorial_onnx.py.

Classification and regression

Supervised problems come in two flavours. Classification predicts a category (spam or not, which drug, which species). Regression predicts a continuous number (an insurance bill, a house price). Naive Bayes, KNN, decision trees and logistic regression are used for classification; linear regression is the regression workhorse. KNN and decision trees can also do regression.

Naive Bayes

A probabilistic classifier based on Bayes' Theorem. In this example, we predict the appropriate drug for a patient based on their health metrics.

How it works

Naive Bayes applies Bayes' theorem with a "naive" assumption: given the class, every feature is independent of every other feature. The assumptions it leans on are:

• Features \(x_i\) are independent of each other once the class \(y\) is known.
• Every feature contributes equally to the probability of the class.
• We can estimate \(P(x_i|y_j)\) for every feature value and class from the training data.

Independence lets us split a hard joint probability into a simple product:

The posterior probability of each class then comes from Bayes' theorem, and we predict the class with the highest value:

The drug dataset used here has age, sex, blood pressure, cholesterol and sodium-to-potassium ratio as inputs, and five drug classes (A, B, C, X, Y) as the target.

K-Nearest Neighbors

KNN classifies a new point by looking at the k closest training examples and taking a majority vote. No training phase — it memorises the whole dataset.

KNN is non-parametric: it assumes nothing about how the data is distributed. The trade-off is cost, since every prediction compares the query to the whole training set, which gets slow on large datasets.

The algorithm

1. Calculate distance from the query point to every training point.
2. Select neighbours: pick the K training points closest to the query.
3. Vote: find the majority class among those K neighbours.
4. Assign that class to the query point.

Choosing K

K is a hyperparameter you set before running the algorithm. A small K follows the noise in the data and overfits; a large K smooths too much and underfits.

Common distance measures

• Euclidean: \(\sqrt{\sum_i (x_i - y_i)^2}\)
• Manhattan: \(\sum_i |x_i - y_i|\)
• Minkowski: \(\left(\sum_i |x_i - y_i|^p\right)^{1/p}\), where \(p=1\) gives Manhattan and \(p=2\) gives Euclidean.

Here \(i\) runs over the dimensions (features).

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)

KNN Explorer

Click anywhere on the canvas to test a query point. The model finds the K closest training points and takes a majority vote. Watch the decision boundary update as you change K.

Decision Tree

A Decision Tree splits the feature space with a series of threshold rules, forming a tree where each leaf is a class. Highly interpretable — you can follow the exact path to any prediction.

A tree has three kinds of node: the root node (the first question), internal nodes (follow-up questions on other features) and leaf nodes (the final predictions). It works like a chain of yes/no questions, and it handles both classification and regression.

How a tree is built

1. Pick the question (feature and threshold) that gives the most useful information for predicting the target.
2. Split the data on it, creating branches.
3. Repeat on each subset, trying the other features.
4. Stop when further splits no longer improve things meaningfully.
5. Assign a prediction to each leaf.

To predict, follow the answers from the root down to a leaf. Limiting the depth (as in the code below) keeps the tree from memorising the training data.

from sklearn.tree import DecisionTreeClassifier
dt = DecisionTreeClassifier(max_depth=4, random_state=42)
dt.fit(X_train, y_train)
y_pred = dt.predict(X_test)

Logistic Regression

Logistic Regression models the probability of a binary outcome using the sigmoid function. Despite the name, it's a classification algorithm — the output is a probability between 0 and 1.

It resembles linear regression, but the straight line is passed through the sigmoid (logistic) function, which squashes any value into the range 0 to 1:

where \(X_1,\dots,X_n\) are the input features. Types of logistic regression:

• Binary: two outcomes, such as spam or not spam.
• Multinomial: three or more unordered categories, such as veg, non-veg or vegan.
• Ordinal: three or more ordered categories, such as a movie rating from 1 to 5.

from sklearn.linear_model import LogisticRegression
lr = LogisticRegression(max_iter=300)
lr.fit(X_train, y_train)
proba = lr.predict_proba(X_test)[:, 1]  # P(survived)

Linear Regression

Linear Regression fits a line (or hyperplane) through data to predict a continuous target. The model learns weights w and bias b such that ŷ = w·x + b minimises the mean squared error.

Put simply, it finds the straight line that best fits the data, showing how the dependent variable changes as the independent variable changes. With one input the model is:

where \(\hat{y}\) is the prediction, \(b_0\) the intercept and \(b_1\) the slope. The least-squares estimates are:

The output is a continuous value, not a class. The insurance dataset predicts the medical costs billed for a person from age, sex, BMI, children, smoker status and region.

from sklearn.linear_model import LinearRegression
lr = LinearRegression()
lr.fit(X_train, y_train)
predictions = lr.predict(X_test)
print(lr.coef_, lr.intercept_)

Artificial Neural Networks

Neural networks are a set of algorithms, modeled loosely after the human brain, that are designed to recognize patterns.