Author(s): Suruthis

Originally published on Towards AI.

Demystifying the ML Engine

1. Traditional vs. Modern ML

Machine learning focuses on training a system to understand underlying patterns in data and predict an appropriate result.

Traditional Machine Learning: Data + Human-Written Rules → Predictions

Modern Machine Learning: Data + Labeled/Unlabeled Datasets → Machine-Written Rules

2. The Three Pillars of Machine Learning

Supervised Learning: Learning with explicit guidance (Data + Labels → Rules).

Unsupervised Learning: Discovering hidden structures (Data + No Labels → Rules).

Reinforcement Learning: Learning dynamically through environment trial-and-error, experience, and feedback loops.

3. Dataset Architecture & Class Balance

a) Target Variable Types (Supervised Learning)

Regression: Predicting continuous numerical values (e.g., housing prices).

Classification: Predicting discrete categorical values (e.g., spam vs. ham).

b) Class Distributions

Balanced Class (50/50): The dataset shares target classes in a near-perfect equilibrium.

Imbalanced Class (e.g., 30/70, 20/40/40, 35/65): The dataset classes are unequally distributed, causing potential model bias toward the majority class.

c) Cardinality Types

Binary Classification: The final prediction exists strictly between two outcomes (Yes/No, 0/1, Spam/Not Spam, Approve/Reject).

Multi-Class Classification: The final prediction exists across more than two distinct classes (Approve/Reject/Maybe, 0/1/2/3, North/East/West/South).

4. Algorithm Deep Dive by Engine Type

Category A: Tree-Based Models (Splitting Criteria)

These models recursively partition data. They are the only algorithms on this list that use localized splitting thresholds rather than optimization functions across the entire dataset.

Decision Tree(DT) (Regression & Classification)

Constructs a flowchart-like structure by asking sequential binary (Yes/No) questions to split data into branches.

Leaf Node Output: Predicts the neighbor average for regression tasks and a majority vote for classification tasks.

Splitting Engine (Classification):

Gini Impurity: Measures uncertainty or disorder in a node without using a logarithmic format (faster to compute).

Entropy: Measures uncertainty or disorder using a logarithmic format providing high precision at a higher computational cost.

Information Gain (IG): Measures exactly how much uncertainty is reduced after splitting a node based on a specific feature. The tree maximizes this gain.

Splitting Engine (Regression):

Uses Mean Squared Error (MSE) or Variance Reduction to create partitions.

Random Forest (RF) (Regression & Classification)

An ensemble method that applies Bagging (Bootstrap Aggregating) to build independent trees in parallel.

Data Splitting: Uses the same mathematical splitting criteria as Decision Trees (Gini, Entropy, or MSE), but forces diversity by applying them to random subsets of features and data rows with replacement.

Evaluation: Left-over rows form the Out-of-Bag (OOB) dataset, yielding a built-in OOB Evaluation Score. Final predictions aggregate via averages (regression) or majority votes (classification).

AdaBoost (Regression & Classification)

An ensemble method that applies Boosting to build models sequentially, where each new model fixes the errors of its predecessor.

Data Splitting: Typically utilizes highly shallow decision trees called Decision Stumps as its weak base learners. It relies on a weighted Gini/Entropy configuration to create those sequential stumps.

Category B: Regression & Linear Models (Loss Functions)

These algorithms do not split data. They fit a continuous mathematical function (like a line or a curve) across the entire feature space by minimizing an error value called a Loss Function.

Linear Regression (Pure Regression)

Constructs an optimal straight line mapping features to a continuous target variable.

Ordinary Least Squares (OLS): A direct mathematical formula calculating the precise slope (m) and intercept (c) to fit y = mx + c.

Gradient Descent: An iterative loop algorithm that guesses random parameters and adjusts them step-by-step to shrink prediction errors.

Loss Function minimized: Minimizes Mean Squared Error (MSE) or Residual Sum of Squares (SSR).

Polynomial Regression (Pure Regression)

An extension of linear regression where the independent variables are raised to higher degrees to fit non-linear curves.

Loss Function minimized: Minimizes the exact same loss functions as linear regression (MSE/SSR).

Stopping Criteria: The degree capacity is capped when the error on validation data stops shrinking, preventing overfitting.

Logistic Regression (Pure Classification)

Despite its name, this is a strict classification algorithm. It applies a Sigmoid function to a linear regression baseline, compressing any real value into an S-shaped probability curve between 0% and 100%.

Loss Function minimized: Uses Log Loss (Binary Cross-Entropy Loss) to penalize wrong confidences. A 100% correct prediction yields a 0 penalty; a 100% wrong prediction penalty reaches infinity (∞).

Category C: Instance-Based Models (Distance Metrics)

K-Nearest Neighbors (KNN) (Regression & Classification)

An instance-based algorithm evaluating a chosen number (K) of closest data points.

No Training Phase: KNN possesses zero training steps, splitting criteria, or loss functions.

Execution Mechanic: Makes predictions on the fly by calculating Distance Metrics (like Euclidean, Manhattan, or Minkowski distance) to find the closest data points to a query. If numerical, it outputs the average of the neighbors; if categorical, it outputs a majority vote.

5. Comprehensive Evaluation Metrics

Regression Performance Metrics

SSE (Sum of Squared Errors):

Sum of the squared distances from actual points to the predicted line.

SSR (Regression Sum of Squares): Sum of the squared distances from the predicted line to the global data mean.

SST (Total Sum of Squares): The overall variance in the data, calculated as SST = SSR + SSE.

MSE (Mean Squared Error): The mathematical mean of the SSE.

RMSE (Root Mean Squared Error): The square root of the MSE, mapping errors back to the baseline unit.

R² (Coefficient of Determination): Calculated universally as 1-SSESST. Measures the proportion of variance predictable from the independent variables. Higher is better.

Classification Performance Metrics

Accuracy: Total correct predictions divided by total predictions. Highly effective for Balanced Classes, but deeply flawed for Imbalanced Classes because it ignores minority class failure.

The Confusion Matrix (Example Scenario: Spam Detection)

True Positive (TP): Correctly predicted positive (Caught actual spam).

True Negative (TN): Correctly predicted negative (Kept clean emails safe).

False Positive (FP): False alarm (Flagged an innocent email as spam).

False Negative (FN): Missed detection (Let spam leak into the main inbox).

Advanced Classification Frameworks:

Precision: Focuses on minimizing False Positives (False Alarms).
Formula: TP/TP+FP.

Recall (Sensitivity): Focuses on minimizing False Negatives (Missed Classes).
Formula: TP/TP+FN.

F1-Score: The harmonic mean that balances Precision and Recall into a single value.

ROC Curve: A graphical curve plotting the True Positive Rate (TPR) against the False Positive Rate (FPR) across various decision thresholds.

AUC Score: The area under the ROC curve ranging between 0 and 1. It delivers a holistic, macro-level metric of overall prediction strength.

Multi-Class Averaging Strategies:

When measuring performance across multiple classes, metrics are combined using three distinct averaging approaches:

• Macro Average: Computes metrics independently for each class and then takes a simple unweighted mean. Treats all classes identically, regardless of how many samples they contain.
• Micro Average: Pools the global counts of TPs, FPs, and FNs across all classes first, then calculates a single overarching metric. Prioritizes total volume, which means it is heavily dominated by majority classes.
• Weighted Average: Calculates the independent metric for each class, then takes an average weighted by the Support (the actual row count) of each class. Reflects true business and operational data volume.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI