Machine Learning (ML) Classification Algorithms Exercises

The distinction between these two categories is fundamental:

• Why Option 2 is correct: Discriminative models (like Logistic Regression or SVMs) try to find the "line" that separates classes directly, rather than modeling the distribution of the classes themselves.
• Why others are incorrect: Modeling \( P(x, y) \) is the definition of a Generative model (like Naive Bayes). Discriminative models are supervised, not unsupervised, and are generally more robust to feature distributions.

This classification refers to the flexibility and complexity of the model:

• Why Option 1 is correct: Parametric models (e.g., Logistic Regression) assume a specific functional form. Non-Parametric models (e.g., K-Nearest Neighbors) make fewer assumptions and grow in complexity as you add more data.
• Why others are incorrect: Both can be used for classification. Non-Parametric models are often slower to predict because they may need to reference the entire dataset.

The decision boundary is the "map" created by the algorithm:

• Why Option 3 is correct: It is the threshold where the model switches from predicting one class to another. In 2D, it's a line; in 3D, it's a plane; in higher dimensions, it's a hyperplane.
• Why others are incorrect: Boundaries represent a transition point, not necessarily a probability of zero. There is no limit on features based on the definition of a boundary.

This distinction relates to the output format:

• Why Option 3 is correct: A Hard Classifier (like a Perceptron) tells you "This is Class A." A Soft Classifier (like Logistic Regression) tells you "There is an 85% chance this is Class A."
• Why others are incorrect: Complexity or data types are not the defining factors; the "hardness" refers to the certainty of the output label versus a probability.

Linear separability is a key concept for simple classifiers:

• Why Option 2 is correct: If a linear equation (like \( w^T x + b = 0 \)) can divide the classes without error, the data is linearly separable.
• Why others are incorrect: This property describes the geometry of the classes relative to each other, not the distribution of individual features or the complexity of non-linear models.

Linear classifiers are restricted to straight lines or flat hyperplanes, which often fail when class data is intertwined. A non-linear boundary allows the model to "bend" its separation logic around the data points. This is achieved by either using non-linear algorithms (like Decision Trees) or by transforming features into higher dimensions where a curve in the original space becomes a complex manifold. It does not limit the number of features, nor does it require a specific shape like a circle; it simply means the separation is not a straight line.

The Concept: A hyperplane is the mathematical generalization of a "divider" to any number of dimensions.

The Logic: In a 2D space, the divider is a 1D line. In a 3D space, it is a 2D plane. Therefore, in an \( n \)-dimensional space, the boundary is an \( (n-1) \)-dimensional hyperplane. It is the geometric surface defined by the linear equation \( w^T x + b = 0 \) that splits the data into two distinct regions.

The Concept: Boundary "smoothness" is a direct reflection of the Bias-Variance tradeoff.

The Logic: A jagged boundary (Model B) shows that the model is chasing every outlier and noise point in the training set (Low Bias). However, this extreme sensitivity usually means the model won't work on new data (High Variance/Overfitting). Model A is likely more robust because it ignores noise to find the general trend.

The Concept: The margin represents the "breathing room" or safety buffer around the decision boundary.

The Logic: Instead of just finding any line that separates two groups, "Max-Margin" classifiers look for the line that is as far away from the nearest points (Support Vectors) as possible. A larger margin generally leads to better generalization because it provides a wider gap to accommodate slight variations in new data.

The Concept: Geometric shapes in classification are defined by the underlying mathematical function of the features.

The Logic: The equation \( x_1^2 + x_2^2 = r^2 \) is the classic algebraic definition of a circle. When a classifier uses this logic, it creates a circular boundary where points inside the radius belong to one class and points outside belong to another. This is a common result when using polynomial feature transformations.

The Concept: Soft classifiers provide a "degree of belief" rather than just a final category.

The Logic: Unlike hard classifiers that output 0 or 1, soft classifiers (like Logistic Regression) output a value between 0 and 1. This represents the confidence or likelihood that a sample belongs to the positive class, allowing users to adjust decision thresholds based on the cost of different types of errors.

The Concept: Calibration ensures that the "80% confidence" reported by a model actually means it is right 80% of the time.

The Logic: Some models are overconfident or underconfident due to their mathematical assumptions. Calibration techniques (like Platt Scaling) adjust the output scores so they represent true empirical probabilities, which is crucial for decision-making systems like medical diagnosis or risk assessment.

The Concept: Model calibration often degrades as model complexity increases.

The Logic: Highly complex models can drive their training loss so low that they output extreme probabilities (0.999...). While they may be accurate on training data, these probabilities often do not reflect the true uncertainty on new data, making the model "uncalibrated" and overconfident.

The Concept: The decision threshold is a business or safety decision, not just a mathematical one.

The Logic: Using 0.5 means we treat both types of errors as equally bad. However, in cases like cancer screening, a False Negative is much worse than a False Positive, so we would likely lower the threshold to catch more cases, even if it reduces precision.

The Concept: The Brier Score assesses the quality of the probabilistic predictions themselves.

The Logic: While Accuracy only looks at whether the label was right, the Brier Score looks at how close the probability was to the truth. If the true label is 1, a prediction of 0.9 is much better than 0.51, and the Brier Score captures this difference in "soft" performance.

The One-vs-Rest strategy works by training one independent binary classifier for each class in the dataset. For a 4-class problem, you end up with four models, where each model identifies how likely a point is to belong to its specific class versus all other classes combined.

When a new prediction is needed, the data point is passed through all four models simultaneously. The final prediction is assigned to the class whose specific model produced the highest confidence or probability score, effectively "winning" the competition among the available categories.

The One-vs-One approach creates a separate binary classifier for every possible pair of classes. While this allows each model to focus on a simpler boundary between two specific groups, the sheer volume of models becomes a computational burden as the number of classes increases.

Mathematically, for \( K \) classes, you must train \( \frac{K(K-1)}{2} \) models. For 100 classes, this would result in 4,950 separate models, which significantly slows down training and prediction compared to the \( K \) models required by the One-vs-Rest strategy.

Independent Sigmoid functions treat each class as a separate "Yes/No" question, which can lead to a situation where the sum of probabilities across all classes is greater than or less than 1. This makes it difficult to interpret the results as a single probability distribution.

The Softmax function solves this by tying all class outputs together. By exponentiating the scores and dividing by the total sum, it forces the outputs to be mutually exclusive and collectively exhaustive, ensuring that the model picks exactly one class with a normalized probability.

Some algorithms, such as Support Vector Machines (SVMs), have a training complexity that increases significantly as the number of samples grows. In OvR, each model is trained on the entire dataset, which can be extremely slow for these specific types of learners.

In the OvO strategy, each binary classifier is only trained on data belonging to two specific classes. This significantly reduces the training set size for each individual model, making it a more efficient choice for algorithms that struggle with large-scale data volume.

Even if your original 10-class dataset is perfectly balanced with 10% of data per class, the binary sub-problems created by OvR will not be. When you train a model for "Class A vs All Else," Class A makes up 10% of the data while "All Else" makes up 90%.

This inherent imbalance can lead to models that are biased toward the "Rest" category, as the classifier sees many more negative examples than positive ones. This often requires additional tuning, such as adjusting class weights, to ensure the model doesn't simply ignore the specific class it is meant to identify.