SIADS 542: Supervised Learning · University of Michigan · December 2025
This project spans two assignments from SIADS 542: Supervised Learning at the University of Michigan, covering the foundational methods of both regression and classification. Rather than treating them separately, the assignments are structured around a shared theme: understanding how model complexity relates to generalization — where underfitting and overfitting live, and how to navigate between them.
The regression work uses a synthetic noisy cubic dataset to explore polynomial feature expansion, bias-variance tradeoff, and Lasso regularization. The classification work applies k-Nearest Neighbors and Support Vector Classifiers to the Breast Cancer Wisconsin dataset, exploring hyperparameter tuning, weighted distance, and the effect of the RBF kernel's gamma parameter on train vs. test accuracy.
Skills demonstrated in this project
The regression portion of the project works with a synthetic dataset where the true underlying function is a known cubic polynomial — but the observed data includes Gaussian noise. This setup makes it possible to evaluate models not just against the noisy test set, but against a "gold standard" noiseless version of the true function, revealing how well each model recovers the actual signal.
Using scikit-learn's PolynomialFeatures and LinearRegression, polynomial models of degrees 1, 3, 7, and 11 were fit to the training data and evaluated by R² on both training and test sets. Degree 1 underfit the data badly — a straight line can't capture a cubic curve. Degree 11 memorized the training data nearly perfectly but generalized poorly to the test set, producing wild oscillations in regions between training points. Degree 3 achieved the best test R², aligning with the true underlying cubic structure.
The same polynomial degrees were re-fit using Lasso regression (α=0.01), which adds an L1 penalty that drives less-useful coefficients to zero. Compared to unregularized linear regression, Lasso produced smoother fits at high polynomial degrees — damping the oscillations seen at degree 11 by effectively ignoring coefficients that weren't supported by the data. Evaluated against the gold standard noiseless test set, degree 3 again performed best, confirming that the regularization was helping the model recover the true function rather than fitting noise.
A KNN regressor with default hyperparameters was fit as a non-parametric baseline. Its R² on the test set provided a useful reference point: it performed reasonably but was outperformed by the correctly-specified polynomial models, illustrating that when the true functional form is recoverable, parametric models with the right structure have an advantage over flexible non-parametric alternatives.
The classification portion applies supervised learning to the Breast Cancer Wisconsin dataset — a real medical dataset where each of 569 patient records is labeled malignant or benign based on 30 digitized cell nucleus measurements. The stakes of the problem frame the model evaluation: missing a true malignant case (false negative) is costlier than a false alarm, which motivated the exploration of recall-focused evaluation alongside overall accuracy.
KNN classifiers were built and evaluated at k=1 and k=15, then a parameter sweep across all odd values of k from 1 to 19 identified the optimal value for test set accuracy. Separately, the same sweep was run optimizing for training set accuracy instead — a deliberate overfitting exercise. At k=1, the model achieved perfect training accuracy but lower test accuracy, illustrating the classic overfitting signature. The gap between training-optimized and test-optimized k values quantified exactly how much performance was being sacrificed by selecting on the wrong metric.
An SVC with an RBF kernel was evaluated across six values of the gamma parameter (1e-7 to 1e-2) using scikit-learn's validation_curve with 3-fold cross-validation. At very low gamma, the decision boundary is too smooth — the model underfits. At very high gamma, the boundary becomes too sensitive to individual training points — the model overfits, with training accuracy near 1.0 and test accuracy dropping sharply. The optimal gamma produced the highest cross-validated test accuracy and was identified programmatically from the resulting accuracy curves.
A separate component of the classification work focused on preprocessing: applying one-hot encoding to categorical features in a housing price dataset using scikit-learn's OneHotEncoder. The implementation correctly fits the encoder on training data only and applies the transform to both sets — avoiding data leakage — and handles unseen categories in the test set through the handle_unknown parameter.