see attahed
Machine Learning, 2022S: HW5 CS 5033 - Machine Learning: Homework 5 Spring 2022 Due: Monday, April 11, 2022 In this homeworkwewill be practicingwith techniques on regularization, model assessment, and model selection. In this direction we will work with two different datasets from the UCI repository. Below I assume that you will be using scikit-learn. Exercise 1 – Preprocessing for Regression (20 points). We will be using a UCI superconduc- tivity data set that is available at: h�ps://archive.ics.uci.edu/ml/datasets/Superconductivty+Data. This dataset has 21,263 examples using 81 real-valued a�ributes, with a real-valued target variable. For this assignment, we will use the file train.csv and we can ignore the file unique_m.csv. The target value that we are predicting is the critical temperature in Kelvins, which is the last column in the dataset. (a) Remove 20% of the examples and keep them for testing. You may assume that all examples are independent, so it does not ma�er which 20% you remove. However, the testing data should not be used until a�er a model has been selected. (b) Split the remaining examples into training (75%) and validation (25%). Thus, you will train with 60% of the full dataset (75% of 80%) and validate with 20% of the full dataset (25% of 80%). Exercise 2 – Regression (30 points). We will be using the dataset from Exercise 1. For this problem please use the scikit-learn method, sklearn.linear_model.ElasticNet. (a) Fit an elastic net model to the training data with each possible combination of the following L1 and L2 regularization weights. • λ1 = 0, 10 −6, 10−5, 10−4, 10−3, 10−2, 10−1, 1 • λ2 = 0, 10 −6, 10−5, 10−4, 10−3, 10−2, 10−1, 1 If you check the documentation, you’ll see that the input arguments for ElasticNet do not include λ1 and λ2. Instead, they include alpha, and l1_ratio. λ1 is alpha * l1_ratio, and λ2 is alpha * (1 - l1_ratio). March 30, 2022 1/3 https://archive.ics.uci.edu/ml/datasets/Superconductivty+Data https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html CS 4033/5033 - Machine Learning: Homework 5 (b) For each model trained in step (a), make a prediction for each training example, using the predict method for sklearn.linear_model.ElasticNet and calculate the mean squared error (MSE) on the training examples. Report these values in your writeup. As a reminder, the mean squared error (MSE) on a dataset of sizem is: MSE = 1 m m∑ i=1 (yi − h(xi)) 2 , where, as usual, h(xi) is the prediction of our hypothesis and yi is the correct (ground truth) prediction as this is defined in the dataset. (c) This time, for each model trained in step (a), make a prediction for each validation example and calculate the mean squared error on the validation examples. Report these values in your writeup. (d) Which model (i.e., pair of λ1 and λ2) performed best on the training data? Which model performed best on the validation data? Report this in your writeup. (e) Find the best hyperparameter set (pair of λ1 and λ2) on the validation data. Train a model with the same λ1 and λ2 on both the training and validation data. Using this model, make predic- tions for each testing example and calculate the mean squared error on the test examples. Report this value in your writeup. Exercise 3 – Preprocessing for Classification (20 points). Here we will do the same thing as we did for Exercise 1, but for a different dataset. In particular, we will be using a UCI simulated electrical grid stability data set that is available here: h�ps://archive.ics.uci.edu/ml/datasets/Electrical+Grid+Stability+Simulated+Data+. This dataset has 10,000 examples using 11 real-valued a�ributes, with a binary target (stable vs. un- stable). The target value that you are predicting is the last column in the dataset. (a) Remove columns 5 and 13 (labeled p1 and stab); p1 is non-predictive and stab is target column that is exactly correlated with the binary target you are trying to predict (if this column is negative, the system is stable). (b) Change the target variable to a number. If the value is stable, change it to 1, and if the value is unstable, change it to 0. (c) Remove 20% of the examples and keep them for testing. You may assume that all examples are independent, so it does not ma�er which 20% you remove. However, the testing data should not be used until a�er a model has been selected. (d) Split the remaining examples into training (75%) and validation (25%). Thus, you will train with 60% of the full dataset (75% of 80%) and validate with 20% of the full dataset (25% of 80%). 2/3 March 30, 2022 https://archive.ics.uci.edu/ml/datasets/Electrical+Grid+Stability+Simulated+Data+ CS 4033/5033 - Machine Learning: Homework 5 Exercise 4 – Logistic Regression (30 points). For this problem you can use the scikit-learn method sklearn.linear_model.LogisticRegression. (a) Fit a logistic regression model with L2 regularization (use the default value of λ) and another logistic regression model with no regularization. Note that, per the documentation, by default the function sklearn.linear_model.LogisticRegression performs L2 regularization and in order not to use regularization we need to pass penalty=’none’ as parameter in the cre- ation of the model. (b) Using the two models created in part (a) make a prediction for each validation example. What is the empirical risk (using the 0-1 loss) of each model on the validation set and what is the confusion matrix of each model (again, on the validation set)? (c) Which model performed be�er on the validation data? Report this in your writeup. We con- sider a model be�er than another one if the empirical risk of the first model is lower than the empirical risk of the second model. In case the empirical risk is a tie, compare the models using precision and determine which one is be�er. Now, train a new logistic regressionmodel on the training and validation data using whichever measure created the best model (i.e., is it beneficial to use L2 regularization in this dataset, or not?) in (a)-(b). Make a prediction for each testing example. Report the empirical risk on the test set and the confusion matrix that corresponds to the predictions of this last model on the test data. March 30, 2022 3/3