Assignment 3- LASSO Regression

In this study, we consider a dataset of claim sizes (severities) from Allstate. The data set has 130 features
and we are uncertain about what the features represent. Hence, the regression models in this study are purely
based on shrinkage and selection techniques.
Since the data is large (188,318 records and 130 features) we chose a training data that conists of 20% of
the data to keep runtimes manageable. We want to develop models to predict loss. The most basic candidate
for predictive modeling: linear regression is first used. This OLS model is regressed on all the 130 features
and is compared to the LASSO regression model later in this report.
We try to use LASSO regression using glmnet for our predictive modeling purposes. Given the size of
our model, LASSO regression helps in selection as well as shrinkage. For the LASSO model, there is a
penalty term which is weighed by a tuning parameter,  Figure 1 visualizes the relationship between
feature coefficients and . It should be noted that when there are large s, the feature coefficients are
essentially set to zero. We must reach a compromise where the feature coefficients and  value makes the
most sense.

Figure 1. Relationship between feature coefficients and tuning parameter, .

Figure 2 illustrates how the mean-squared error of our in-sample (blue) and out-of-sample data decreases
as model complexity i.e. number of features in the model increases. It is worth mentioning that as the
model becomes more complex, the in-sample Mean-squared error becomes smaller and our LASSO
model starts resembling our OLS regression model.

Figure 2. Mean-squared-error of the test and train sets as a function of the model complexity.
Furthermore, Figure 3 compares the relationship between Mean-squared error and l. It is clear that the
mean-squared error increases exponentially as log () goes beyond 4. In order to tune our parameter , we
use the cross-validation technique with 5 folds. With this technique we get a minimum =1.285.

Figure 3. Mean-squared-error as a function of the tuning parameter, .

Now that we have our LASSO regression model tuned, we must evaluate its performance. From further
calculations and by comparing our model predictions to the test set data we obtain a R2=0.474. This value
by itself is quite low and shows that our model may not be completely reliable. We also compare the
LASSO regression model to the OLS regression model in Table 1. It should be noted that with LASSO
regression model, the in-sample RMSE increases slightly and contrastingly out-of-sample RMSE
decreases significantly by 40%. This shows that the LASSO regression model is more efficient with Out-
of-sample predictions than the OLS regression model. However, we should keep in mind that even the
LASSO model is not perfect and may not be used as the key factor in decision making in this case.
Table 1. In-sample, Out-of-sample RMSEs for Linear and LASSO regression models.
In-sample RMSE Out-of-sample RMSE
Linear Regression model 1929.04 3584.16
LASSO Regression model 1994.70 2122.16