Quantcast
Channel: Question and Answer » regularization
Viewing all articles
Browse latest Browse all 33

Choosing between feature selection and regularization to overcome over-fitting in categorical regression

$
0
0

In order to overcome over-fitting during a regression process over categorical features, one can either

1) Apply L1/L2/Elastic regularization during the regression, for example as answered here

When to use regularization methods for regression?

2) Filter out categories which appear less than X times, where X is at least 10-20 ,for example

Minimum number of observations for multiple linear regression

The advantage in method 2 is that I can set with this rule of thumb approach the value of X to be 10-20, while the regularization method requires a costly computational cross validation step to be able to choose properly the correct regularization parameter, which would in fact multiply the regression running time by a factor. In addition, it greatly reduces the number of features when the categories have a long tail distribution – This can mean a big decrease in running time and memory consumption.

What is the common practice? When should I prefer one method over the other? Is the second method an acceptable one, or is the correct way to always perform some sort of regularization?


Viewing all articles
Browse latest Browse all 33

Trending Articles