Comments on: Feature Hash Code Collisions in Linear Classifiers

By: Yoav Goldberg

Yoav Goldberg — Tue, 06 May 2008 17:51:49 +0000

My intuition is that the small set of core predictive features is quite stable, and that a very good approximation of it can be obtained by training on a relatively small sample of the data (in the setting described by Kuzman, where the interest is mostly in test time, these features can be obtained from all the data). Once this feature set is available, a very efficient recognizer can be compiled. So the overhead seem pretty small.

Ofcourse, the actual benefit from this needs to be verified empirically.

By: Bob Carpenter

Bob Carpenter — Tue, 06 May 2008 16:19:37 +0000

You’d need to inspect each feature to see if it’s common. This reintroduces something like a table lookup for every feature to see if it’s a common feature.

In addition to being common, features need to be selected for discriminative power (either by looking at fitted coefficients in a table-based model or by using some kind of filtering approach like information gain or chi-squared tests).

By: Yoav Goldberg

Yoav Goldberg — Sun, 04 May 2008 09:53:29 +0000

Another thing worth trying: from the nature of NLP tasks, a small core of the predictive features are very common, while most of the features are rare. Maybe allocating different ranges for common and non-common features (making sure common features does not collide interntally and externally) could be beneficial to classification accuracy. This has the possible (negligible?) cost of saving a table of ~300 entries for the common features.

By: Kuzman Ganchev

Kuzman Ganchev — Sat, 03 May 2008 03:26:19 +0000

For what it’s worth, we tried adding multiple differently hashed copies of the features in some preliminary experiments. We didn’t see any improvement over single copies (I think there was a small consistent decrease in performance). I suspect that this is because we were using a relatively small number of slots (at most twice as many as the number of features). Certainly, if m=n, then adding sufficiently many copies will eventually make classification impossible, since all the features will be mixed with each other. Maybe if there are many more slots than features, then adding multiple copies of features would help, since then most of the second copies will land in empty bins.