What makes CRFs computationally challenging is making sure p(tags|words) sums to 1.0 over the exponentially sized set of possible tag sequences. Comptutationally, you usually start with something proportional to p(tags|words) and then normalize by summing. The challenge is that there are exponentially many sequences of tags. The good news is that you can use a version of forward/backward (a linear dynamic programming algorithm) to solve the summation.

What I understand from your explanation is that CRF generates features in a fashion similar to logistic regression, that is binomial probabilities from an arbitrary number of regression coefficients (betas). If it’s so, then thanks for that. This helps.

Would you mind elaborating a bit on what you mean by ‘normalization?’

HMMs model label transitions as a multinomial p(tag|previousTag) per previous tag. In NLP, emissions are also typically multinomial (like naive Bayes) for p(word|tag).

CRFs are more like logistic regression in that they extract features from the input words using arbitrary amounts of context, outside knowledge, etc. The words are explanatory variables and are not themselves modeled. The chain part means you can also extract features from pairs of labels plus the context. The key differentiator of CRFs is that they normalize over the whole input, not per tag.

Really, though, you should read one of the tutorials.

I’m not the world’s best reader of other people’s code (that’d be Ezra), but CRF algorithms are particularly difficult because they embed I/O, tokenization, serialization, feature extraction, forward-backward, and some kind of regularized logistic regression optimizer. The research packages tend to provide lots of options because they’ve been doing research on these things for years. I get lost just at the “what exactly are these input types” stage with Mallet and JavaNLP.

I’m also not sure the people who implemented this stuff wouldn’t do it differently given a second chance.