Comments on: Deleting Values in Welford’s Algorithm for Online Mean and Variance http://lingpipe-blog.com/2009/07/07/welford-s-algorithm-delete-online-mean-variance-deviation/ Natural Language Processing and Text Analytics Fri, 07 Jun 2013 17:17:33 +0000 hourly 1 http://wordpress.com/ By: lingpipe http://lingpipe-blog.com/2009/07/07/welford-s-algorithm-delete-online-mean-variance-deviation/#comment-7633 Mon, 09 Aug 2010 17:32:16 +0000 http://lingpipe-blog.com/?p=1771#comment-7633 This is not the place to ask a general question! We have a mailing list and e-mail (see the LingPipe home page).

The usual thing to do is to treat the word to count maps as vectors with the words as the dimensions and then use standard vector cosine to compare them. This is all implemented in LingPipe, though has nothing to do with this post. Often, there’s a TF/IDF rescaling of the counts. Check out LingPipe’s class spell.TfIdfDistance for details and an implementation.

]]> By: Joseph http://lingpipe-blog.com/2009/07/07/welford-s-algorithm-delete-online-mean-variance-deviation/#comment-7629 Mon, 09 Aug 2010 10:30:06 +0000 http://lingpipe-blog.com/?p=1771#comment-7629 I’m a student at University of Manitiba and had a question I thought you might be able to answer. I’m trying to identify and algorith that would be best suited to help me identify relationship between two arrays (each array generated froma separate archival story/article). Each array contains 1 column of keyword entries paired to a second column of frequencies that the keyword occurs within the original text.

The idea is that I should be able to compare these two arrays and identify if there are enough keywords in common to deem these two articles related.

Any suggestions on stats models or formulas to use? Point me in the right direction?

Thanks!

]]>