Comments on: Speeding up K-means Clustering with Algebra and Sparse Vectors

By: lingpipe

lingpipe — Wed, 09 Dec 2009 19:57:53 +0000

Thanks for the link. I know that paper and anyone doing k-means should read it. Even if you don’t use the algorithm, it provides great insight into the geometry of the problem.

The main reason I didn’t use it was the extra memory overhead.

I would have sworn I’d discussed exactly Elkan’s algorithm on this blog entry, but can’t find any mention of it here or elsewhere.

By: jeff

jeff — Wed, 09 Dec 2009 18:17:13 +0000

Check out:
http://cseweb.ucsd.edu/~elkan/kmeansicml03.pdf

C. Elkan. Using the Triangle Inequality to Accelerate k-Means (pdf). In Proceedings of the Twentieth International Conference on Machine Learning (ICML’03), pp. 147-153.

By: lingpipe

lingpipe — Fri, 13 Mar 2009 18:53:29 +0000

@sth4nth: Cool! I should’ve kept going with the algebra. It’s obvious the constants don’t matter, but I’m always surprised when you can replace distances with products. The key to the approach I outlined in the entry was using sparse multiplications, because the S vectors (in your notation) are sparse at around 1% or less of the C vectors in the applications we’re considering.

@dz: Thanks! I keep forgetting that Euclidean distance is a real metric. I’m so used to having to generalize to pseudo-metrics where the triangle ineuqality fails.

@myself: The other thing I really need to do now is cache distances vs. centroids when the centroids don’t change. After a few dozen epochs with K=1000, lots of the clusters don’t change at all between epochs, so that whole columns of sth4nth’s C’ * S matrix are constant.

By: dz

dz — Fri, 13 Mar 2009 15:35:29 +0000

Don’t forget the triangle inequality speed-up: if you pre-compute distances between centroids d(c,c’) before each iteration, then you don’t have to compute many distances between examples and centroids d(x,c) at all, because d(x,c) >= d(x,c’) – d(c,c’)

By: sth4nth

sth4nth — Fri, 13 Mar 2009 05:15:36 +0000

The third term of (1) can be dropped not the first one. Sorry for that error but the idea is the same

By: sth4nth

sth4nth — Fri, 13 Mar 2009 03:07:14 +0000

Since it’s hard to write formula in comment, I’m gonna write the idea in matlab language.

In each iteration, you want to find the samples that are closest to centers.
Assume you have two matrices C and S. Each colum of C is a center vector and Each colum of S is a sample vector. Then the distance matrix

D=sum(C.^2,1)’*ones(1,n)-2*C’*S+ones(k,1)*sum(S.^2,1); //(1)
D is a k x n matrix with each element is the distance between a sample and a center.

Since you also want to find the min disances, the first term of right hand side of (1) can be droped and the third term (length of all samples) can precomputed. Then you only need to compute C’*S in each iteration. This will be efficient even for dense matrix. For a naive implementation, each iteration will cost O(n*k*d), the matrix multiplication certainly can do better than that, not to mention if the vector is sparse.