Comments on: Log Sum of Exponentials

By: Martin

Martin — Tue, 14 Jul 2009 00:46:41 +0000

In my limited experience, the more useful function to approximate is f(x) = log(1+exp(x)). You only need to do that for x>=0 or x<=0. For sufficiently large x, log(1+exp(x)) ~= x; and for sufficiently small x (e.g. x < 750 in IEEE double precision), log(1+exp(x)) ~= 0. Moreover, the difference between the function and its asymptotes is symmetric about 0, i.e., f(x) – x == f(-x). So you only need an approximation for f near 0 and only on one side of 0. A rational approximation of f can usually be evaluated much faster than log (or log1p) and exp, since each one of those is actually computed by a polynomial or rational approximation in turn.

(Aside: f should be familiar, though what usually shows up is df/dx.)

By: lingpipe

lingpipe — Fri, 26 Jun 2009 04:01:05 +0000

sth4nth is talking about his or her comment on my softmax without overflow blog entry. Sorry I'm so slow with all this arithmetic precision stuff; I should've definitely put exp(2) and exp(2) together in this case.

By: sth4nth

sth4nth — Fri, 26 Jun 2009 02:45:24 +0000

That’s what I said in the replay of the logistic regression post.

By: lingpipe

lingpipe — Thu, 25 Jun 2009 20:37:28 +0000

Thanks for reminding me about the special log calculation. For everyone else who's not knee deep in these derivations, the binary case is: log(exp(a) + exp(b)) = a + log(1 + exp(b-a)) and this is where the special calculation for log(1+x) comes into play. Check out John Cook's discussion of log(1+x), which uses a Taylor approximation for x < 1e-4 (that is, x < 0.0001), and simply makes a lib call otherwise.

By: Jonathan Graehl

Jonathan Graehl — Thu, 25 Jun 2009 20:12:24 +0000

Many of us have defined the same operation pairwise so as to have (with operator overloading) numbers which have the usual operations. In adding pairwise you can use a special function to compute log(1+x) which is more accurate for small x – “log1p”. You could use that in your vector operation as well, although it would only be relevant if all the numbers were MUCH smaller than the maximum.