Averages are statistics calculated over a set of samples. If you have a set of samples , their average, often written , is defined by
Means are properties of distributions. If is a discrete probability mass function over the natural numbers , then its mean is defined by
If is a continuous probability density function over the real numbers , then its mean, if it exists, is defined by
This also shows how summations over discrete probability functions, relate to integrals over continuous probability functions, . (Distributions can also be mixed, like spike and slab priors, but the math gets more complicated due to the need to unify the notion of summation and integration.)
To confuse matters further, there are expectations. Expectations are properties of (some) random varaibles. The expectation of a random variable is the mean of its distribution. If is a discrete random variable with probability mass function , then its expectation is defined to be
If is a continuous random variable with probability density function , then
Samples don’t have means per se. They have averages. But sometimes the average is called the “sample mean”. Just to confuse things.
Averages as Estimates of the Mean
Gauss showed that the average of a set of independent, identically distributed (i.i.d.) samples from a distribution is a good estimate of the mean.
What’s good about the average as an estimator of the mean? First, it’s unbiased, meaning the expectation of the average of a set of i.i.d. samples from a distribution is the mean of the distribution. Second, it has the lowest expected mean square error among all estimators of the mean. That’s why everyone likes square error (that, and its convexity, which I discussed in a previous blog post on Mean square error, or why committees won the Netflix Prize).
What about Medians?
The median is a good estimator too. Laplace proved that it has the lowest expected absolute error among estimators (I just learned it was Laplace from the Wikipedia entry on median unbiased estimators). It’s also more robust to outliers.
More on Estimators
The Wikipedia page on estimators is a good place to start.
Of course, in Bayesian statistics, we’re more concerned with a full characterization of posterior uncertainty, not just a point (or even interval) estimate.
- Means are properties of distributions.
- Expectations are properties of random variables.
- Averages or sample means are statistics calculated from samples.