Bayesian Statistical Inference

This method, or class of methods, is easy to describe in the framework of an approach to statistical inference (i.e all of experimental science?) which is more than two hundred years old, dating from 1763! Bayes Theorem about conditional probabilities states that

$$P(A\vert B)P(B)=P(B\vert A)P(A)=P(A,B).$$

As a theorem, it is an easy consequence of the definitions of joint probabilities (denoted by $P(A,B))$, conditional probabilities (denoted by $P(A\vert B)$) and marginal or unconditional probabilities (denoted by $P(A)$). In words, one could say that the fraction of trials $A$ and $B$ both happen ($P(A,B)$) is the product of (i) the fraction of trials in which $A$ happens ($P(A)$) irrespective of $B$, and (ii) the further fraction of $A$-occurences which are also $B$-occurences ($P(B\vert A)$). The other form for $P(A\vert B)$ follows by interchanging the roles of $A$ and $B$.

The theorem acquires its application to statistical inference when we think of $A$ as a hypothesis which is being tested by measuring some data $B$. In real life, with noisy and incomplete data, we never have the luxury of measuring $A$ directly, but only something depending on it in a nonunique fashion. If we understand this dependence, i.e understand our experiment, we know $P(B\vert A)$. If only, (and this is a big IF!), someone gave us $P(A)$, then we would be able to compute the dependence of $P(A\vert B)$ on $A$ from Bayes theorem.

$$P(A\vert B)=P(B\vert A)P(A)/P(B).$$

Going from $P(B\vert A)$ to $P(A\vert B)$ may not seem to be a big step for a man, but it is a giant step for mankind. It now tells us the probability of different hypotheses $A$ being true based on the given data $B$. Remember, this is the real world. More than one hypothesis is consistent with a given set of data, so the best we can do is narrow down the possibilities. (If ``hypothesis'' seems too abstract, think of it as a set of numbers which occur as parameters in a given model of the real world)