Lecture Notes : Independence

Sources 


Getting into the thick of it. 

Last lecture - Given a situation will create a new sample space because we have encountered a new information. Thus the older probability must be changed. 

P(AB)=P(AB)P(B)P(A|B) = \frac{P(A \cap B)}{P(B)}  

Total probability can be seen as 

P(B)=P(A)P(BA)+P(Ac)P(BAc)P(B) = P(A)P(B|A) + P(A^c)P(B|A^c)

Moving to Bayes’ rule  

P(AiB)=P(Ai)P(BAi)P(B)P(A_i |B) = \frac{P(A_i)P(B|A_i)}{P(B)}

Conditional probability using tree diagram. 

Independence 

We have two events and we can call them independent when you get information about event B and it does not change your beliefs about A 

Mathematically speaking 

P(BA)=P(B)P(B|A) = P(B)

This is where the multiplication. That is P(AB)=P(A).P(B)P(A \cap B) = P(A). P(B)

We will say that two events are independent if and only if their probability of happening simultaneously is equal to the product of their two individual probabilities

We can have events of zero probability. If an event A has zero probability it’s independent. 

There's nothing wrong with that. If A has 0 probability, then A intersection B will also have zero probability, because it's an even smaller event. And so we're going to get zero is equal to zero. A corollary of what I just said, if an event A has zero probability, it's actually independent of any other event in our model, because we're going to get zero is equal to zero. And the definition is going to be satisfied

The probability of A given B is equal to 0. Probability of A is equal to 1/3. So again, these two are different. So we had some initial beliefs about A, but as soon as we are told that B occurred, our beliefs about A changed. And so since our beliefs changed, that means that B conveys information about A.

AUDIENCE: Can you draw independence on a Venn diagram? PROFESSOR: No, the Venn diagram is never enough to decide independence. So the typical picture in which you're going to have independence would be one event this way, and another event this way. You need to take the probability of this times the probability of that, and check that, numerically, it's equal to the probability of this intersection. So it's more than a Venn diagram. Numbers need to come out right.

Conditional Independence

So if we're told that the event C has happened, then were transported in a conditional universe where the only thing that matters are conditional probabilities. And this is just the same plain, previous definition of independence, but applied in a conditional universe

P(ABC)=P(AC)P(BC)P(A \cap B | C) = P(A|C)P(B|C) — Conditional Independence  

The mathematical definition that actually does the job, and leads to all the formulas of this kind, is the following. We're going to say that the collection of events are independent if we can find the probability of their joint occurrence by just multiplying probabilities. And that will be true even if you look at sub-collections of these events.