Lecture_05.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 5: Conditioning Continued, Law of Total Probability\n",
    "\n",
    "\n",
    "## Stat 110, Prof. Joe Blitzstein, Harvard University\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Thinking conditionally is a condition for thinking."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# _How can we solve a problem?_\n",
    "\n",
    "1. Try simple and/or extreme cases.\n",
    "1. Try to break the problem up into simpler pieces; recurse as needed.\n",
    "\n",
    "![title](images/L0501.png)\n",
    "\n",
    "Let $A_1, A_2, \\dots, A_n$ partition sample space $S$ into disjoint regions that sum up to $S$. Then\n",
    "\n",
    "\\begin{align}\n",
    "    P(B) &= P(B \\cap A_1) + P(B \\cap A_2) + \\dots + P(B \\cap A_n) \\\\\n",
    "         &= P(B|A_1)P(A_1) + P(B|A_2)P(A_2) + \\dots + P(B|A_n)P(A_n)\n",
    "\\end{align}\n",
    "\n",
    "Note that statistics is as much of an art as it is a science, and choosing the right partitioning is key. Poor choices of partitions may results in many, even more difficult to solve sub-problems.\n",
    "\n",
    "This is known as the __Law of Total Probability__. Conditional probability is important in its own right, and sometimes we use conditional probability to solve problems of unconditional probability, as above with $P(B)$.\n",
    "\n",
    "But conditional probability can be very subtle. You really need to think when using conditional probability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ex. 1\n",
    "\n",
    "Let's consider a 2-card hand drawn from a standard playing deck. What is the probability of drawing 2 aces, given that we know one of the cards is an ace?\n",
    "\n",
    "\\begin{align}\n",
    "    P(\\text{both are aces | one is ace}) &= \\frac{P(\\text{both are aces})}{P(\\text{one is ace})} \\\\\n",
    "    &= \\frac{P(\\text{both are aces})}{1 - P(\\text{neither is ace})} \\\\\n",
    "    &= \\frac{\\binom{4}{2}/\\binom{52}{2}}{1 - \\binom{48}{2}/\\binom{52}{2}} \\\\\n",
    "    &= \\frac{1}{33}\n",
    "\\end{align}\n",
    "\n",
    "But now think about this: What is the probability of drawing 2 aces, knowing that one of the cards __is the ace of spades__?\n",
    "\n",
    "\\begin{align}\n",
    "    P(\\text{both are aces | ace of spades}) &= P(\\text{other card is also an ace}) \\\\\n",
    "    &= \\frac{3}{51} \\\\\n",
    "    &= \\frac{1}{17}\n",
    "\\end{align}\n",
    "\n",
    "_Notice how the fact that we know we have the ace of spades nearly doubles the probability of having 2 aces._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ex. 2\n",
    "\n",
    "Suppose there is a test for a disease, and this test is touted as being \"95% accurate\". The disease in question afflicts 1% of the population. Now say that there is a patient who tests positive for this disease under this test.\n",
    "\n",
    "First we define the events in question:\n",
    "\n",
    "Let $D$ be the event that the patient actually has the disease.\n",
    "\n",
    "Let $T$ be the event that the patient tests positive.\n",
    "\n",
    "Since that phrase \"95% accurate\" is ambiguous,  we need to clarify that.\n",
    "\n",
    "\\begin{align}\n",
    "    P(T|D) = P(T^c|D^c) = 0.95\n",
    "\\end{align}\n",
    "\n",
    "In other words, __conditioning on whether or not the patient has the disease__, we will assume that the test is 95% accurate.\n",
    "\n",
    "_What exactly are we trying to find?_\n",
    "\n",
    "What the patient really wants to know is not $P(T|D)$, which is the accuracy of the test; but rather $P(D|T)$, or the probability she has the disease given that the test returns positive. Fortunately, we know how $P(T|D)$ relates to $P(D|T)$.\n",
    "\n",
    "\\begin{align}\n",
    "    P(D|T) &= \\frac{P(T|D)P(D)}{P(T)} ~~~~ & &\\text{... Bayes Rule} \\\\\n",
    "    &= \\frac{P(T|D)P(D)}{P(T|D)P(D) + P(T|D^c)P(D^c)} ~~~~ &  & \\text{... by the Law of Total Probability} \\\\\n",
    "    &= \\frac{(0.95)(0.01)}{(0.95)(0.01) + (0.05)(0.99)} ~~~~ & & \\text{... the rarity of the disease competes with the rarity of true negatives}\\\\\n",
    "    &\\approx 0.16\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Common Pitfalls\n",
    "\n",
    "1. Mistaking $P(A|B)$ for $P(B|A)$. This is also known as the [Prosecutor's Fallacy](https://en.wikipedia.org/wiki/Prosecutor%27s_fallacy), where instead of asking about the _probability of guilt (or innocence) given all the evidence_, we make the mistake of concerning ourselves with the _probability of the evidence given guilt_. An example of the Prosecutor's Fallacy is the [case of Sally Clark](https://en.wikipedia.org/wiki/Sally_Clark).\n",
    "\n",
    "1. Confusing _prior_ $P(A)$ with _posterior_ $P(A|B)$. Observing that event $A$ occurred does __not__ mean that $P(A) = 1$. But $P(A|A) = 1$ and $P(A) \\neq 1$.\n",
    "\n",
    "1. Confusing _independence_ with __conditional independence__. This is more subtle than the other two.\n",
    "\n",
    "### Definition\n",
    "\n",
    "Events $A$ and $B$ are __conditionally independent__ given event $C$, if\n",
    "\n",
    "\\begin{align}\n",
    "    P(A \\cap B | C) = P(A|C)P(B|C)\n",
    "\\end{align}\n",
    "\n",
    "In other words, conditioning on event $C$ does not give us any additional information on $A$ or $B$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _Does conditional independence given $C$ imply unconditional independence? _\n",
    "\n",
    "### Ex. Chess Opponent of Unknown Strength\n",
    "\n",
    "Short answer, _no_.\n",
    "\n",
    "Consider playing a series of 5 games against a chess opponent of unknown strength. If we won all 5 games, then we would have a pretty good idea that we are the better chess player. So winning each successive game actually is providing us with information about the strength of our opponent.\n",
    "\n",
    "If we had prior knowledge about the strength of our opponent, meaning we condition on the strength of our opponent, then winning one game would not provide us with any additional information on the probability of winning the next.\n",
    "\n",
    "But if we do not condition on the strength of our opponent, meaning that we have no prior knowledge about our opponent, then successively winning a string a games actually _does_ give us information about the probability of winning the next game.\n",
    "\n",
    "So the games are conditionally independent given the strength of our opponent, but _not_ independent unconditionally."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _Does unconditional independence imply conditional independence given $C$?_\n",
    "\n",
    "### Ex. Popcorn and the Fire Alarm\n",
    "\n",
    "Again, short answer is _no_.\n",
    "\n",
    "You can see this in the case of some phenomenon with multiple causes.\n",
    "\n",
    "Let $A$ be the event of the fire alarm going off.\n",
    "\n",
    "Let $F$ be the event of a fire.\n",
    "\n",
    "Let $C$ be the event of someone making popcorn.\n",
    "\n",
    "Suppose that either $F$ (an actual fire) or $C$ (the guy downstairs popping corn) will result in $A$, the fire alarm going off.\n",
    "Further suppose that $F$ and $C$ are independent: knowing that there's a fire $F$ doesn't tell me anything about anyone making popcorn $C$; and vice versa.\n",
    "\n",
    "But the probability of a fire given that the alarm goes off __and__ no one is making any popcorn is given by  $P(F|A,C^c) = 1$. After all, if the fire alarm goes off and no one is making popcorn, there can only be one explanation: _there must be a fire_.\n",
    "\n",
    "So $F$ and $C$ may be independent, but they are not _conditionally independent_ when we condition on event $A$. Knowing that nobody is making any popcorn when the alarm goes off can only mean that there is a fire.\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View [Lecture 5: Conditioning Continued, Law of Total Probability | Statistics 110](http://bit.ly/2PncEYX) on YouTube."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}