Lecture_21.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 21: Covariance, Correlation, Variance of a sum, Variance of Binomial & Hypergeometric\n",
    "\n",
    "\n",
    "## Stat 110, Prof. Joe Blitzstein, Harvard University\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Definition\n",
    "\n",
    "Covariance of any 2 random variables $X, Y$ is defined as\n",
    "\n",
    "\\begin{align}\n",
    "  \\operatorname{Cov}(X, Y) &= \\mathbb{E}\\left( (X - \\mathbb{E}(X)) (Y - \\mathbb{E}(Y)) \\right) \\\\\n",
    "  &= \\mathbb{E}(XY) - \\mathbb{E}(X) \\, \\mathbb{E}(Y) & \\quad \\text{similar to definition of variance}\n",
    "\\end{align}\n",
    "\n",
    "Covariance is a measure of how $X, Y$ might vary _in tandem_. \n",
    "\n",
    "If the product of $(X - \\mathbb{E}(X))$ and $(Y - \\mathbb{E}(Y))$ is _positive_ then that means that both values are either _positive_ ($X,Y$ tend to be greater than their respective means); or they are both _negative_ ($X,Y$ tend to be less than their means).\n",
    "\n",
    "Correlation is defined in terms of covariance, as you will see in a bit.\n",
    "\n",
    "### Properties\n",
    "\n",
    "\\begin{align}\n",
    "  & \\text{[1]} & \\operatorname{Cov}(X,X) &= \\operatorname{Var}(X) \\\\\n",
    "  \\\\\n",
    "  & \\text{[2]} & \\operatorname{Cov}(X,Y) &= \\operatorname{Cov}(Y,X) \\\\\n",
    "  \\\\\n",
    "  & \\text{[3]} & \\operatorname{Cov}(X, c) &= 0 & \\quad \\text{for some constant }c \\\\\n",
    "  \\\\\n",
    "  & \\text{[4]} & \\operatorname{Cov}(cX, Y) &= c \\, \\operatorname{Cov}(X,Y) & \\quad \\text{bilinearity} \\\\\n",
    "  \\\\\n",
    "  & \\text{[5]} & \\operatorname{Cov}(X, Y+Z) &= \\operatorname{Cov}(X,Y) + \\operatorname{Cov}(X,Z) & \\quad \\text{bilinearity} \\\\\n",
    "  \\\\\n",
    "  & \\text{[6]} & \\operatorname{Cov}(X+Y, Z+W) &= \\operatorname{Cov}(X,Z) + \\operatorname{Cov}(X,W) + \\operatorname{Cov}(Y,Z) + \\operatorname{Cov}(Y,W) & \\quad \\text{applying (5)} \\\\\n",
    "  \\\\\n",
    "  & \\Rightarrow & \\operatorname{Cov} \\left( \\sum_{i=1}^{m} a_{i} \\, X_{i},  \\sum_{j=1}^{n} b_{j} \\, Y_{j} \\right) &= \\sum_{i,j} a_{i} \\, b_{j} \\, Cov \\left( X_{i}, Y_{j} \\right) \\\\\n",
    "  \\\\\n",
    "  & \\text{[7]} & \\operatorname{Var}(X_1 + X_2) &=  \\operatorname{Cov}(X_1+X_2, X_1+X_2) \\\\\n",
    "  & & &= \\operatorname{Cov}X_1, X_1) + \\operatorname{Cov}(X_1, X_2) + \\operatorname{Cov}(X_2, X_1) + \\operatorname{Cov}(X_2, X_2) \\\\\n",
    "  & & &= \\operatorname{Var}(X_1) + \\operatorname{Var}(X_2) + 2 \\, \\operatorname{Cov}(X_1, X_2) \\\\\n",
    "  \\\\\n",
    "  & \\Rightarrow & \\operatorname{Var}(X_1 + X_2) &= \\operatorname{Var}(X_1) + \\operatorname{Var}(X_2) & \\quad \\text{if }X_1, X_2 \\text{ are uncorrelated} \\\\\n",
    "  \\\\\n",
    "  & \\Rightarrow & \\operatorname{Var}(X_1 + \\dots + X_n) &= \\operatorname{Var}(X_1) + \\dots + \\operatorname{Var}(X_n) + 2 \\, \\sum_{i \\lt j} Cov(X_i, X_j) \\\\\n",
    "\\end{align} \n",
    "\n",
    "#### Bilinearity\n",
    "\n",
    "If you imagine treating one variable as fixed and then start working with the other, it sort of looks like _linearity_. It also sort of looks like the distributive property, too. \n",
    "\n",
    "Bilinearity lets us\n",
    "\n",
    "\\begin{align}\n",
    "  \\operatorname{Cov}(\\lambda X, Y) &= \\lambda \\, \\operatorname{Cov}(X, Y) = \\operatorname{Cov}(X, \\lambda Y) & \\quad \\text{freely move scaling factors} \\\\\n",
    "  \\\\\n",
    "  \\operatorname{Cov}(X, Y_1 + Y_2) &= \\operatorname{Cov}(X, Y_1) + \\operatorname{Cov}(X, Y_2) \\\\\n",
    "  \\operatorname{Cov}(X_1 + X_2, Y) &= \\operatorname{Cov}(X_1, Y) + \\operatorname{Cov}(X_2, Y) & \\quad \\text{distribution in }\\operatorname{Cov} \\text{ operator} \\\\\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Theorem If $X,Y$ are independent, then they are also uncorrelated, i.e., $\\operatorname{Cov}(X,Y) = 0$\n",
    "\n",
    "The converse is not true, however: $\\operatorname{Cov}(X,Y) = 0$ does not necessarily mean that $X,Y$ are independent.\n",
    "\n",
    "Consider $Z \\sim \\mathcal{N}(0,1)$, and $X \\sim Z, Y \\sim Z^2$.\n",
    "\n",
    "\\begin{align}\n",
    "  \\operatorname{Cov}(X,Y) &= \\mathbb{E}(XY) - \\mathbb{E}(X)\\,\\mathbb{E}(Y) \\\\\n",
    "  &= \\mathbb{E}(Z^3) - \\mathbb{E}(Z) \\, \\mathbb{E}(Z^2) \\\\\n",
    "  &= 0 - 0 & \\quad \\text{odd moments of }Z \\text{ are 0} \\\\\n",
    "  &= 0\n",
    "\\end{align}\n",
    "\n",
    "But given $X \\sim Z$, we know *exactly* everything about $Y$. And knowing $Y$, gives us everything about $X$, _except for the sign_.\n",
    "\n",
    "So $X,Y$ are dependent, yet their covariance (and hence correlation) is 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Definition\n",
    "\n",
    "Correlation is defined in terms of covariance.\n",
    "\n",
    "\\begin{align}\n",
    "  \\operatorname{Corr}(X,Y) &= \\frac{\\operatorname{Cov}(X,Y)}{\\sigma_{X} \\, \\sigma_{Y}} = \\frac{\\operatorname{Cov}(X,Y)}{\\sqrt{\\operatorname{Var}(X)} \\, \\sqrt{\\operatorname{Var}(Y)}} \\\\\n",
    "  \\\\\n",
    "  &= \\operatorname{Cov} \\left( \\frac{X-\\mathbb{E}(X)}{\\sigma_{X} }, \\frac{Y-\\mathbb{E}(Y)}{\\sigma_{Y}} \\right) & \\quad \\text{standardize first, then find covariance} \\\\\n",
    "\\end{align}\n",
    "\n",
    "Correlation is dimensionless, and like standard deviation, is unit-less.\n",
    "\n",
    "Correlation also ranges from -1 to 1. This is what happens when we divide by $\\sqrt{\\operatorname{Var}(X)} \\, \\sqrt{\\operatorname{Var}(Y)}$\n",
    "\n",
    "### Theorem $-1 \\le \\operatorname{Corr}(X,Y) \\le 1$\n",
    "\n",
    "Without loss of generality, assume that $X,Y$ are already standarized (mean 0, variance 1).\n",
    "\n",
    "\\begin{align}\n",
    "  \\operatorname{Va}r(X + Y) &=\\operatorname{Var}(X) + \\operatorname{Var}(Y) + 2\\, \\operatorname{Corr}(X,Y) \\\\\n",
    "  &= 1 + 1 + 2 \\, \\rho & \\quad \\text{ where } \\rho = \\operatorname{Corr}(X,Y) \\\\\n",
    "  &= 2 + 2 \\, \\rho \\\\\n",
    "  0 &\\le 2 + 2 \\, \\rho & \\quad \\text{since }Var \\ge 0 \\\\\n",
    "  0 &\\le 1 + \\rho & \\Rightarrow \\rho \\text{ has a floor of }-1 \\\\\n",
    "  \\\\\n",
    "  \\operatorname{Var}(X - Y) &= \\operatorname{Var}(X) + \\operatorname{Var}(-Y) - 2\\, \\operatorname{Corr}(X,Y) \\\\\n",
    "  &= 1 + 1 - 2 \\, \\rho & \\quad \\text{ where } \\rho = \\operatorname{Corr}(X,Y) \\\\\n",
    "  &= 2 - 2 \\, \\rho \\\\\n",
    "  0 &\\le 2 - 2 \\, \\rho & \\quad \\text{since }Var \\ge 0 \\\\\n",
    "  0 &\\le 1 - \\rho & \\Rightarrow \\rho \\text{ has a ceiling of }1 \\\\\n",
    "  \\\\\n",
    "  &\\therefore -1 \\le \\operatorname{Corr}(X,Y) \\le 1 &\\quad \\blacksquare\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## $\\operatorname{Cov}$ in a Multinomial\n",
    "\n",
    "Given $(X_1, \\dots , X_k) \\sim \\operatorname{Mult}(n, \\vec{p})$, find $\\operatorname{Cov}(X_i, X_j)$ for all $i,j$.\n",
    "\n",
    "\\begin{align}\n",
    "  \\text{case 1, where } i=j \\text{ ...} \\\\\n",
    "  \\\\\n",
    "  \\operatorname{Cov}(X_i, X_j) &= \\operatorname{Var}(X_i) \\\\\n",
    "  &= n \\, p_i \\, (1 - p_i) \\\\\n",
    "  \\\\\n",
    "  \\\\\n",
    "  \\text{case 2, where } i \\ne j \\text{ ...} \\\\\n",
    "  \\\\\n",
    "  \\operatorname{Var}(X_i + X_j) &= \\operatorname{Var}(X_i) + \\operatorname{Var}(X_j) + 2 \\, \\operatorname{Cov}(X_i, X_j) & \\quad \\text{property [7] above} \\\\\n",
    "  n \\, (p_i + p_j) \\, (1 - (p_i + p_j)) &= n \\, p_i \\, (1 - p_i) + n \\, p_j \\, (1 - p_j) + 2 \\, \\operatorname{Cov}(X_i, X_j) & \\quad \\text{lumping property} \\\\\n",
    "  2 \\, \\operatorname{Cov}(X_i, X_j) &= n \\, (p_i + p_j) \\, (1 - (p_i + p_j)) - n \\, p_i \\, (1 - p_i) - n \\, p_j \\, (1 - p_j) \\\\\n",
    "  &= n \\left( (p_i + p_j) - (p_i + p_j)^2 - p_i + p_i^2 - p_j + p_j^2  \\right) \\\\\n",
    "  &= n ( -2 \\, p_i \\, p_j ) \\\\\n",
    "  \\operatorname{Cov}(X_i, X_j) &= - n \\, p_i \\, p_j\n",
    "\\end{align}\n",
    "\n",
    "Notice how case 2 where $i \\ne j$ yields $\\operatorname{Cov}(X_i, X_j) = - n \\, p_i \\, p_j$, which is negative. This should agree with your intuition that for $(X_1, \\dots , X_k) \\sim \\operatorname{Mult}(n, \\vec{p})$, categories $i,j$ are competing with another, and so they cannot be positively correlated nor independent of each other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Variance of a Binomial\n",
    "\n",
    "Applying what we now know of covariance, we can obtain the variance of $X \\sim \\operatorname{Bin}(n, p)$.\n",
    "\n",
    "We can describe $X = X_1 + \\dots + X_n$ where $X_i$ are i.i.d. $\\operatorname{Bern}(p)$.\n",
    "\n",
    "Now consider _indicator random variables_. Let $I_A$ be the indicator of event $A$. \n",
    "\n",
    "\\begin{align}\n",
    "  I_A &\\in \\{0, 1\\} \\\\\n",
    "  \\\\\n",
    "  \\Rightarrow I_A^2 &= I_A \\\\\n",
    "  I_A^3 &= I_A \\\\\n",
    "  & \\text{...} \\\\\n",
    "  \\\\\n",
    "  \\Rightarrow I_A \\, I_B &= I_{A \\cap B} & \\quad \\text{for another event } B \\\\\n",
    "  \\\\\n",
    "  Var(X_i) &=  \\mathbb{E}X_i^2 - ( \\mathbb{E}X_i)^2 \\\\\n",
    "  &= p_i - p_i^2 \\\\\n",
    "  &= p_i \\, (1 - p_i) \\\\\n",
    "  &= p \\, q & \\quad \\text{variance of Bernoulli} \\\\\n",
    "  \\\\\n",
    "  \\Rightarrow Var(X) &= Var(X_i + \\dots + X_n) \\\\\n",
    "  &= \\operatorname{Var}(X_i) + \\dots + \\operatorname{Var}(X_n) + 2 \\, \\left( \\sum_{i \\ne j} Cov(X_i, X_j) \\right) \\\\\n",
    "  &= n \\, p \\, q + 2 \\, (0) & \\quad \\operatorname{Cov}(X_i,X_j) = 0 \\text{ since } X_i \\text{ are i.i.d.} \\\\\n",
    "  &= n \\, p \\, q\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Variance of Hypergeometric\n",
    "\n",
    "Given $X \\sim \\operatorname{HGeom}(w, b, n)$ where \n",
    "\n",
    "\\begin{align}\n",
    "  X &= X_1 + \\dots + X_n \\\\\n",
    "  \\\\\n",
    "  X_j &= \n",
    "  \\begin{cases}\n",
    "    1, &\\text{ if }j^{th} \\text{ ball is White}  \\\\\n",
    "    0, &\\text{ otherwise } \\\\\n",
    "  \\end{cases} \\\\\n",
    "\\end{align}\n",
    "\n",
    "Recall that the balls are sampled _without replacement_, so the draws are not independent.\n",
    "\n",
    "However, for any draw, it is not the case that any particular ball would prefer to be drawn for that particular iteration. So there is some symmetry we can leverage!\n",
    "\n",
    "\\begin{align}\n",
    "  \\operatorname{Var}(X) &= n \\, \\operatorname{Var}(X_1) + 2 \\, \\binom{n}{2} \\operatorname{Cov}(X_1, X_2) & \\quad \\text{symmetry!} \\\\\n",
    "  \\\\\n",
    "  \\text{now } \\operatorname{Cov}(X_1, X_2) &= \\mathbb{E}(X_1 X_2) - \\mathbb{E}(X_1) \\, \\mathbb{E}(X_2) \\\\\n",
    "  &= \\frac{w}{w+b} \\, \\frac{w-1}{w+b-1} - \\left( \\frac{w}{w+b} \\right)^2 & \\quad \\text{recall } I_A \\, I_B = I_{A \\cap B} \\text{; and symmetry} \\\\\n",
    "\\end{align}\n",
    "\n",
    "Prof. Blitzstein runs out of time, but the rest is just algebra. _To be concluded in the next lecture._\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix: Variance, Covariance and Linear Algebra\n",
    "\n",
    "Let's relate what we know about variance, covariance and correlation with the concept of the [dot product](https://www.mathsisfun.com/algebra/vectors-dot-product.html) in linear algebra:\n",
    "\n",
    "\\begin{align}\n",
    "  \\operatorname{Cov}(X, Y) &= X \\cdot Y \\\\\n",
    "            &= x_1 \\, y_1 + \\dots + x_n \\, y_n \\\\\n",
    "  \\\\\n",
    "  \\operatorname{Var}(X) &= \\operatorname{Cov}(X, X) \\\\\n",
    "         &= X \\cdot X \\\\\n",
    "         &= x_1 \\, x_1 + \\dots + x_n \\, x_n \\\\\n",
    "         &= |X|^2 \\\\\n",
    "  \\\\\n",
    "  \\Rightarrow |X| &=  \\sigma_{X} \\\\\n",
    "                  &= \\sqrt{\\operatorname{Var}(X)} \\\\\n",
    "                  &= \\sqrt{x_1 \\, x_1 + \\dots + x_n \\, x_n} & \\quad \\text{the size of }\\vec{X} \\text{ is its std. deviation} \\\\\n",
    "\\end{align}\n",
    "\n",
    "Now let's take this a bit further...\n",
    "\n",
    "\\begin{align}\n",
    "  X \\cdot Y &= |X| \\, |Y| \\, cos \\theta \\\\\n",
    "  \\\\\n",
    "  \\Rightarrow cos \\theta &= \\frac{X \\cdot Y}{|X| \\, |Y|} \\\\\n",
    "                         &= \\frac{\\operatorname{Cov}(X, Y)}{\\sigma_{X} \\, \\sigma_{Y}} \\\\\n",
    "                         &= \\rho & \\quad \\text{... otherwise known as }\\operatorname{Corr}(X,Y) \\\\\n",
    "\\end{align}\n",
    "\n",
    "Since $\\rho_{X,Y} = cos_{X,Y} \\theta$, we can see that the correlation of $X,Y$ is also the factor that projects vector $X$ onto $Y$, or vice versa. Now you can see why we can use cosine similarity as a metric to measure how close 2 vectors $X,Y$ are, since it is $\\operatorname{Corr}(X,Y)$.\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View [Lecture 21: Covariance and Correlation | Statistics 110](http://bit.ly/2MXxn89) on YouTube."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}