Lecture_18.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 18: MGFs to get moments of Expo and Normal, sums of Poissons, joint distributions\n",
    "\n",
    "\n",
    "## Stat 110, Prof. Joe Blitzstein, Harvard University\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MGF for $\\operatorname{Expo}(1)$\n",
    "\n",
    "Let $X \\sim \\operatorname{Expo}(1)$.\n",
    "\n",
    "We begin by finding the MGF of $X$:\n",
    "\n",
    "\\begin{align}\n",
    "  M(t) &= \\mathbb{E}(e^{tX}) &\\quad \\text{ definition of MGF} \\\\\n",
    "       &= \\int_{0}^{\\infty} e^{-x} \\, e^{tx} dx \\\\\n",
    "       &= \\int_{0}^{\\infty} e^{-x(1-t)} dx \\\\\n",
    "       &= \\boxed{\\frac{1}{1-t}} &\\quad \\text{ for } t < 1\n",
    "\\end{align}\n",
    "\n",
    "In finding the moments, by definition we have:\n",
    "\n",
    "* $M'(0) = \\mathbb{E}(X)$\n",
    "* $M''(0) = \\mathbb{E}(X^2)$\n",
    "* $M'''(0) = \\mathbb{E}(X^3)$\n",
    "* ... and so on ...\n",
    "\n",
    "Even though finding derivatives of $\\frac{1}{1-t}$ is not all that bad, it is nevertheless annoying busywork. But since we know that the $n^{th}$ moment is the coefficient of the $n^{th}$ term of the Taylor Expansion of $X$, we can leverage that fact instead.\n",
    "\n",
    "\\begin{align}\n",
    "  \\frac{1}{1-t} &= \\sum_{n=0}^{\\infty} t^n &\\quad \\text{ for } |t| < 1 \\\\ \n",
    "                &= \\sum_{n=0}^{\\infty} \\frac{n! \\, t^n}{n!} &\\quad \\text{ since we need the form } \\sum_{n=0}^{\\infty} \\left( \\frac{\\mathbb{E}(X^{n}) \\, t^{n}}{n!}\\right) \\\\\n",
    "   \\\\\n",
    "   \\Rightarrow \\mathbb{E}(X^n) &= \\boxed{n!}\n",
    "\\end{align}\n",
    " \n",
    "And now we can simply _generate_ arbitary moments for r.v. $X$!\n",
    "\n",
    "* $\\mathbb{E}(X) = 1! = 1$\n",
    "* $\\mathbb{E}(X^2) = 2! = 2$\n",
    "* $\\Rightarrow \\operatorname{Var}(X) = \\mathbb{E}(X^2) - \\mathbb{E}X^2 = 2 - 1 = 1$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## MGF for $\\operatorname{Expo}(\\lambda)$\n",
    " \n",
    "Let $Y \\sim \\operatorname{Expo}(\\lambda)$.\n",
    "\n",
    "We begin with\n",
    "\n",
    "\\begin{align}\n",
    "  \\text{let } X &= \\lambda Y \\\\\n",
    "  \\text{and so } X &= \\lambda Y \\sim \\operatorname{Expo}(1) \\\\\n",
    "  \\\\\n",
    "  \\text{then } Y &= \\frac{X}{\\lambda} \\\\\n",
    "  Y^n &= \\frac{X^n}{\\lambda^n} \\\\\n",
    "  \\\\\n",
    "  \\Rightarrow \\mathbb{E}(Y^n) &= \\frac{\\mathbb{E}(X^n)}{\\lambda^n} \\\\\n",
    "  &= \\boxed{\\frac{n!}{\\lambda^n}}\n",
    "\\end{align}\n",
    "\n",
    "And as before, we now can simply _generate_ arbitary moments for r.v. $Y$!\n",
    "\n",
    "* $\\mathbb{E}(Y) = \\frac{1!}{\\lambda^1} = \\frac{1}{\\lambda}$\n",
    "* $\\mathbb{E}(Y^2) = \\frac{2!}{\\lambda^2} = \\frac{2}{\\lambda^2}$\n",
    "* $\\Rightarrow \\operatorname{Var}(Y) = \\mathbb{E}(Y^2) - \\mathbb{E}Y^2 = \\frac{2}{\\lambda^2} - \\left(\\frac{1}{\\lambda}\\right)^2 = \\frac{1}{\\lambda^2}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## MGF for standard Normal $\\mathcal{N}(0, 1)$\n",
    "\n",
    "Let $Z \\sim \\mathcal{N}(0,1)$; find **all** its moments.\n",
    "\n",
    "We have seen before that, by symmetry, $\\mathbb{E}(Z^{2n+1}) = 0$ for all odd values of $n$.\n",
    "\n",
    "So we will focus in on the _even_ values of $n$.\n",
    "\n",
    "Now the MGF $M(t) = e^{t^2/2}$. Without taking _any_ derivatives, we can immediately Taylor expand that, since it is continuous everywhere.\n",
    "\n",
    "\\begin{align}\n",
    "  M(t) &= e^{t^2/2} \\\\\n",
    "       &= \\sum_{n=0}^{\\infty} \\frac{\\left(t^2/2\\right)^n}{n!} \\\\\n",
    "       &= \\sum_{n=0}^{\\infty} \\frac{t^{2n}}{2^n \\, n!} \\\\\n",
    "       &= \\sum_{n=0}^{\\infty} \\frac{(2n)! \\, t^{2n}}{2^n \\, n! \\, (2n)!} &\\quad \\text{ since we need the form } \\sum_{n=0}^{\\infty} \\left( \\frac{\\mathbb{E}(X^{n}) \\, t^{n}}{n!}\\right) \\\\\n",
    "       \\\\\n",
    "       \\Rightarrow \\mathbb{E}(Z^{2n}) &= \\boxed{\\frac{(2n)!}{2^n \\, n!}}\n",
    "\\end{align}\n",
    "\n",
    "Let's double-check this with what know about $\\mathbb{Var}(Z)$\n",
    "\n",
    "* by symmetry, we know that $\\mathbb{E}(Z) = 0$\n",
    "* at $n = 1$, $\\mathbb{E}(Z^2) = \\frac{2!}{2 \\times 1!} = 1$ \n",
    "* $\\Rightarrow \\operatorname{Var}(Z) = \\mathbb{E}(Z^2) - \\mathbb{E}Z^2 = 1 - 0 = 1$\n",
    "* at $n = 2$, $\\mathbb{E}(Z^4) = \\frac{4!}{4 \\times 2!} = 3$\n",
    "* at $n = 3$, $\\mathbb{E}(Z^6) = \\frac{6!}{8 \\times 3!} = 15$\n",
    "\n",
    "And so you might have noticed a pattern here. Let us rewrite those even moments once more:\n",
    "\n",
    "* at $n = 1$, $\\mathbb{E}(Z^2) = 1$ \n",
    "* at $n = 2$, $\\mathbb{E}(Z^4) = 1 \\times 3 = 3$ \n",
    "* at $n = 3$, $\\mathbb{E}(Z^6) = 1 \\times 3 \\times 5 = 15$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## MGF for $\\operatorname{Pois}(\\lambda)$\n",
    "\n",
    "Let $X \\sim \\operatorname{Pois}(\\lambda)$; now let's consider MGFs and how to use them to find sums of random variables (convolutions).\n",
    "\n",
    "\\begin{align}\n",
    "  M(t) &= \\mathbb{E}(e^{tX}) \\\\\n",
    "       &= \\sum_{k=0}^{\\infty} e^{tk} \\, \\frac{\\lambda^k e^{-\\lambda}}{k!} \\\\\n",
    "       &= e^{-\\lambda} \\sum_{k=0}^{\\infty} \\frac{\\lambda^k e^{tk}}{k!} \\\\\n",
    "       &= e^{-\\lambda} e^{\\lambda e^t} &\\quad \\text{but the right is just another Taylor expansion} \\\\\n",
    "       &= \\boxed{e^{\\lambda (e^t - 1)}}\n",
    "\\end{align}\n",
    "\n",
    "\n",
    "Now let's let $Y \\sim \\operatorname{Pois}(\\mu)$, and it is independent of $X$. Find the distribution of $(X + Y)$.\n",
    "\n",
    "You may recall that with MGFs, all we need to do is _multiply_ the MGFs.\n",
    "\n",
    "\\begin{align}\n",
    "  e^{\\lambda (e^t - 1)} \\, e^{\\mu (e^t - 1)} &= e^{(\\lambda + \\mu)(e^t - 1)} \\\\\n",
    "  \\\\\n",
    "  \\Rightarrow X + Y &\\sim \\mathcal{Pois}(\\lambda + \\mu)\n",
    "\\end{align}\n",
    "\n",
    "When adding a Poisson distribution $X$ with another independent Poisson distribution $Y$, the resulting convolution $X + Y$ will also be Poisson. This interesting relation only happens with Poisson distributions.\n",
    "\n",
    "Now think about what happens with $X$ and $Y$ are _not_ independent.\n",
    "\n",
    "Let $Y = X$, so that $X + Y = 2X$, which is clearly *not* Poisson, as\n",
    "* $X + Y = 2X$ is an even function which cannot be Poisson, since Poisson can take on all values both even _and_ odd\n",
    "* Mean $\\mathbb{E}(2x) = 2\\lambda$, but $\\operatorname{Var}(2X) = 4\\lambda$, and since the mean and variance are _not_ equal, this cannot be Poisson"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some definitions\n",
    "\n",
    "In the most basic case of two r.v. in a joint distribution, consider both r.v.'s _together_:\n",
    "\n",
    "> **Joint CDF**\n",
    ">\n",
    "> In the general case, the joint CDF fof two r.v.'s is $ F(x,y) = P(X \\le x, Y \\le y)$\n",
    "\n",
    "<p/> \n",
    "\n",
    "> **Joint PDF**\n",
    "> $f(x, y)$ such that, in the *continuous* case $P((X,Y) \\in B) = \\iint_B f(x,y)\\,dx\\,dy$\n",
    "\n",
    "#### Joint PMF\n",
    "\n",
    "$f(x, y)$ such that, in the *discrete* case\n",
    "\n",
    "\\begin{align}\n",
    "  P(X=x, Y=y)\n",
    "\\end{align}\n",
    "\n",
    "We also can consider a single r.v. of a joint distribution:\n",
    "\n",
    "#### Independence and Joint Distributions\n",
    "\n",
    "$X, Y$ are independent iff $F(x,y) = F_X(x) \\, F_Y(y)$. \n",
    "\n",
    "\\begin{align}\n",
    "  P(X=x, Y=y) &= P(X=x) \\, P(Y=y) &\\quad \\text{discrete case} \\\\\n",
    "  \\\\\\\\\n",
    "  f(x, y) &= f_X(x) \\, f_Y(y)     &\\quad \\text{continuous case}\n",
    "\\end{align}\n",
    "\n",
    "... with the caveat that this must be so for *all* $\\text{x, y} \\in \\mathbb{R}$\n",
    "\n",
    "#### Marginals (and how to get them)\n",
    "\n",
    "$P(X \\le x)$ is the *marginal distribution* of $X$, where we consider one r.v. at a time.\n",
    "\n",
    "In the case of a two-r.v. joint distribution, we can get the marginals by using the joint distribution itself:\n",
    "\n",
    "\\begin{align}\n",
    "  P(X=x) &= \\sum_y P(X=x, Y=y) &\\quad \\text{marginal PMF, discrete case, for } x \\\\\n",
    "  \\\\\\\\\n",
    "  f_Y(y) &= \\int_{-\\infty}^{\\infty} f_{(X,Y)}(x,y) \\, dx &\\quad \\text{marginal PDF, continuous case, for } y\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Discrete Joint Distribution\n",
    "\n",
    "Let $X, Y$ be both Bernoulli. $X$ and $Y$ may be independent; or they might be dependent. They may or may not have the same $p$. But they are both related in the form of a *joint distribution*.\n",
    "\n",
    "We can lay out this joint distribution in a $2 \\times 2$ contigency table like below:\n",
    "\n",
    "\n",
    "|       | $Y=0$ | $Y=1$ |\n",
    "|-------|:-----:|:-----:|\n",
    "| $X=0$ |  2/6  |  1/6  |\n",
    "| $X=1$ |  2/6  |  1/6  |\n",
    "\n",
    "In order to be a joint distribution, all of the values in our contigency table must be positive; and they must all sum up to 1. The example above shows such a PMF.\n",
    "\n",
    "Let's add the marginals for $X$ and $Y$ to our $2 \\times 2$ contigency table:\n",
    "\n",
    "\n",
    "|       | $Y=0$ | $Y=1$ |  ...  |\n",
    "|:-----:|:-----:|:-----:|:-----:|\n",
    "| $X=0$ |  2/6  |  1/6  |  3/6  |\n",
    "| $X=1$ |  2/6  |  1/6  |  3/6  |\n",
    "|  ...  |  4/6  |  2/6  |       |\n",
    "\n",
    "\n",
    "Observe how in our example, we have:\n",
    "\n",
    "\\begin{align}\n",
    "  P(X=0,Y=0) &= P(X=0) \\, P(Y=0) \\\\\n",
    "             &= 3/6 \\times 4/6 = 12/36 &= \\boxed{2/6} \\\\\n",
    "  \\\\\n",
    "  P(X=0,Y=1) &= P(X=0) \\, P(Y=1) \\\\\n",
    "             &= 3/6 \\times 2/6 = 6/36 &= \\boxed{1/6} \\\\\n",
    "  P(X=1,Y=0) &= P(X=1) \\, P(Y=0) \\\\\n",
    "             &= 3/6 \\times 4/6 = 12/36 &= \\boxed{2/6} \\\\\n",
    "  \\\\\n",
    "  P(X=1,Y=1) &= P(X=1) \\, P(Y=1) \\\\\n",
    "             &= 3/6 \\times 2/6 = 6/36 &= \\boxed{1/6} \\\\\n",
    "\\end{align}\n",
    "\n",
    "and so you can see that $X$ and $Y$ are independent.\n",
    "\n",
    "Now here's an example of a two r.v. joint distribution where $X$ and $Y$ are _dependent_; check it out for yourself.\n",
    "\n",
    "|       | $Y=0$ | $Y=1$ |\n",
    "|:-----:|:-----:|:-----:|\n",
    "| $X=0$ |  1/3  |   0   |\n",
    "| $X=1$ |  1/3  |  1/3  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Continuous Joint Distribution\n",
    "\n",
    "Now say we had Uniform distributions on a square such that $x,y \\in [0,1]$. \n",
    "\n",
    "The joint PDF would be constant on/within the square; and 0 outside.\n",
    "\n",
    "\\begin{align}\n",
    "  \\text{joint PDF} &=\n",
    "    \\begin{cases}\n",
    "      c &\\quad \\text{if } 0 \\le x \\le 1 \\text{, }  0 \\le y \\le 1 \\\\\n",
    "      \\\\\n",
    "      0 &\\quad \\text{otherwise}\n",
    "    \\end{cases}\n",
    "\\end{align}\n",
    "\n",
    "![title](images/L1801.png)\n",
    "\n",
    "In 1-dimension space, if you integrate $1$ over some interval you get the _length_ of that interval.\n",
    "\n",
    "In 2-dimension space, if you integrate $1$ over some region, you get the _area_ of that region.\n",
    "\n",
    "Normalizing $c$, we know that $c = \\frac{1}{area} = 1$.\n",
    "\n",
    "Marginally, $X$ and $Y$ are independent $\\mathcal{Unif}(1)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Dependent, Continuous Joint Distribution\n",
    "\n",
    "Now say we had Uniform distributions on a _disc_ such that $x^2 + y^2 \\le 1$. \n",
    "\n",
    "\n",
    "#### Joint PDF\n",
    "\n",
    "In this case, the joint PDF is $\\operatorname{Unif}$ over the area of a disc centered at the origin with radius 1.\n",
    "\n",
    "\\begin{align}\n",
    "  \\text{joint PDF} &=\n",
    "    \\begin{cases}\n",
    "      \\frac{1}{\\pi} &\\quad \\text{if } x^2 + y^2 \\le 1 \\\\\n",
    "      \\\\\n",
    "      0 &\\quad \\text{otherwise}\n",
    "    \\end{cases}\n",
    "\\end{align}\n",
    "\n",
    "#### Marginal PDF\n",
    "\n",
    "![title](images/L1802.png)\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View [Lecture 18: MGFs Continued | Statistics 110](http://bit.ly/2oMTzTP) on YouTube."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}