Lecture_31.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 31: Markov chains, Transition Matrix, Stationary Distribution\n",
    "\n",
    "\n",
    "## Stat 110, Prof. Joe Blitzstein, Harvard University\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What are Markov chains?\n",
    "\n",
    "Markov Chains are an example of a stochastic process, or sequences of random variables evolving over the dimension of time or space. They were invented or discovered by Markov as way to answer the question _Does free will exist?_ However, their application is very wide.\n",
    "\n",
    "Say we have a sequence of random variables $X_0, X_1, X_2, \\cdots$. In our studies up until now, we were assuming that the distributions of these random variables were _independent and individually distributed_. If the index represents time, then this is like starting with a fresh, new independent random variable at each step. \n",
    "\n",
    "A more interesting case is when the random variables are in some way related. But in assuming such a case, the relations can get very complex.\n",
    "\n",
    "Markov Chains are a compromise: they are one level of complexity beyond _i.i.d._, and they come with some very useful properties. For example, think of $X_n$ as the state of a system at a particular and discrete time, like that for a wandering partical jumping from state to state.\n",
    "\n",
    "The indexes or states can be with regards to:\n",
    "\n",
    "* discrete time\n",
    "* continuous time\n",
    "* discrete space\n",
    "* continuous space\n",
    "\n",
    "For this simple introduction, we will limit ourselves to the case of _discrete time, with a finite number of states_ (discrete space), where $n \\in \\mathbb{Z}_{\\geq 0}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is the Markov Property? \n",
    "\n",
    "Keeping to our limitations of discrete time and space, assume that $n$ means \"now\" (whatever that might be). We assume $n+1$ to be the \"future\". Then consider that the \"future\" might be characterized by $P(X_{n+1} = j)$, where\n",
    "\n",
    "\\begin{align}\n",
    "  P(X_{n+1} = j | X_{n} = i, X_{n} = i_{n-1}, \\cdots , X_{0} = i_{0})\n",
    "\\end{align}\n",
    "\n",
    "This means that the future depends upon all of the former states in existence. Very complex.\n",
    "\n",
    "But what if we explicitly assumed a simpler model? What if we said that the future is conditionally independent of the past, _given the present_? Then we could say\n",
    "\n",
    "\\begin{align}\n",
    "  P(X_{n+1} = j | X_{n} = i, X_{n} = i_{n-1}, \\cdots , X_{0} = i_{0}) &= P(X_{n+1} = j | X_{n} = i)\n",
    "\\end{align}\n",
    "\n",
    "That is, if this property holds, if we know the current value $X_n$, everything else in the past is irrelevant. This is the **Markov assumption**; this is the **Markov Property**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Definition: transition probability\n",
    "\n",
    "The transition probability in a Markov Chain is given by $P(X_{n+1} = j | X_n = i) = q_{ij}$. If the probability in transitioning from states does not depend on time, then we say the Markov Chain is _homogeneous_.\n",
    "\n",
    "For this simple introduction, again, we will assume the Markov Chains to be homogeneous.\n",
    "\n",
    "Here is a graphical example of a simple 4-state Markov Chain, listing all of its transition probabilities:\n",
    "\n",
    "![title](images/L3101.png)\n",
    "\n",
    "\n",
    "### Definition: transition matrix\n",
    "\n",
    "The _transition matrix_ is then just the matrix representation of a Markov chain's transition probabilities in the form $Q = \\left[ q_{ij} \\right]$in the columns, with the rows representing each possible state. Since our introductory example is for a  discrete, finite space, elements of a row are the complete listing of the probability mass function for the state in question.\n",
    "\n",
    "For the example above, that'd be\n",
    "\n",
    "\\begin{align}\n",
    "  Q &= \\begin{pmatrix}\n",
    "    \\frac{1}{3} & \\frac{2}{3} & 0 & 0 \\\\\n",
    "    \\frac{1}{2} & 0 & \\frac{1}{2} & 0 \\\\\n",
    "    0 & 0 & 0 & 1 \\\\\n",
    "    \\frac{1}{2} & 0 & \\frac{1}{4} & \\frac{1}{4} \\\\    \n",
    "    \\end{pmatrix}\n",
    "\\end{align}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Usage\n",
    "\n",
    "We can use a Markov chain as a model to describe a system evolving over time. The case of $P(X_{n+1} = j | X_n = i) = q_{ij}$ only looks at the present $X_n$, ignoring events of the past. Using such a *first order* model might be too strong a starting assumption of conditional independence. We could, however, extend this model such that $P(X_{n+1} = j | X_n = i, X_{n-1} = k)$, to consider the last 2 states (second order), or even more past states.\n",
    "\n",
    "There are also such things called Markov Chain Monte Carlo, where you synthesize your own Markov Chain to let you do large-scale simulations (computer-assisted, of course). \n",
    "\n",
    "But originally, Markov came up with this idea to show there was a way to reconcile the Law of Large Numbers but relax the condition of _independent, individually distributed_, and instead of completely ignoring the past states of system, condition on the present state (first-order) to predict the future. Of course, this could be extended to condition on states prior to the present as well. He wanted to go a step beyond _i.i.d._. His  experiment was with the Russian langage and its vowels and consonants, a chain of two states."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the Transition Matrix\n",
    "\n",
    "Suppose time $n, X_n$ has distribution $\\vec{s}$ ($1 \\times m$ row vector). Again, in our discrete, finite state example, $\\vec{s}$ is simply the PMF listed out, so its entries are non-negative and sum up to 1.0.\n",
    "\n",
    "_What is the PMF for $X_{n+1}$?_\n",
    "\n",
    "\\begin{align}\n",
    "  P(X_{n+1} = j) &=  \\sum_{i} P(X_{n+1} = j | X_n = i) \\, P(X_n = i) &\\quad \\text{ by Law of Total Probability}\\\\\n",
    "  &= \\sum_{i}^{j} q_{ij} \\, s_i &\\quad \\text{ definition of transition matrix; and definition of transition matrix row} \\\\\n",
    "  &= \\sum_{i}^{j} s_i \\, q_{ij} &\\quad \\text{ but this is the } j^{th} \\text{ entry of } \\vec{s} \\, Q\n",
    "\\end{align}\n",
    "\n",
    "\n",
    "* So row vector $\\vec{s} \\, Q$ is the distribution at time $n+1$.\n",
    "* So row vector $\\vec{s} \\, Q^{2}$ is the distribution at time $n+2$ ($\\vec{s}\\,Q$ plays the role of $\\vec{s}$).\n",
    "* So row vector $\\vec{s} \\, Q^{3}$ is the distribution at time $n+3$ ($\\vec{s}\\,Q^2$ plays the role of $\\vec{s}$).\n",
    "* and so on, and so forth...\n",
    "\n",
    "Similarly, we can think about $m$ steps into the future.\n",
    "\n",
    "\\begin{align}\n",
    "  P(X_{n+1} = j | X_n = i) &= q_{ij} &\\text{ by definition; 1 step into the future } \\\\\n",
    "  \\\\\\\\ \n",
    "  P(X_{n+2} = j | X_n = i) &= \\sum_k P(X_{n+2} = j | X_{n+1} = k, X_n = i) \\, P(X_{n+1} = k | X_n = i) &\\text{ 2 steps into the future } \\\\\n",
    "  &= \\sum_k P(X_{n+2} = j | X_{n+1} = k \\require{enclose} \\, \\enclose{horizontalstrike}{, X_n = i}) \\, P(X_{n+1} = k | X_n = i) &\\text{ only the present counts } \\\\\n",
    "  &= \\sum_k q_{kj} \\, q_{ik} \\\\\n",
    "  &= \\sum_k q_{ik} \\, q_{kj} &\\text{ but this is the } (i,j) \\text{ entry of } Q^2 \\\\\n",
    "  \\\\\\\\\n",
    "  P(X_{n+m} = j | X_n = i) &= \\left[ q_{i,j} \\in Q^m  \\right]\n",
    "\\end{align}\n",
    "\n",
    "\n",
    "So you see that powers of the transition matrix give us a lot of information!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stationary Distribution\n",
    "\n",
    "#### Definition\n",
    "\n",
    "$\\vec{s}$ is stationary for the Markov chain we are considering if\n",
    "\n",
    "\\begin{align}\n",
    "  \\vec{s} \\, Q &= \\vec{s}\n",
    "\\end{align}\n",
    "\n",
    "You should immediately recognize that $\\vec{s}$ is an eigenvector of transition matrix $Q$. This means that if the chain starts off in $\\vec{s}$, after one step into the future it remains at $\\vec{s}$. Another step into the future, and we _still_ remain at $\\vec{s}$, ad nauseum. If the chain starts off with the stationary distribution, it will have that distribution forever.\n",
    "\n",
    "But that raises some questions:\n",
    "\n",
    "1. Does a stationary distribution exist? (all elements of $\\vec{s} \\in \\mathbb{Z}_{\\ge 0}$)\n",
    "1. Is it unique?\n",
    "1. Does the chain converge to $\\vec{s}$?\n",
    "1. How can we compute it?\n",
    "\n",
    "It turns out, the answers to questions 1 through 3 is _yes_.\n",
    "\n",
    "Can we compute the stationary distribution _efficiently_? In certain examples of Markov chains, we can do so very easily and without using matrices! \n",
    "\n",
    "But that is the topic of the next lecture.\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View [Lecture 31: Markov Chains | Statistics 110](http://bit.ly/2MXMgr3) on YouTube."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}