What is it? Students will write a short, formal expository paper about some application of the ideas of linear algebra. This will involve independent reading, thinking, and learning. Students will...

MARKOV PROCESSES OR CHAINS



As an application of Linear Algebra.




The prerequisites for the Markov Processes are :

Linear Systems

Matrices

Intuitive understanding of Limits
Markov Chains
A Markov Process
Let us consider a physical or that undergoes a process of change such that at any moment, it can occupy
one of a finite number of states. Suppose that such a system changes with time from one state to
another and at scheduled times the state of the system is observed. Let us say, these changes are not
predictable, but they are governed by probability distribution. These changes incorporate a simple sort
of dependence structure, that is, the conditional distribution of future states of the system, given some
information about past states, depends only on the most recent piece of information. Which could be
simplified like, what matters in predicting the future of a system is its present state, and not the path by
which the system got to its present state.

If the state of the system at any observation cannot be predicted with certainty, but the probability that
a given state occurs can be predicted by just knowing the state of the system at the preceding
observation, then the process of change is called a Markov chain or Markov process.

Definition :

If a Markov Chain has k possible states, which we label as 1,2,...,k, then the probability that the
system is in state i at any observation after it was in state j at the preceding observation is denoted by
pij and is called transitional probability from state j to state i. The matrix P = [pij] is called transitional
matrix of Markov Chain.

In a three-state Markov Chain, the transition matrix is of the following form :


Preceding State

1 2 3


p11 p12 p13 1

p21 p22 p23 2 New State

p31 p32 p33 3



In this matrix, p31 is the probability that the system will change from state 1 to state 3 ; p22 is the
probability that the system will be state 2 if it was in state 2 ; and so on.




Let us look at two examples showing Transition Matrix of a Markov Chain

Example 1 :

A taxi service agency has three taxi stations, labeled by 1, 2 and 3. The taxi driver may pick up the car
from any of the one location and returns it to any of the three locations at the end of his working day.
The agency finds the taxi driver returns the car to the various stations according to the following
probabilities :

Picked from station

1 2 3

0.8 0.3 0.2 1

0.1 0.2 0.6 2 Returned to station

0.1 0.5 0.2 3



This matrix is the transition matrix of the system considered as Markov Chain.
We can infer from this matrix that, the probability that the driver picks the car from station 1 and
returns it to station 2 is 0.3. Similarly, the probability that he picks up the car from station 1 and returns
it to station 3 is 0.1.

Example 2 :

A survey was conducted by the blood donation society of a college. It...