Plan of Action: Markov Chains

Bhanoth Preetam
Roll no: 22B1254
June-July 2024

1 Objectives
• Review the prerequisites to Markov chains including stochastic processes, probabil-
ity distributions, linear algebra etc.

• Familiarize myself with the properties of Markov chains and its types.

• Understand its applications in modelling stochastic processes in fields of medicine,

behavioral economics, reinforcement learning, and specifically monte carlo simula-
tions.

• Implementing a Monte Carlo Simulation to gain a deeper practical understanding

of Markov Chains.

2 Timeline
Week 1: Review of stochastic, probability distributions, linear algebra, Number Theory,
and Real analysis.

Week 2: Discrete-time Markov Chains, Transition Matrices, Properties of Markov Chains

- Irreducibility, Ergodicity etc.

Week 3: Convergence to Steady State, Lyapunov Functions, Renewal Theory.

Week 4: Markov Chains in Graphs - Random Walks, Graph Analysis, Non-Homogenous

Markov Chains.

June 20: Mid Term Report.

Week 5: Continuous-Time Markov Chains, Poisson Processes.

Week 6: Applications in Queueing Theory, Reinforcement Learning and Behavioral Eco-

nomics.

Week 7: Learning and Modelling the Monte Carlo Simulation.

Week 8: Data analysis on the model and making the final report.

Mid-July: Final Report Submission.

1
3 References to be used:
• Markov Chains, Gibbs Fields, Monte Carlo Simulations and Queues - Pierre Bre-
maud.

• Stochastic Processes - Sheldon Ross.

• Probabilistic Aspects of Computer Science: Markovian Models - S. Haddad.

• Introduction to Stochastic Models Lecture Series - Dr. Hao Wu.

• Mark Chains Lecture Series - Richard Weber.