MWENGE CATHOLIC UNIVERSITY

(MWECAU)

FACULTY OF SCIENCE

DEPARTMENT OF NATURAL SCIENCES AND INFORMATION

TECHNOLOGY.

PRROGRAM: BSc. MATHEMATICS AND STATISTICS

COURSE NAME: STOCHASTIC PROCESS

COURSE CODE: STA 304

COURSE INSTRUCTOR: MADAM SABIANA ZUBERI

NATURE OF TASKS: GROUP WORK (1)

SN NAME REG NO

01 ALLY MAKALA T/DEG/2022/0068

02 GEOFREY .M. ORDAS T/DEG/2022/0853

03 BENEDICTO .P. MPIGACHAI T/DEG/2022/0831

04 RAMADHANI .M. HASSANI T/DEG/2022/0922

 MARKOV CHAIN
INTRODUCTION

Markov chains are mathematical models used to predict the probability of moving from one state
to another in a system, assuming that the next state depends only on the current state (known as
the "memoryless" property or the Markov property).

Another meaning of Markov chain.

Markov chains are mathematical models that describe a system's transitions between states, relying
only on the current state for future predictions (Verma, 2023).

Historically;

The concept of the Markov chain was developed by Russian mathematician Andrey Andreyevich
Markov (1856–1922). Markov introduced this idea in 1906 when he published a paper on a
specific type of stochastic process that exhibited "memoryless" behavior, which later became
known as the Markov property.

A picture of markov

Short description about Andrey Markov

 Born: June 14, 1856, Ryazan, Russia

 Died: July 20, 1922 (age 66 years), Saint Petersburg, Russia
 Books: The Correspondence between A.A. Markov and A.A. Chuprov on the Theory of
Probability and Mathematical Statistics
 Known for: Markov chains; Markov processes; stochastic processes
 Siblings: Vladimir Andreevich Markov
 Doctoral advisor: Pafnuty Chebyshev
There are different types of Markov chains, including discrete-time and continuous-time.
Discrete-time models apply in scenarios like decision-making, while continuous models are used
in processes that change over continuous time, like queue management (Simplilearn, 2022).

This concept is widely applicable across disciplines such as finance, computer science, biology,
and linguistics due to its simplicity in modeling probabilistic sequences where each step is based
solely on the previous one.

MATHEMATICAL MODELS OF MARKOV CHAIN

Markov chains form a crucial part of stochastic processes, where outcomes are partly random and
dependent on a previous state.

The following are some mathematical models for Markov chains within stochastic processes,

a. Continuous-Time Markov Chains (CTMC):

A Continuous-Time Markov Chain (CTMC) is a type of stochastic process where changes, or

transitions, between states can occur at any continuous point in time, rather than at fixed, discrete
intervals. In CTMCs, the future state depends only on the current state (the Markov property) and
is independent of the sequence of events that led to the current state.

𝐝𝛑(𝐭)
Mathematically; = 𝛑(𝐭)𝐐
𝐝𝐭

Equation Meaning: This differential equation describes how the probability distribution 𝛑(𝐭)
over the states of a continuous-time Markov chain changes over time.

Interpretation of Terms:
𝛑(𝐭) : This is the state probability vector at time t, where each element πi (𝐭) represents the
probability of being in state i at time t.
Q: Known as the transition rate matrix, Q describes the instantaneous rates of moving from one
state to another. Each element qij represents the rate of transitioning from state i to j (when i is not
equal to j) and -qii represents the rate of leaving state i.
Practical Use: Solving this differential equation helps determine the state probabilities at any
future time, crucial in fields like queuing theory or biological modeling where the time between
events varies continuously.

b. Discrete time Markov chain (DTMC)

A discrete time Markov chain is a sequence of random variables X0, X1, X2… where the future
state Xt+1 depend only on the present state Xt, not on the past states Xt-1, Xt-2… The transitions
probabilities between states are represented by transition matrix P, where each entry P ij gives the
probabilities of transitioning from sate i to the state j.

Mathematically; P (Xt+1 = j |Xt=i) =Pij,

That means, P (Xt+1| Xt, Xt-1… X0) = P (Xt+1|Xt)

c. Hidden Markov Models (HMMs):

A Hidden Markov Model (HMM) is a statistical model that represents a system where an
underlying process modeled as a Markov process is not directly observable (hidden) but can be
inferred through a series of observed events.

HMMs are particularly useful for modeling stochastic processes where we assume that the
observed data sequence is influenced by a sequence of hidden (latent) states.

Example; stock price prediction, HMMs are used in financial modeling to analyze stock market
trends and predict price movement

Explanation; the hidden state may represent market conditions like “bull”, “bear” or “stagnant”
markets. These condition are not directly observable, but they affect the observable stock prices
and volume data. The HMMs can infer the most probable sequence of hidden states based on the
stock price movements over time, helping analysts understand the underlying market trends.

Mathematically; P (xt|xt-1) =πP

Equation Meaning: This equation shows the probability of observing a sequence in an HMM.
Specifically, it calculates the probability of the current observed state xt given the previous
observed state xt-1.
Interpretation of Terms:
xt: Represents the observed output at time t , which is influenced by a hidden or unobservable
state.

π: The initial distribution, providing the probability of starting in each hidden state.

P: The transition matrix between hidden states. This matrix contains probabilities Pij, representing
the likelihood of transitioning from hidden state i to hidden state j.

Practical Use: In HMMs, this equation is used to decode sequences where the true states are
hidden, such as in language processing or genomics.

d. Markov Decision Processes (MDP): Bellman Equation

A Markov Decision Process (MDP) is a mathematical framework used for modeling decision-
making in scenarios where outcomes are partly random and partly under the control of a decision-
maker. MDPs are essential for fields like reinforcement learning and operations research, as they
provide a structured approach to optimizing decisions over time in uncertain environments.
𝒏
Mathematically, V(s) = MaxaϵA(R(s,a) + ϒ∑𝒔′ 𝐏(𝐬′|s,a) V(S′))

Equation Meaning: The Bellman Equation calculates the optimal value function V(s), which
represents the maximum expected reward an agent can achieve from state “s” by choosing the best
actions over time.

Interpretation of Terms:

V(s): The value of being in state “s”, considering future rewards.

MaxaϵA: This maximization indicates that the agent will choose the action a from the action set A
that yields the highest value.

R(s,a): The immediate reward obtained from taking action in state .

ϒ: The discount factor (0 ≤ ϒ < 1), determining how future rewards are weighted compared to
immediate rewards.

P(S′|s, a): The probability of transitioning to a future state from state given action.
Practical Use: This equation is fundamental in reinforcement learning, where it helps an agent
decide optimal actions for long-term rewards, useful in areas like robotics and financial modeling.

EXAMPLE OF MARKOV CHAIN

Consider a simple coin tossing experiment repeated for a number of times (costively), two possible
outcomes for each trial are ‘Head’ and ‘Tail’. Assume that Head occurs with probability p and that
Tail occurs with probability q, so that p + q = 1.

Let us denote the outcomes of the toss of the unbiased coin by Xn.

Then Xn = {1 if head occurs and 0 if tail occurs for n= 1, 2, 3…}

That is, Pr {Xn =1} =p, and Pr h {Xn=0} =q. Hence the sequence of random variables, X1, X2…
Can be written as (Xn: n≥1), which is a Markov chain.

MARKOV PROPERTY.

The Markov property is the principle that, in a stochastic process, the future state depends only on
the current state, not on any prior states. This "Memoryless" characteristic means that knowing the
present is sufficient to predict future behavior without needing past information.

In mathematical terms, if Xt represents the state of a process at time t, the Markov property is
defined such that, for any times s<t, the conditional probability distribution of Xt given Xs is
independent of any states prior to S.

This means, P (Xt | Xs, Xs-1….X0) =P (Xt | Xs), highlighting that the future is conditionally
independent of the past once the present is known.

This property is particularly useful in simplifying analysis and computations in various fields,
including finance, genetics, and queuing theory, where it allows for efficient modeling of systems
that evolve over time without needing to track entire histories.
CHARACTERISTICS OF MARKOV PROPERTY

 Memorylessness:

The Markov property is fundamentally defined by memorylessness, meaning that the future state
depends only on the current state and not on any past states. This feature allows the probability of
future events to be determined solely by the present situation, making it a useful model for systems
where the history is irrelevant to the prediction of future states (LibreTexts, 2022).

 State Transition Independence:

In a Markov process, the probability of transitioning from one state to another remains consistent
over time. This time-homogeneous behavior is essential for modeling stable systems where the
likelihood of changes between states does not fluctuate over time, such as certain financial models
(Bass, 2022).

 Predictive Simplicity:

Since only the current state matters for future predictions, Markov processes simplify
computations by eliminating the need to account for historical data. This makes them particularly
efficient for applications in data science and machine learning, where they’re used in predictive
text models and other applications requiring minimal computational overhead (BuiltIn, 2022).

 Discrete or Continuous Time:

Markov processes can be either in discrete or continuous time. Discrete-time Markov chains
progress in fixed time steps, while continuous time Markov processes transition at variable
intervals. This versatility enables them to model a wide range of systems, from queuing systems
in logistics to population dynamics in ecology (Bass, 2022).

 Applicability to Diverse Fields:

The Markov property supports various applications beyond mathematics, including engineering,
economics, biology, and artificial intelligence. By adapting state and transition dynamics, these
processes are versatile tools across disciplines (Bass, 2022).
EXAMPLE OF MARKOV PROPERTY
Weather prediction Model

Suppose we have a simple weather model with two possible states:

i. Sunny (S)
ii. Rainy (R)

We know the following probabilities based on historical data:

 If it’s Sunny today, there’s a 70% chance it will be Sunny tomorrow and a 30% chance it
will be Rainy.
 If it’s Rainy today, there’s a 40% chance it will be Sunny tomorrow and a 60% chance it
will be Rainy.

Let’s represent these probabilities in a transition matrix P

1. Transition matrix
The transition matrix P is given by:
0.7 0.3
P=( )
0.4 0.6
 P11 = 0.7: Probability of staying sunny if it’s sunny today.
 P12 = 0.3: Probability of transitioning from Sunny to Rainy.
 P21 = 0.4: Probability of transitioning from Rainy to Sunny.
 P22 = 0.6: Probability of staying Rainy if it’s Rainy today.
2. Initial state Vector
Let’s we start with a 100% chance of it being Sunny:
1
π(0) = ( )
0
3. Finding Future States
To find the probability distribution over states after n days, we calculate π(n) = π(0) × Pn.
Calculation for day 1
Using π(0) and P:
0.7 0.3
π(1) = π(0) × P = (1 0) ( ) = (0.7 0.3)
0.4 0.6
Calculation for Day 2

0.7 0.3
π(2) = π(1) × P = (0.7 0.3) ( ) = (0.61 0.39)
0.4 0.6
4. Finding the stationary Distribution
To find the stationary distribution π, we solve for πP = π with the condition that the sum of
the probabilities equal to 1.

Let π =(π𝑆 π𝑅 ). Then:

(π𝑆 π𝑅 ) (0.7 0.3)= (π𝑆 π𝑅 )

0.4 0.6
0.7πS + 0.4πR = πS
0.3 πS +0.6 πR = πR
Solving these equations with πR + πS = 1:
0.4πR = 0.3πS
3
πR = 4 πS
3
πR + 4 πS=1
7
πS =1
4
4
πS = 7
3
πR = 7

π=( 37)
7

5. Interpretation
 Short-Term Prediction: The probabilities calculate for Days 1 and 2 give short-term
forecasts.
 Long-Term Behavior: The stationary distribution π represents the long-term
behavior, indicating that after many transitions, the system will stabilize at around
57% Sunny and 43% Rainy, regardless of the initial conditions.

This is a simple demonstration of how the Markov property allows us to predict future states and
identify long-term behavior based solely on the present state and transition probabilities.
ASSUMPTION OF MARKOV PROPERTY

a) Finite State Space

Explanation: In a Markov chain, there is a limited, well-defined set of possible states that the
process can be in. This makes modeling simpler, as each state can be explicitly tracked in a
transition matrix.

Example: Consider customer satisfaction ratings for a product, where the states are defined as
"Very Satisfied," "Satisfied," "Neutral," "Dissatisfied," and "Very Dissatisfied." Here, customer
responses are finite and categorized, allowing the company to create a transition matrix to predict
how satisfaction might shift over time.

b) Time-Homogeneous Transition Probabilities

Explanation: This assumption means that the probability of moving from one state to another
remains the same over time. The transition matrix, therefore, does not change as the system
evolves.

Example: In web browsing behavior, the probability that a user moves from one page to another
within a website might remain constant over time. For example, if 30% of users move from the
homepage to the product page and 70% exit, these probabilities are assumed to hold regardless of
when the user visits. This allows the website to predict traffic flow without adjusting for specific
times.

c) Mutually Exclusive and Collectively Exhaustive States

Explanation: Each state in a Markov process is mutually exclusive (the process can only be in
one state at a time) and collectively exhaustive (covering all possible states).

Example: For an ATM, possible states might be "Idle," "Processing Transaction," or "Out of
Service." Only one state is active at any time, and the defined states account for all potential
conditions of the machine, ensuring complete representation. This structure enables efficient
prediction of ATM downtime or transaction handling rates.
 d. Memoryless Property (Markov Property)

Explanation: This property specifies that the future state of the process depends only on the
current state, not on the sequence of events that preceded it.

Example: In a board game, a player’s next move depends only on their current position, not on
how they reached that position. If a player is on "Start," their next roll will determine the following
state without regard for past rolls, making the process memoryless. This simplifies analysis of
probable future states based on the current position alone.

e. Stationary Distribution

Explanation: In many Markov chains, it’s assumed that there is a stable, long-term distribution
of state probabilities, meaning the likelihood of being in each state stabilizes over time.

Example: In a simplified model of traffic at intersections, the traffic light cycles between "Green,"
"Yellow," and "Red." Over a long enough period, the system will reach a stationary distribution
where each light state appears with predictable probabilities (e.g., 50% Green, 10% Yellow, and
40% Red), assuming each light state has a consistent time duration. This stationary distribution
helps in predicting average wait times for vehicles.

LIMITATION OF MARKOV PROPERTY

The Markov property has several key limitations:

a) Memorylessness: It assumes future states depend only on the current state, ignoring all
past states, which limits its accuracy in processes where history matters (e.g., financial
markets).
b) Static Transition Probabilities: Markov models assume that transition probabilities
remain constant over time, which doesn't align with real-world situations where
probabilities often vary due to external factors or seasonality.
c) Finite State Space: Markov processes are typically limited to a finite or countable set of
states, making them unsuitable for continuous or complex systems like temperature or
stock prices.
d) Long-Term Stability: The property assumes the system will reach a stable, stationary
distribution, which is unrealistic for dynamic systems that evolve continuously.
 e) Ignores External Influences: Markov processes don’t account for external shocks or
influences, which can significantly affect the transitions in real-world scenarios like policy
changes in economics or healthcare models.

SUMMARIZATION

Markov chains provide a framework for modeling systems with probabilistic state transitions and
are built on the Markov property, emphasizing the present state's exclusive influence on future
transitions. Markov chains come in various forms, including discrete and continuous time chains,
and can be analyzed for their long-term behavior through stationary distributions.

CONCLUSION

Markov chains, grounded in the Markov property, offer an intuitive yet powerful way to model
systems where future states are probabilistically determined by the present. The ability to predict
long-term behavior through stationary distributions makes Markov chains especially valuable
across domains, from finance to biology to artificial intelligence. Their application in modeling
real-world processes with probabilistic transitions makes them indispensable for both theoretical
analysis and practical applications in forecasting, optimization, and decision-making.
REFERENCE

 Bhat, B. R. (2022). Stochastic Processes and Their Applications. World Scientific.

 Polanco, C. (2023). Markov Chain Process: Theory and Cases. Bentham Science
Publishers.
 Van Gaans, O. (2023). An Introduction to Continuous-Time Markov Chains on Countable
State Spaces. Springer.
 Bartlett, M.S., Stochastic processes, 3rd edition, CUP, Cambridge (1978)