Continuous-time Markov chain

A continuous-time Markov chain (CTMC) is a continuous <a href="/facts/Stochastic_process/Ng1n1dnB">stochastic process</a> in which, for each state, the process will change state according to an <a href="/facts/Exponential_random_variable/yY3EdV0u">exponential random variable</a> and then move to a different state as specified by the probabilities of a <a href="/facts/Stochastic_matrix/2cRWAqC3">stochastic matrix</a>. An equivalent formulation describes the process as changing state according to the least value of a set of exponential random variables, one for each possible state it can move to, with the parameters determined by the current state.
An example of a CTMC with three states 
 
 
 
 {
 0
 ,
 1
 ,
 2
 }
 
 
 {\displaystyle \{0,1,2\}}
 
 is as follows: the process makes a transition after the amount of time specified by the holding time—an exponential random variable 
 
 
 
 
 E
 
 i
 
 
 
 
 {\displaystyle E_{i}}
 
, where i is its current state. Each random variable is independent and such that 
 
 
 
 
 E
 
 0
 
 
 ∼
 
 Exp
 
 (
 6
 )
 
 
 {\displaystyle E_{0}\sim {\text{Exp}}(6)}
 
, 
 
 
 
 
 E
 
 1
 
 
 ∼
 
 Exp
 
 (
 12
 )
 
 
 {\displaystyle E_{1}\sim {\text{Exp}}(12)}
 
 and 
 
 
 
 
 E
 
 2
 
 
 ∼
 
 Exp
 
 (
 18
 )
 
 
 {\displaystyle E_{2}\sim {\text{Exp}}(18)}
 
. When a transition is to be made, the process moves according to the jump chain, a <a href="/facts/Discrete-time_Markov_chain/u4qADb4z">discrete-time Markov chain</a> with stochastic matrix:

[
            
              
                
                  0
                
                
                  
                    
                      1
                      2
                    
                  
                
                
                  
                    
                      1
                      2
                    
                  
                
              
              
                
                  
                    
                      1
                      3
                    
                  
                
                
                  0
                
                
                  
                    
                      2
                      3
                    
                  
                
              
              
                
                  
                    
                      5
                      6
                    
                  
                
                
                  
                    
                      1
                      6
                    
                  
                
                
                  0
                
              
            
            ]
          
        
        .
      
    
    {\displaystyle {\begin{bmatrix}0&{\frac {1}{2}}&{\frac {1}{2}}\\{\frac {1}{3}}&0&{\frac {2}{3}}\\{\frac {5}{6}}&{\frac {1}{6}}&0\end{bmatrix}}.}

Equivalently, by the property of <a href="/facts/Competing_exponentials/yY3EdV0u">competing exponentials</a>, this CTMC changes state from state i according to the minimum of two random variables, which are independent and such that 
 
 
 
 
 E
 
 i
 ,
 j
 
 
 ∼
 
 Exp
 
 (
 
 q
 
 i
 ,
 j
 
 
 )
 
 
 {\displaystyle E_{i,j}\sim {\text{Exp}}(q_{i,j})}
 
 for 
 
 
 
 i
 ≠
 j
 
 
 {\displaystyle i\neq j}
 
 where the parameters are given by the Q-matrix 
 
 
 
 Q
 =
 (
 
 q
 
 i
 ,
 j
 
 
 )
 
 
 {\displaystyle Q=(q_{i,j})}

[
            
              
                
                  −
                  6
                
                
                  3
                
                
                  3
                
              
              
                
                  4
                
                
                  −
                  12
                
                
                  8
                
              
              
                
                  15
                
                
                  3
                
                
                  −
                  18
                
              
            
            ]
          
        
        .
      
    
    {\displaystyle {\begin{bmatrix}-6&3&3\\4&-12&8\\15&3&-18\end{bmatrix}}.}

Each non-diagonal entry 
 
 
 
 
 q
 
 i
 ,
 j
 
 
 
 
 {\displaystyle q_{i,j}}
 
 can be computed as the probability that the jump chain moves from state i to state j, divided by the expected holding time of state i. The diagonal entries are chosen so that each row sums to 0.
A CTMC satisfies the <a href="/facts/Markov_property/g1Fs9GFS">Markov property</a>, that its behavior depends only on its current state and not on its past behavior, due to the memorylessness of the exponential distribution and of discrete-time Markov chains.

Continuous-time Markov chain open-in-new

Continuous-time Markov chain