The Chetaev instability theorem for dynamical systems states that if there exists, for the system x ˙ = X ( x ) {\displaystyle {\dot {\textbf {x}}}=X({\textbf {x}})} with an equilibrium point at the origin, a continuously differentiable function V(x) such that

1. the origin is a boundary point of the set G = { x ∣ V ( x ) > 0 } {\displaystyle G=\{\mathbf {x} \mid V(\mathbf {x} )>0\}} ;
2. there exists a neighborhood U {\displaystyle U} of the origin such that V ˙ ( x ) > 0 {\displaystyle {\dot {V}}({\textbf {x}})>0} for all x ∈ G ∩ U {\displaystyle \mathbf {x} \in G\cap U}

then the origin is an unstable equilibrium point of the system.

This theorem is somewhat less restrictive than the Lyapunov instability theorems, since a complete sphere (circle) around the origin for which V {\displaystyle V} and V ˙ {\displaystyle {\dot {V}}} both are of the same sign does not have to be produced.

It is named after Nicolai Gurevich Chetaev.