In machine learning, a neural differential equation is a differential equation whose right-hand side is parametrized by the weights θ of a neural network. In particular, a neural ordinary differential equation (neural ODE) is an ordinary differential equation of the form
d h ( t ) d t = f θ ( h ( t ) , t ) . {\displaystyle {\frac {\mathrm {d} \mathbf {h} (t)}{\mathrm {d} t}}=f_{\theta }(\mathbf {h} (t),t).} In classical neural networks, layers are arranged in a sequence indexed by natural numbers. In neural ODEs, however, layers form a continuous family indexed by positive real numbers. Specifically, the function h : R ≥ 0 → R {\displaystyle h:\mathbb {R} _{\geq 0}\to \mathbb {R} } maps each positive index t to a real value, representing the state of the neural network at that layer.
Neural ODEs can be understood as continuous-time control systems, where their ability to interpolate data can be interpreted in terms of controllability.
Connection with residual neural networks
Neural ODEs can be interpreted as a residual neural network with a continuum of layers rather than a discrete number of layers.3 Applying the Euler method with a unit time step to a neural ODE yields the forward propagation equation of a residual neural network:
h ℓ + 1 = f θ ( h ℓ , ℓ ) + h ℓ , {\displaystyle \mathbf {h} _{\ell +1}=f_{\theta }(\mathbf {h} _{\ell },\ell )+\mathbf {h} _{\ell },}
with ℓ being the ℓ-th layer of this residual neural network. While the forward propagation of a residual neural network is done by applying a sequence of transformations starting at the input layer, the forward propagation computation of a neural ODE is done by solving a differential equation. More precisely, the output h out {\displaystyle \mathbf {h} _{\text{out}}} associated to the input h in {\displaystyle \mathbf {h} _{\text{in}}} of the neural ODE is obtained by solving the initial value problem
d h ( t ) d t = f θ ( h ( t ) , t ) , h ( 0 ) = h in , {\displaystyle {\frac {\mathrm {d} \mathbf {h} (t)}{\mathrm {d} t}}=f_{\theta }(\mathbf {h} (t),t),\quad \mathbf {h} (0)=\mathbf {h} _{\text{in}},}
and assigning the value h ( T ) {\displaystyle \mathbf {h} (T)} to h out {\displaystyle \mathbf {h} _{\text{out}}} .
Universal differential equations
In physics-informed contexts where additional information is known, neural ODEs can be combined with an existing first-principles model to build a physics-informed neural network model called universal differential equations (UDE).4567 For instance, an UDE version of the Lotka-Volterra model can be written as8
d x d t = α x − β x y + f θ ( x ( t ) , y ( t ) ) , d y d t = − γ y + δ x y + g θ ( x ( t ) , y ( t ) ) , {\displaystyle {\begin{aligned}{\frac {dx}{dt}}&=\alpha x-\beta xy+f_{\theta }(x(t),y(t)),\\{\frac {dy}{dt}}&=-\gamma y+\delta xy+g_{\theta }(x(t),y(t)),\end{aligned}}}
where the terms f θ {\displaystyle f_{\theta }} and g θ {\displaystyle g_{\theta }} are correction terms parametrized by neural networks.
See also
External links
References
Chen, Ricky T. Q.; Rubanova, Yulia; Bettencourt, Jesse; Duvenaud, David K. (2018). "Neural Ordinary Differential Equations" (PDF). In Bengio, S.; Wallach, H.; Larochelle, H.; Grauman, K.; Cesa-Bianchi, N.; Garnett, R. (eds.). Advances in Neural Information Processing Systems. Vol. 31. Curran Associates, Inc. arXiv:1806.07366. https://proceedings.neurips.cc/paper_files/paper/2018/file/69386f6bb1dfed68692a24c8686939b9-Paper.pdf ↩
Ruiz-Balet, Domènec; Zuazua, Enrique (2023). "Neural ODE Control for Classification, Approximation, and Transport". SIAM Review. 65 (3): 735–773. arXiv:2104.05278. doi:10.1137/21M1411433. ISSN 0036-1445. https://epubs.siam.org/doi/10.1137/21M1411433 ↩
Chen, Ricky T. Q.; Rubanova, Yulia; Bettencourt, Jesse; Duvenaud, David K. (2018). "Neural Ordinary Differential Equations" (PDF). In Bengio, S.; Wallach, H.; Larochelle, H.; Grauman, K.; Cesa-Bianchi, N.; Garnett, R. (eds.). Advances in Neural Information Processing Systems. Vol. 31. Curran Associates, Inc. arXiv:1806.07366. https://proceedings.neurips.cc/paper_files/paper/2018/file/69386f6bb1dfed68692a24c8686939b9-Paper.pdf ↩
Christopher Rackauckas; Yingbo Ma; Julius Martensen; Collin Warner; Kirill Zubov; Rohit Supekar; Dominic Skinner; Ali Ramadhan; Alan Edelman (2024). "Universal Differential Equations for Scientific Machine Learning". arXiv:2001.04385 [cs.LG]. /wiki/ArXiv_(identifier) ↩
Xiao, Tianbai; Frank, Martin (2023). "RelaxNet: A structure-preserving neural network to approximate the Boltzmann collision operator". Journal of Computational Physics. 490: 112317. arXiv:2211.08149. Bibcode:2023JCoPh.49012317X. doi:10.1016/j.jcp.2023.112317. https://linkinghub.elsevier.com/retrieve/pii/S0021999123004126 ↩
Silvestri, Mattia; Baldo, Federico; Misino, Eleonora; Lombardi, Michele (2023), Mikyška, Jiří; de Mulatier, Clélia; Paszynski, Maciej; Krzhizhanovskaya, Valeria V. (eds.), "An Analysis of Universal Differential Equations for Data-Driven Discovery of Ordinary Differential Equations", Computational Science – ICCS 2023, vol. 10476, Cham: Springer Nature Switzerland, pp. 353–366, doi:10.1007/978-3-031-36027-5_27, ISBN 978-3-031-36026-8, retrieved 2024-08-18 978-3-031-36026-8 ↩
Christoph Plate; Carl Julius Martensen; Sebastian Sager (2024). "Optimal Experimental Design for Universal Differential Equations". arXiv:2408.07143 [math.OC]. /wiki/ArXiv_(identifier) ↩
Patrick Kidger (2021). On Neural Differential Equations (PhD). Oxford, United Kingdom: University of Oxford, Mathematical Institute. https://ora.ox.ac.uk/objects/uuid:af32d844-df84-4fdc-824d-44bebc3d7aa9 ↩