Gaussian splatting

<h2 id="3d-gaussian-splatting">3D Gaussian splatting</h2>
"3DGS" redirects here. It could also refer to <a href="/facts/3D_GameStudio/W9PaKBSp">3D GameStudio</a>, a game development kit.

3D Gaussian splatting (3DGS) is a technique used in the field of <a href="/facts/Real-time_computer_graphics/jrILnb0K">real-time</a> radiance field rendering.<a class="footnote-ref" id="fnref:5" href="#fn:5">5</a> It enables the creation of high-quality real-time novel-view scenes by combining multiple photos or videos, addressing a significant challenge in the field.
The method represents scenes with 3D Gaussians that retain properties of continuous volumetric radiance fields, integrating sparse points produced during camera calibration. It introduces an <a href="/facts/Anisotropic/YMqTQWVB">Anisotropic</a> representation using 3D Gaussians to model radiance fields, along with an interleaved optimization and density control of the Gaussians. A fast visibility-aware rendering algorithm supporting anisotropic splatting is also proposed, catered to GPU usage.<a class="footnote-ref" id="fnref:6" href="#fn:6">6</a>

<h3>Method</h3>

The method involves several key steps:

<ul><li>Input: A set of images of a static scene along with camera positions, expressed as a sparse <a href="/facts/Point_cloud/dN8eG4AU">point cloud</a>.</li>
<li>3D Gaussians: Definition of mean, covariance matrix, and opacity for each Gaussian.</li>
<li>Color representation: Using <a href="/facts/Spherical_harmonics/bn98Gv2L">spherical harmonics</a> to model view-dependent appearance.</li>
<li>Optimization algorithm: Optimizing the parameters using <a href="/facts/Stochastic_gradient_descent/HbcaYqQP">stochastic gradient descent</a> to minimize a loss function combining L1 loss and D-SSIM, inspired by the Plenoxels work.<a class="footnote-ref" id="fnref:7" href="#fn:7">7</a></li>
<li>Rasterizer: Implementing a tile-based rasterizer for fast sorting and backward pass, enabling efficient blending of Gaussian components.</li></ul>
The method uses differentiable 3D Gaussian splatting, which is unstructured and explicit, allowing rapid rendering and projection to 2D splats. The covariance of the Gaussians can be thought of as configurations of an ellipsoid, which can be mathematically decomposed into a scaling matrix and a rotation matrix. The gradients for all parameters are derived explicitly to overcome any overhead due to <a href="/facts/Automatic_differentiation/dQcdhBAM">autodiff</a>.
The optimization creates a dense set of 3D Gaussians that represent the scene as accurately as possible. Each step of rendering is followed by a comparison to the training views available in the dataset.

<h3>Results and evaluation</h3>

The authors[who?] tested their algorithm on 13 real scenes from previously published datasets and the synthetic Blender dataset.<a class="footnote-ref" id="fnref:8" href="#fn:8">8</a> They compared their method against state-of-the-art techniques like Mip-NeRF360,<a class="footnote-ref" id="fnref:9" href="#fn:9">9</a> InstantNGP,<a class="footnote-ref" id="fnref:10" href="#fn:10">10</a> and Plenoxels.<a class="footnote-ref" id="fnref:11" href="#fn:11">11</a> Quantitative evaluation metrics used were PSNR, L-PIPS, and SSIM.
Their fully converged model (30,000 iterations) achieves quality on par with or slightly better than Mip-NeRF360,<a class="footnote-ref" id="fnref:12" href="#fn:12">12</a> but with significantly reduced training time (35–45 minutes vs. 48 hours) and faster rendering (real-time vs. 10 seconds per frame). At 7,000 iterations (5–10 minutes of training), their method achieves comparable quality to InstantNGP<a class="footnote-ref" id="fnref:13" href="#fn:13">13</a> and Plenoxels.<a class="footnote-ref" id="fnref:14" href="#fn:14">14</a>
For synthetic bounded scenes (Blender dataset<a class="footnote-ref" id="fnref:15" href="#fn:15">15</a>), they achieved state-of-the-art results even with random initialization, starting from 100,000 uniformly random Gaussians.

<h3>Limitations</h3>
Some limitations of the method include:

<ul><li>Elongated artifacts or "splotchy" Gaussians in some areas.</li>
<li>Occasional popping artifacts due to large Gaussians created by the optimization, especially in regions with view-dependent appearance.</li>
<li>Higher memory consumption compared to NeRF-based solutions, though still more compact than previous point-based approaches.</li>
<li>May require hyperparameter tuning (e.g., reducing position learning rate) for very large scenes.</li>
<li>Peak GPU memory consumption during training can be high (over 20 GB) in the current unoptimized prototype.</li></ul>
The authors[who?] note that some of these limitations could potentially be addressed through future improvements like better culling approaches, antialiasing, regularization, and compression techniques.

<h2 id="3d-temporal-gaussian-splatting">3D Temporal Gaussian splatting</h2>
Extending 3D Gaussian splatting to dynamic scenes, 3D Temporal Gaussian splatting incorporates a time component, allowing for real-time rendering of dynamic scenes with high resolutions.<a class="footnote-ref" id="fnref:16" href="#fn:16">16</a> It represents and renders dynamic scenes by modeling complex motions while maintaining efficiency. The method uses a HexPlane to connect adjacent Gaussians, providing an accurate representation of position and shape deformations. By utilizing only a single set of canonical 3D Gaussians and predictive analytics, it models how they move over different timestamps.<a class="footnote-ref" id="fnref:17" href="#fn:17">17</a>
It is sometimes referred to as "4D Gaussian splatting"; however, this naming convention implies the use of 4D Gaussian primitives (parameterized by a 4×4 mean and a 4×4 covariance matrix). Most work in this area still employs 3D Gaussian primitives, applying temporal constraints as an extra parameter of optimization.
Achievements of this technique include real-time rendering on dynamic scenes with high resolutions, while maintaining quality. It showcases potential applications for future developments in film and other media, although there are current limitations regarding the length of motion captured.<a class="footnote-ref" id="fnref:18" href="#fn:18">18</a>

<h2 id="applications">Applications</h2>
3D Gaussian splatting has been adapted and extended across various computer vision and graphics applications, from dynamic scene rendering to autonomous driving simulations and 4D content creation:

<ul><li>Text-to-3D using Gaussian Splatting: Applies 3D Gaussian splatting to text-to-3D generation.<a class="footnote-ref" id="fnref:19" href="#fn:19">19</a></li>
<li>End-to-end Autonomous Driving: Mentions 3D Gaussian splatting as a data-driven sensor simulation method for autonomous driving, highlighting its ability to generate realistic novel views of a scene.<a class="footnote-ref" id="fnref:20" href="#fn:20">20</a></li>
<li>SuGaR: Proposes a method to extract precise and fast meshes from 3D Gaussian splatting.<a class="footnote-ref" id="fnref:21" href="#fn:21">21</a></li>
<li>SplaTAM: Applies 3D Gaussian-based radiance fields to Simultaneous Localization and Mapping (SLAM), leveraging fast rendering and optimization capabilities to achieve state-of-the-art results.<a class="footnote-ref" id="fnref:22" href="#fn:22">22</a></li>
<li>Align Your Gaussians: Uses dynamic 3D Gaussians for 4D content creation from text.<a class="footnote-ref" id="fnref:23" href="#fn:23">23</a></li></ul>
<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Computer_graphics/lapBE8wv">Computer graphics</a></li>
<li><a href="/facts/Neural_radiance_field/ruc9oFuU">Neural radiance field</a></li>
<li><a href="/facts/Volume_rendering/cpfXEhYo">Volume rendering</a></li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1">Westover, Lee Alan (July 1991). "SPLATTING: A Parallel, Feed-Forward Volume Rendering Algorithm" (PDF). Retrieved October 18, 2023. <a href="https://articles.tomasparks.name/publications/Westover1991.pdf" target="_blank">https://articles.tomasparks.name/publications/Westover1991.pdf</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">Huang, Jian (Spring 2002). "Splatting" (PPT). Retrieved 5 August 2011. <a href="http://web.eecs.utk.edu/~huangj/CS594S02/splatting.ppt" target="_blank">http://web.eecs.utk.edu/~huangj/CS594S02/splatting.ppt</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
<li id="fn:3">Kerbl, Bernhard; Kopanas, Georgios; Leimkuehler, Thomas; Drettakis, George (2023-07-26). "3D Gaussian Splatting for Real-Time Radiance Field Rendering". ACM Transactions on Graphics. 42 (4): 139:1–139:14. arXiv:2308.04079. doi:10.1145/3592433. ISSN 0730-0301. <a href="https://dl.acm.org/doi/10.1145/3592433" target="_blank">https://dl.acm.org/doi/10.1145/3592433</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></li>
<li id="fn:4">Wu, Guanjun; Yi, Taoran; Fang, Jiemin; Xie, Lingxi; Zhang, Xiaopeng; Wei, Wei; Liu, Wenyu; Tian, Qi; Wang, Xinggang (June 2024). 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 20310–20320. arXiv:2310.08528. doi:10.1109/CVPR52733.2024.01920. <a href="https://ieeexplore.ieee.org/document/10656774/" target="_blank">https://ieeexplore.ieee.org/document/10656774/</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></li>
<li id="fn:5">Kerbl, Bernhard; Kopanas, Georgios; Leimkuehler, Thomas; Drettakis, George (2023-07-26). "3D Gaussian Splatting for Real-Time Radiance Field Rendering". ACM Transactions on Graphics. 42 (4): 139:1–139:14. arXiv:2308.04079. doi:10.1145/3592433. ISSN 0730-0301. <a href="https://dl.acm.org/doi/10.1145/3592433" target="_blank">https://dl.acm.org/doi/10.1145/3592433</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></li>
<li id="fn:6">Kerbl, Bernhard; Kopanas, Georgios; Leimkuehler, Thomas; Drettakis, George (2023-07-26). "3D Gaussian Splatting for Real-Time Radiance Field Rendering". ACM Transactions on Graphics. 42 (4): 139:1–139:14. arXiv:2308.04079. doi:10.1145/3592433. ISSN 0730-0301. <a href="https://dl.acm.org/doi/10.1145/3592433" target="_blank">https://dl.acm.org/doi/10.1145/3592433</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></li>
<li id="fn:7">Fridovich-Keil, Sara; Yu, Alex; Tancik, Matthew; Chen, Qinhong; Recht, Benjamin; Kanazawa, Angjoo (June 2022). "Plenoxels: Radiance Fields without Neural Networks". 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. pp. 5491–5500. arXiv:2112.05131. doi:10.1109/cvpr52688.2022.00542. ISBN 978-1-6654-6946-3. <a href="978-1-6654-6946-3" target="_blank">978-1-6654-6946-3</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></li>
<li id="fn:8">Mildenhall, Ben; Srinivasan, Pratul P.; Tancik, Matthew; Barron, Jonathan T.; Ramamoorthi, Ravi; Ng, Ren (2020), "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis", Lecture Notes in Computer Science, Cham: Springer International Publishing, pp. 405–421, doi:10.1007/978-3-030-58452-8_24, ISBN 978-3-030-58451-1, retrieved 2024-09-25 <a href="978-3-030-58451-1" target="_blank">978-3-030-58451-1</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></li>
<li id="fn:9">Barron, Jonathan T.; Mildenhall, Ben; Verbin, Dor; Srinivasan, Pratul P.; Hedman, Peter (June 2022). "Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields". 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. pp. 5460–5469. arXiv:2111.12077. doi:10.1109/cvpr52688.2022.00539. ISBN 978-1-6654-6946-3. <a href="978-1-6654-6946-3" target="_blank">978-1-6654-6946-3</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></li>
<li id="fn:10">Müller, Thomas; Evans, Alex; Schied, Christoph; Keller, Alexander (July 2022). "Instant neural graphics primitives with a multiresolution hash encoding". ACM Transactions on Graphics. 41 (4): 1–15. arXiv:2201.05989. doi:10.1145/3528223.3530127. ISSN 0730-0301. <a href="https://dx.doi.org/10.1145/3528223.3530127" target="_blank">https://dx.doi.org/10.1145/3528223.3530127</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></li>
<li id="fn:11">Fridovich-Keil, Sara; Yu, Alex; Tancik, Matthew; Chen, Qinhong; Recht, Benjamin; Kanazawa, Angjoo (June 2022). "Plenoxels: Radiance Fields without Neural Networks". 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. pp. 5491–5500. arXiv:2112.05131. doi:10.1109/cvpr52688.2022.00542. ISBN 978-1-6654-6946-3. <a href="978-1-6654-6946-3" target="_blank">978-1-6654-6946-3</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></li>
<li id="fn:12">Barron, Jonathan T.; Mildenhall, Ben; Verbin, Dor; Srinivasan, Pratul P.; Hedman, Peter (June 2022). "Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields". 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. pp. 5460–5469. arXiv:2111.12077. doi:10.1109/cvpr52688.2022.00539. ISBN 978-1-6654-6946-3. <a href="978-1-6654-6946-3" target="_blank">978-1-6654-6946-3</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></li>
<li id="fn:13">Müller, Thomas; Evans, Alex; Schied, Christoph; Keller, Alexander (July 2022). "Instant neural graphics primitives with a multiresolution hash encoding". ACM Transactions on Graphics. 41 (4): 1–15. arXiv:2201.05989. doi:10.1145/3528223.3530127. ISSN 0730-0301. <a href="https://dx.doi.org/10.1145/3528223.3530127" target="_blank">https://dx.doi.org/10.1145/3528223.3530127</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></li>
<li id="fn:14">Fridovich-Keil, Sara; Yu, Alex; Tancik, Matthew; Chen, Qinhong; Recht, Benjamin; Kanazawa, Angjoo (June 2022). "Plenoxels: Radiance Fields without Neural Networks". 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. pp. 5491–5500. arXiv:2112.05131. doi:10.1109/cvpr52688.2022.00542. ISBN 978-1-6654-6946-3. <a href="978-1-6654-6946-3" target="_blank">978-1-6654-6946-3</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></li>
<li id="fn:15">Mildenhall, Ben; Srinivasan, Pratul P.; Tancik, Matthew; Barron, Jonathan T.; Ramamoorthi, Ravi; Ng, Ren (2020), "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis", Lecture Notes in Computer Science, Cham: Springer International Publishing, pp. 405–421, doi:10.1007/978-3-030-58452-8_24, ISBN 978-3-030-58451-1, retrieved 2024-09-25 <a href="978-3-030-58451-1" target="_blank">978-3-030-58451-1</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></li>
<li id="fn:16">Wu, Guanjun; Yi, Taoran; Fang, Jiemin; Xie, Lingxi; Zhang, Xiaopeng; Wei, Wei; Liu, Wenyu; Tian, Qi; Wang, Xinggang (June 2024). 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 20310–20320. arXiv:2310.08528. doi:10.1109/CVPR52733.2024.01920. <a href="https://ieeexplore.ieee.org/document/10656774/" target="_blank">https://ieeexplore.ieee.org/document/10656774/</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></li>
<li id="fn:17">Franzen, Carl (16 October 2023). "Actors' worst fears come true? New 3D Temporal Gaussian Splatting method captures human motion". venturebeat.com. VentureBeat. Retrieved October 18, 2023. <a href="https://venturebeat.com/ai/actors-worst-fears-come-true-new-4d-gaussian-splatting-method-captures-human-motion" target="_blank">https://venturebeat.com/ai/actors-worst-fears-come-true-new-4d-gaussian-splatting-method-captures-human-motion</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></li>
<li id="fn:18">Franzen, Carl (16 October 2023). "Actors' worst fears come true? New 3D Temporal Gaussian Splatting method captures human motion". venturebeat.com. VentureBeat. Retrieved October 18, 2023. <a href="https://venturebeat.com/ai/actors-worst-fears-come-true-new-4d-gaussian-splatting-method-captures-human-motion" target="_blank">https://venturebeat.com/ai/actors-worst-fears-come-true-new-4d-gaussian-splatting-method-captures-human-motion</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></li>
<li id="fn:19">Chen, Zilong; Wang, Feng; Wang, Yikai; Liu, Huaping (2024-06-16). "Text-to-3D using Gaussian Splatting". 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vol. abs/2211.0 1324. IEEE. pp. 21401–21412. arXiv:2309.16585. doi:10.1109/cvpr52733.2024.02022. ISBN 979-8-3503-5300-6. <a href="979-8-3503-5300-6" target="_blank">979-8-3503-5300-6</a> <a href="#fnref:19" class="footnote-back-ref">↩</a></li>
<li id="fn:20">Chen, Li; Wu, Penghao; Chitta, Kashyap; Jaeger, Bernhard; Geiger, Andreas; Li, Hongyang (2024). "End-to-end Autonomous Driving: Challenges and Frontiers". IEEE Transactions on Pattern Analysis and Machine Intelligence. PP (12): 10164–10183. arXiv:2306.16927. doi:10.1109/tpami.2024.3435937. ISSN 0162-8828. PMID 39078757. <a href="https://dx.doi.org/10.1109/tpami.2024.3435937" target="_blank">https://dx.doi.org/10.1109/tpami.2024.3435937</a> <a href="#fnref:20" class="footnote-back-ref">↩</a></li>
<li id="fn:21">Guédon, Antoine; Lepetit, Vincent (2024-06-16). "SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering". 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. pp. 5354–5363. arXiv:2311.12775. doi:10.1109/cvpr52733.2024.00512. ISBN 979-8-3503-5300-6. <a href="979-8-3503-5300-6" target="_blank">979-8-3503-5300-6</a> <a href="#fnref:21" class="footnote-back-ref">↩</a></li>
<li id="fn:22">Keetha, Nikhil; Karhade, Jay; Jatavallabhula, Krishna Murthy; Yang, Gengshan; Scherer, Sebastian; Ramanan, Deva; Luiten, Jonathon (2024-06-16). "SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM". 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. pp. 21357–21366. doi:10.1109/cvpr52733.2024.02018. ISBN 979-8-3503-5300-6. <a href="979-8-3503-5300-6" target="_blank">979-8-3503-5300-6</a> <a href="#fnref:22" class="footnote-back-ref">↩</a></li>
<li id="fn:23">Ling, Huan; Kim, Seung Wook; Torralba, Antonio; Fidler, Sanja; Kreis, Karsten (2024-06-16). "Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models". 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. pp. 8576–8588. arXiv:2312.13763. doi:10.1109/cvpr52733.2024.00819. ISBN 979-8-3503-5300-6. <a href="979-8-3503-5300-6" target="_blank">979-8-3503-5300-6</a> <a href="#fnref:23" class="footnote-back-ref">↩</a></li>
</ol>

Gaussian splatting open-in-new

Gaussian splatting