• Images
• Videos
• Documents
• Books
• Articles

Library

The library is designed to reduce computing power and memory use and to train large distributed models with better parallelism on existing computer hardware.²³ DeepSpeed is optimized for low latency, high throughput training. It includes the Zero Redundancy Optimizer (ZeRO) for training models with 1 trillion or more parameters.⁴ Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under MIT License and available on GitHub.⁵

The team claimed to achieve up to a 6.2x throughput improvement, 2.8x faster convergence, and 4.6x less communication.⁶

See also

• Free and open-source software portal

• Comparison of deep learning software
• Deep learning
• Machine learning
• TensorFlow

Further reading

• Rajbhandari, Samyam; Rasley, Jeff; Ruwase, Olatunji; He, Yuxiong (2019). "ZeRO: Memory Optimization Towards Training A Trillion Parameter Models". arXiv:1910.02054 [cs.LG].

External links

• AI at Scale - Microsoft Research
• GitHub - microsoft/DeepSpeed
• ZeRO & DeepSpeed: New system optimizations enable training models with over 100 billion parameters - Microsoft Research

References

1. "Microsoft Updates Windows, Azure Tools with an Eye on The Future". PCMag UK. May 22, 2020. https://uk.pcmag.com/news-analysis/127085/microsoft-updates-windows-azure-tools-with-an-eye-on-the-future ↩

2. Yegulalp, Serdar (February 10, 2020). "Microsoft speeds up PyTorch with DeepSpeed". InfoWorld. https://www.infoworld.com/article/3526449/microsoft-speeds-up-pytorch-with-deepspeed.html ↩

3. "Microsoft unveils "fifth most powerful" supercomputer in the world". Neowin. 18 June 2023. https://www.neowin.net/news/microsoft-unveils-fifth-most-powerful-supercomputer-in-the-world/ ↩

4. "Microsoft trains world's largest Transformer language model". February 10, 2020. https://venturebeat.com/2020/02/10/microsoft-trains-worlds-largest-transformer-language-model/ ↩

5. "microsoft/DeepSpeed". July 10, 2020 – via GitHub. https://github.com/microsoft/DeepSpeed ↩

6. "DeepSpeed: Accelerating large-scale model inference and training via system optimizations and compression". Microsoft Research. 2021-05-24. Retrieved 2021-06-19. https://www.microsoft.com/en-us/research/blog/deepspeed-accelerating-large-scale-model-inference-and-training-via-system-optimizations-and-compression/ ↩