Course Outline
Introduction
- Overview of challenges in scaling deep learning
- Overview of DeepSpeed and its key features
- DeepSpeed compared to other distributed deep learning libraries
Getting Started
- Setting up the development environment
- Installing PyTorch and DeepSpeed
- Configuring DeepSpeed for distributed training
DeepSpeed Optimisation Features
- DeepSpeed training pipeline
- ZeRO (memory optimisation)
- Activation checkpointing
- Gradient checkpointing
- Pipeline parallelism
Scaling Models with DeepSpeed
- Basic scaling techniques using DeepSpeed
- Advanced scaling methodologies
- Performance considerations and best practices
- Debugging and troubleshooting techniques
Advanced DeepSpeed Topics
- Advanced optimisation techniques
- Using DeepSpeed with mixed precision training
- Deploying DeepSpeed across different hardware (e.g., GPUs, TPUs)
- DeepSpeed with multiple training nodes
Integrating DeepSpeed with PyTorch
- Integrating DeepSpeed into PyTorch workflows
- Using DeepSpeed with PyTorch Lightning
Troubleshooting
- Debugging common DeepSpeed issues
- Monitoring and logging
Summary and Next Steps
- Recap of key concepts and features
- Best practices for deploying DeepSpeed in production
- Further resources to deepen your understanding of DeepSpeed
Requirements
- Intermediate understanding of deep learning principles
- Experience with PyTorch or similar deep learning frameworks
- Familiarity with Python programming
Audience
- Data scientists
- Machine learning engineers
- Developers
Testimonials (3)
I really liked the end where we took the time to play around with CHAT GPT. The room was not set up the best for this- instead of one large table a couple of small ones so we could get into small groups and brainstorm would have helped
Nola - Laramie County Community College
Course - Artificial Intelligence (AI) Overview
Working from first principles in a focused way, and moving to applying case studies within the same day
Maggie Webb - Department of Jobs, Regions, and Precincts
Course - Artificial Neural Networks, Machine Learning, Deep Thinking
It felt like we were going through directly relevant information at a good pace (i.e. no filler material)