Hi everyone! 😀 In the last video we've seen how to accelerate the speed of our programs with Pytorch and CUDA - today we will take it another step further with Torch-TensorRT!
We will focus on a Machine Learning process called Inference (which is when the model is trained, perfected and ready to make a prediction).
For this we will load a state-of-the-art artificial neural network and we will use it to classify a picture of my cat! 🙀🙀🙀
Specifically - we will borrow ResNet50 for our little Pytorch experiment! 😉
We will also run a speed test comparing Pytorch models running on CPU, on CUDA and on Torch-TensorRT - which of these do you think is faster??

⏲️ TIMESTAMPS ⏲️
-----------------------------------
00:00 - intro
01:05 - clone Torch-TensorRT
01:40 - install and setup Docker
03:52 - install Nvidia Container Toolkit & Nvidia Docker 2
05:02 - Torch-TensorRT container (option #1)
07:22 - Torch-TensorRT Nvidia NGC container (option #2)
09:00 - import Pytorch
09:16 - load ResNet50
10:25 - load sample image
11:45 - sample image transforms
14:48 - batch size
16:19 - prediction with ResNet50
17:12 - softmax function
18:07 - ImageNet class number to name mapping
20:10 - predict top 5 classes of sample image (topk)
23:33 - speed test benchmark function
27:33 - CPU benchmarks
28:13 - CUDA benchmarks
30:09 - trace model
31:20 - convert traced model into a Torch-TensorRT model
33:02 - TensorRT benchmarks
34:32 - download Jupyter Notebook
34:50 - HOW DID I MISS THIS???
35:31 - thanks for watching!

🛑 REFERENCED TUTORIALS 🛑
----------------------------------------------------------------------
⭐ CUDA Parallel Computing for beginners:
   • CUDA Simply Explained - GPU vs CPU Pa...  
⭐ Neural Networks for beginners:
   • Neural Network Simply Explained - Dee...  
⭐ Machine Learning Databases:
   • Machine Learning Databases and How to...  
⭐ Gradient Descent:
   • Gradient Descent - Simply Explained! ...  

⭐ INSTALLATION LINKS AND BASH COMMANDS ⭐
--------------------------------------------------------------------------------------
1. Clone Torch-TensorRT and change directory:
$ git clone https://github.com/NVIDIA/Torch-TensorRT
$ cd Torch-TensorRT

2. Docker installation guide:
https://docs.nvidia.com/datacenter/cl...

$ curl https://get.docker.com | sh \ && sudo systemctl --now enable docker
$ sudo groupadd docker
$ sudo usermod -aG docker $USER
$ newgrp docker
$ docker run hello-world

3. Nvidia Docker 2 installation:

$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \ && curl -s -L https://nvidia.github.io/nvidia-docke... | sudo apt-key add - \ && curl -s -L https://nvidia.github.io/nvidia-docke... er.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
$ sudo apt-get update
$ sudo apt-get install -y nvidia-docker2
$ sudo systemctl restart docker
$ sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi

4. Official Torch TensorRT Container:

$ docker build -t torch_tensorrt -f ./docker/Dockerfile .
$ docker run --gpus=all --rm -it -v $PWD:/Torch-TensorRT --net=host --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 torch_tensorrt:latest bash
$ cd /Torch-TensorRT/notebooks
$ jupyter notebook --allow-root --ip 0.0.0.0 --port 8888

5. Nvidia NGC Container (alternative to 4):
https://catalog.ngc.nvidia.com/orgs/n...

$ docker pull nvcr.io/nvidia/pytorch:21.12-py3
$ docker run --net=host --gpus all -it --rm -v local_dir:/container_dir nvcr.io/nvidia/pytorch:21.12-py3
$ jupyter notebook --allow-root --ip 0.0.0.0 --port 8888

6. My cats picture:
https://github.com/MariyaSha/Inferenc...

7. Transforms Normalize - Pytorch Documentation:
https://pytorch.org/vision/stable/tra...

8. ImageNet class to name mapping:
https://github.com/pytorch/hub/blob/m...

9. Complete Notebook on Github:
https://github.com/MariyaSha/Inferenc...

---------------------------------------------------------------------------
💗 THANK YOU SO MUCH FOR WATCHING! 💗

- Sound effect by: https://www.zapsplat.com
- Icons by: https://www.flaticon.com/
- Resnet50 image by: https://commons.wikimedia.org/wiki/Fi...