Rico贾若童的博客

Deep Learning - Deeplab Series Theories

Deeplab, ASPP

Motivation UNet has restricted receptive fields. It is sufficient for identifying local areas such as tumors (in medical images). When we learn larger image patches, UNet was not sufficient. Deep...

Posted by Rico's Nerd Cluster on August 1, 2022

Deep Learning - Spatial Pyramid Pooling

SPP, Ablation Study

SPP (Spatial Pyramid Pooling) Layer In Deep learning, “spatial dimensions” means “height and width”. Some networks need fixed input size for their fully connected layers. When does that happen? ...

Posted by Rico's Nerd Cluster on August 1, 2022

Deep Learning - PyTorch Profiling

PyTorch Profiler

Sample Code 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 #!/usr/bin/env python3 import torch import torchvision.models as models from torch.profiler import profile...

Posted by Rico's Nerd Cluster on July 25, 2022

Deep Learning - Tools

wandb, tqdm

Wandb wandb is a visualization tool that records various deep learning experiment data. It never discloses what databases it uses, but it might be a combination of cloud-based scalable databases s...

Posted by Rico's Nerd Cluster on July 23, 2022

Deep Learning - Model Deployment

Model Deployment

Deployment On Edge Devices Inferencing Using C++ Embedded systems Need streaming. ZeroMQ might be better than Websocket. Need an article A serializable model mean...

Posted by Rico's Nerd Cluster on May 21, 2022

Deep Learning - Knowledge Distillation

Knowledge Distillation

Introduction To Knowledge Distillation The goal of Knowledge Distillation is to train a small “student” network to mimic the output of a large “teacher” network. This is a.k.a “model compression” ...

Posted by Rico's Nerd Cluster on May 21, 2022

Deep Learning - Inferencing

Autograd Profiler

Autograd Profiler PyTorch’s Autograd Profiler provides information on the resources (CPU and GPU) for each operation in a model. 1 2 3 4 5 6 import torch.autograd.profiler as profiler with profi...

Posted by Rico's Nerd Cluster on May 20, 2022

Deep Learning - Mixed Floating Point Training

FP16, BF16, Mixed Precision Training

Refresher: Floating Point Calculation A floating point is represented as sign bit | exponent | mantissa. 0 | 10000001 | 10000000000000000000000 represents 6 because: Sign bit 0 represents posi...

Posted by Rico's Nerd Cluster on May 17, 2022

Deep Learning - Speedup Tricks

Torch Optimizer Tricks, Mixed Precision Training

General Speed-Up Tricks If you look to use albumentations for augmentation, sticking to the [batch, H, W, Channels] (channel last) could make data loading faster tensor.contiguou...

Posted by Rico's Nerd Cluster on May 17, 2022

Deep Learning - Common Oopsies

Underflow, Weight Manipulation

Underflow torch.softmax(X) X is zero due to underflow. Sizing Be careful with the last batch if you want to initialize any tensor that’s specific to each batch’s sizes, because it could b...

Posted by Rico's Nerd Cluster on May 17, 2022

Rico's Nerd Cluster

Deep Learning - Deeplab Series Theories

Deeplab, ASPP

Deep Learning - Spatial Pyramid Pooling

SPP, Ablation Study

Deep Learning - PyTorch Profiling

PyTorch Profiler

Deep Learning - Tools

wandb, tqdm

Deep Learning - Model Deployment

Model Deployment

Deep Learning - Knowledge Distillation

Knowledge Distillation

Deep Learning - Inferencing

Autograd Profiler

Deep Learning - Mixed Floating Point Training

FP16, BF16, Mixed Precision Training

Deep Learning - Speedup Tricks

Torch Optimizer Tricks, Mixed Precision Training

Deep Learning - Common Oopsies

Underflow, Weight Manipulation

FEATURED TAGS

ABOUT ME

FRIENDS