Rico贾若童的博客

Deep Learning - Mixed Floating Point Training

FP16, BF16, Mixed Precision Training

Refresher: Floating Point Calculation A floating point is represented as sign bit | exponent | mantissa. 0 | 10000001 | 10000000000000000000000 represents 6 because: Sign bit 0 represents posi...

Posted by Rico's Nerd Cluster on May 17, 2022

Deep Learning - Speedup Tricks

Torch Optimizer Tricks, Mixed Precision Training

General Speed-Up Tricks If you look to use albumentations for augmentation, sticking to the [batch, H, W, Channels] (channel last) could make data loading faster tensor.contiguou...

Posted by Rico's Nerd Cluster on May 17, 2022

Deep Learning - Common Oopsies

Underflow, Weight Manipulation

Underflow torch.softmax(X) X is zero due to underflow. Sizing Be careful with the last batch if you want to initialize any tensor that’s specific to each batch’s sizes, because it could b...

Posted by Rico's Nerd Cluster on May 17, 2022

Deep Learning - Strategies Part 2 Training And Tuning

Bias And Variance, And Things To Try For Performance Improvement From My Experience

Orthogononalization Orthogonalization in ML means designing a machine learning system such that different aspects of the model can be adjusted independently. This is like “orthogonal vector” so th...

Posted by Rico's Nerd Cluster on May 17, 2022

Deep Learning - Strategies Part 1 Before Model Training

Error Metrics, Data Preparation Principles, Transfer Learning, Multi-Task Learning

Start Your Development Early, Then Iterate Even after many years in speech recognition, Andrew still had some difficulties bringing up a speech recognition system that’s super robust to noises. So...

Posted by Rico's Nerd Cluster on May 17, 2022

Deep Learning - Data Augmentations

Albumentations

Pre-processing Shuffle Data 1 2 3 4 5 6 7 8 9 shuffled_main_dataset = torch.utils.data.Subset( main_dataset, torch.randperm(dataset_size) ) # train_dataset is a Subset object # main_data...

Posted by Rico's Nerd Cluster on May 14, 2022

Deep Learning - PyTorch Data Loading

RESNET-50 Data Loading, Data Transforms, Custom Data Loading

Dataset and Data Loading Data Set and Data Loading in All-Together In Torch In PyTorch, data is stored in the DataSet object. We can read input data all together, or read them one by one. Then, f...

Posted by Rico's Nerd Cluster on May 11, 2022

Deep Learning - Bert

Masked Autoencoder (MAE) The main idea of the Masked Autoencoder (MAE) is to mask parts of an input (e.g., image features) and train a model to reconstruct the original input by leveraging the rem...

Posted by Rico's Nerd Cluster on April 10, 2022

Deep Learning - Bert

Introduction Bert (BiDirectional Encoder Representation Transformer) is great for tasks like question-answering, NER (Named Entity Recognition), sentence classification, etc. Bert is not a transla...

Posted by Rico's Nerd Cluster on April 10, 2022

Deep Learning - Neural Machine Translation

Hands-On Attention Project

Introduction And Data Preparation The goal of the project is experimenting with date translations, i.e., (“25th of June, 2009”) into machine-readable dates (“2009-06-25”). We need to truncate data...

Posted by Rico's Nerd Cluster on April 7, 2022

Rico's Nerd Cluster

Deep Learning - Mixed Floating Point Training

FP16, BF16, Mixed Precision Training

Deep Learning - Speedup Tricks

Torch Optimizer Tricks, Mixed Precision Training

Deep Learning - Common Oopsies

Underflow, Weight Manipulation

Deep Learning - Strategies Part 2 Training And Tuning

Bias And Variance, And Things To Try For Performance Improvement From My Experience

Deep Learning - Strategies Part 1 Before Model Training

Error Metrics, Data Preparation Principles, Transfer Learning, Multi-Task Learning

Deep Learning - Data Augmentations

Albumentations

Deep Learning - PyTorch Data Loading

RESNET-50 Data Loading, Data Transforms, Custom Data Loading

Deep Learning - Bert

Deep Learning - Bert

Deep Learning - Neural Machine Translation

Hands-On Attention Project

FEATURED TAGS

ABOUT ME

FRIENDS