Intro Here, we are going to discuss the internals of PyTorch Autograd module. The most of us don’t have to know about this. I was the same till I came across this error: 1 unsupported operand type(s) for *: 'float' and 'NoneType' This came from executing the following code: 1 2 3 4 5 6 import torch a = torch.tensor(5.0, requires_grad=True) * 0.1 b = torch.tensor(2.0, requires_grad=True) c = a + b c.

TTS (Text To Speech) TTS can be viewed as a sequence-to-sequence mapping problem; from a sequence of discrete symbols (text) to a real-valued time series (speech signals). A typical TTS pipeline has two parts; 1) text analysis and 2) speech synthesis. The text analysis part typically includes a number of natural language processing (NLP) steps, such as sentence segmentation, word segmentation, text normalization, part-of-speech (POS) tagging, and grapheme-to-phoneme (G2P) conversion. It takes a word sequence as input and outputs a phoneme sequence with a variety of linguistic contexts.

Distributed Training Why? Need more compute power to process large batches in parallel (DDP) Uses collective communication Large model that couldn’t be fit in memory of one GPU (RPC) Uses P2P communication All of the above XD DDP in Pytorch Every GPU has a model replica, controlled by a process. Every process fetches different batch of data. Forward. Overlapping between computation of and communication(broadcast - allreduced) of gradient. Validation 4 steps-recipe to Distributed Training Initialize Distributed Group 1 init_process_group(backend='nccl') Data Local Training 1 2 3 4 # Wrap train dataset into DataLoader train_loader = DataLoader(train_dataset, batch_size=batch_size) Distributed Training 1 2 3 4 5 6 7 8 9 10 # Create distributed sampler pinned to rank sampler = DistributedSampler(train_dataset, num_replicas=world_size, rank=rank, shuffle=True) # May be True # Wrap train dataset into DataLoader train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=False, # Must be False!

Intro this is intended to be some kind or reference to go through whenever I face some kind of bug, or error, that I don’t know how to solve.usually these kinds of error that doesn’t make sense, or we don’t know the cause of them, and the worst it, that we don’t find many ppl facing the same issue, so internet can’t be so useful then. I learned theses facts the hard way, spending so much time trying to figure out the root of the issue.

Decoupled Neural Interfaces using Synthetic Gradients In NN, the training process, has 3 bottle-necks forward lock: you need to calculate teh output of the previous layer before you can can go into next layer in forward pass backward pass: the same, but for backward propagation weights lock: you can’t update weights unless you do for weights in next layer the paper trying to unlock these bootle-necks by decoupling each layer, to be sufficient alone it does that by introducing, a Synthetic Gradient Model, that can predict the gradient for the current layer, without waiting for the gradient of the next layer this was we can calculate gradient and update weights as soon as we calculate the activation of the current layer Synthetic Gradient Model can be just a simple NN that is trained to output the gradient of the layer

Beatiful commands git log --oneline --decorate --all --graph git merge --abort ==> abort merge, and get back like it never happened git reset --hard ==> is your way to lose all uncommited work in your working directory git fast forward is basically that git moves the commit pointer upward to the new posotion, without creating a merge commit or anything you can merge with --no-ff flag, to disable the fast forward merge and force git to create the merge commit Git Bisect used when something broke, and you know what did broke, but you can’t figure out when did it broke you just give it a testing criteria to test the commit history against Methodology everything inside git is an object all your local branches are located in .