What is Peer-review

Peer-review is a corner-stone in scientific research. It serves a quality check for scientific drafts. It is formally defined as: Peer review is the evaluation of work by one or more people with similar competencies as the producers of the work.

Why is it important

It’s important as, only after an article passes peer review can it be officially published, and it’s the knowledge shared with the scientific community and the world.

You and your Research

https://www.cs.virginia.edu/~robins/YouAndYourResearch.html

Main Ideas

• Commit to your idea
• Ask yourself: What are the important problems in my field?
• Communicate with the bright minds
• Ask the important questions
• Closed door, or open door
  • when you work with door closed
    • you work harder
    • you work on the wrong thing
  • open door working
    • get many interruptions
    • get important clues
  • My take on this is we should keep an open mind about what other people are doing and where the research is heading

Intro

Here, we are going to discuss the internals of PyTorch Autograd module. The most of us don’t have to know about this. I was the same till I came across this error:

1

unsupported operand type(s) for *: 'float' and 'NoneType'

This came from executing the following code:

1
2
3
4
5
6

import torch
a = torch.tensor(5.0, requires_grad=True) * 0.1
b = torch.tensor(2.0, requires_grad=True)
c = a + b
c.backward()
a += + 0.1 * a.grad

But why? We defined that the gradient of a should be calculated by putting requires_grad to be True!

TTS (Text To Speech)

TTS can be viewed as a sequence-to-sequence mapping problem; from a sequence of discrete symbols (text) to a real-valued time series (speech signals). A typical TTS pipeline has two parts; 1) text analysis and 2) speech synthesis. The text analysis part typically includes a number of natural language processing (NLP) steps, such as sentence segmentation, word segmentation, text normalization, part-of-speech (POS) tagging, and grapheme-to-phoneme (G2P) conversion. It takes a word sequence as input and outputs a phoneme sequence with a variety of linguistic contexts. The speech synthesis part takes the context-dependent phoneme sequence as its input and outputs a synthesized speech waveform.

Distributed Training

Why?

• Need more compute power to process large batches in parallel (DDP)

  • Uses collective communication

• Large model that couldn’t be fit in memory of one GPU (RPC)

  • Uses P2P communication

• All of the above XD

DDP in Pytorch

• Every GPU has a model replica, controlled by a process.
• Every process fetches different batch of data.
• Forward.
• Overlapping between computation of and communication(broadcast - allreduced) of gradient.
• Validation

Intro

• this is intended to be some kind or reference to go through whenever I face some kind of bug, or error, that I don’t know how to solve.usually these kinds of error that doesn’t make sense, or we don’t know the cause of them, and the worst it, that we don’t find many ppl facing the same issue, so internet can’t be so useful then.
• I learned theses facts the hard way, spending so much time trying to figure out the root of the issue.

Debugging Facts

• here are some tips, to help whenever you are facing some error, trying get a package to work, or you these kinds of error that takes you life few days, you know.

Package installation issues

Python version

• this might sound trivial, but every package version, only works with some python versions
• so you should run pip install package-name== to get the package versions supported by your python version

Update Pip

• whenever there’s a dependency conflict, or versions conflict, and you can’t install a package, then check your pip and update it if possible
• I spent like 4 days clueless why a simple package that I installed a week before, won’t install now, like out of a sudden, it won’t install anymore, due to dependency packages conflict
• when I upgraded pip, it simply worked

Check for broken installations

• many times we would try to install some something, then for whatever reason, it wouldn’t complete successful. But then we would try to install again, and we would encounter strange errors, that we don’t know the root for them.
• there error could be because there’s a broken installation, that messed things up.
• so, when we delete, or remove this broken installation, and install again, it just works.