—> With our learnings from T5 and GPT it’s proven that everything will be generative. So let’s dig deep into text generation!
—> We need no annotators, we can just do data augmentation
—> Distillation is now a part of the pipeline - robust and smaller models
—> We have found recipes for handling very long text and get O(n) transformers - A lot of text is more than a few lines long which is a trouble for O(n^2) transformers
—> Training LMs via adversarial methods is better+faster than masking tokens
—> Better evaluation methods for NLP will lead to more robust models
—> Ability to do sample efficient NLP is a super-power
—> Low resource NLP on minority languages needs to be prioritized