Umass Cs685 F21 Advanced Nlp Tokenization

Summary

from words to subwords to characters/bytes to learned multilingual language models, many-to-many machine translation, multilingual datasets, zero-shot transfer for low-resource ... REALM, k-NN LM, do LMs use retrievals? slides: people.cs. prompt-based learning, prefix tuning, prompt tuning slides: people.cs. BPE, ByT5, charformer slides: course schedule: people.cs. attention mechanisms, recurrent neural language models, self-attention slides: ... T5, text-to-text pretraining, Common Crawl, decoding algorithms, greedy search, beam search, sampling slides: ... language models, n-gram models slides: people.cs. exam review slides: course schedule: people.cs. Transfer learning, pretraining + fine-tuning, ELMo, BERT notes: people.cs. BERT, fine-tuning, classification tasks, QA tasks, RoBERTa, ELECTRA, ALBERT, TransformerXL notes: ...