Natural Language Processing course

Official support channel: #course_nlp_huawei in ODS.ai slack. If you are not registered, please join the community.

Lecturers:

• Valentin Malykh (personal page)

Teaching Assitants:

Logistics:

The lectures are presented at 10:15 MSK each Tuesday online.

Course started: 14th of September 2020

The main lectures end: 16th of November 2020

Description

Natural Language Processing (NLP) is a domain of research whose objective is to analyze and understand human languages and develop technologies to enable human machine interactions with natural languages. NLP is an interdisciplinary field involving linguistics, computer sciences and artificial intelligence. The goal of this course is to provide students with comprehensive knowledge of NLP. Students will be equiped with the principles and theories of NLP, as well as various NLP technologies, including rule-based, statistical and neural network ones. After this course, students will be able to conduct NLP research and develop state-of-the-art NLP systems.

Syllabus

Unit 1: Introduction and Basic Text Processing

• Lecture 01:
  • Theories:
    • NLP Research Questions and Tasks
    • Math & Lingustics Background
    • Chomsky Hierarchy of Grammars and Automata
    • Text Segmentation
    • Tokenization and Stemming
    • Morphology and Universal Morphology Corpus
    • Word frequncies and Zipf’s Law
    • Collocations and Multi-word Expressions
  • Practice:
    • Python Programming & NumPy & Jupyter Notebook
    • NLTK

Unit 2: NLP Techniques

• Lecture 02:
  • Theories:
    • Machine Learning basics
    • Classifiers, Logistic Regressions
    • Stochastic Gradient Descend
    • Vector Space Models and TF-IDFs
    • Text Classification
    • Sentiment Analysis
  • Practice:
    • PyTorch & TensorFlow
    • Assignment 1: Word2Vec
• Lecture 03:
  • Theories:
    • Distributional Semantics and Word Embeddings
    • Word2Vec and Evaluation
    • Softmax and Cross-entropy Loss
    • GLoVe, Fasttext
  • Practice:
    • Word2Vec, Doc2Vec
  • Assignment:
    • 1st assignment is open.
• Lecture 04:
  • Theories:
    • Artificial Neural Networks (ANNs)
    • Multilayer Perceptrons (MLPs)
    • Backpropagation
    • Convolutional Neural Networks (CNN)
    • Text Classification with CNNs
  • Practice:
    • Topiс Modeling and Visualization
• Lecture 05:
  • Theories:
    • Part-of-Speech (POS) Tagging
    • Named Entity Recognition (NER)
    • Maximum Entropy (ME)
    • Sequence Labelling
    • Hidden Markov Models (HMMs)
    • Viterbi Search and Forward-Backword Algorithm
    • Conditional Random Fields (CRFs)
• Lecture 06:
  • Theories:
    • Neural Language Models
    • Recurrent Neural Networks (RNNs)
    • Long Short Term Memory (LSTM) Units
    • Bi-LSTM-CRF Models for Sequence Labeling
  • Practice:
    • Neural Networks Tips and Tricks
    • Regularizations
    • Dropout
    • Initialization
  • Assignments:
    • Assignment 1 Answers
• Lecture 07:
  • Theories:
    • Syntactic Parsing
    • Treebanks
    • Probabilistic Phrase Structure Grammars (PCFGs)
    • Constituent Parings with PCFG
    • Dependency Parsing
    • Parsing with Neural Networks
    • Semantic Role Labeling (optional)
    • Coreference Resolution (optional)
    • Discourse Parsing (optional)
  • Practice:
    • Final Project Selection

Unit 3: NLP Applications

• Lecture 08:
  • Theories:
    • Statistical Machine Translation
    • Statistical Language Models
    • IBM Models
    • Log-linear Framework and Phrase-based Models
    • Beam Search Decoding
    • Machine Translation Evaluation and BLEU
    • Sequence-to-sequence Models
    • Attention Mechanisms
    • RNN-based Neural Machine Translation (NMT)
  • Practice:
    • RNN-based NMT
• Lecture 09:
  • Theories:
    • Subword Level and Character Level NMT
    • Transformers
    • Transformer-based NMT
    • Pre-trained Language Models
    • BERT
    • GPT-2
  • Practice:
    • Transformers
• Lecture 10:
  • Theories:
    • Question Answering
    • Semantic Parsing (optional)
    • Dialog
• Lecture 11 (Guest):
  • Theories:
    • Guest Lectures
  • Practice:
• Lecture 12 (Project Presentation):
  • Theories:
  • Practice:
    • Final Project Presentation

Prerequisites

• Basic Python Programming
• College Calculus, Linear Algebra
• Basic Probability and Statistics
• Foundations of Machine Learning

Assignments

There will be two assignments on the course.

Projects

The participants will be suggested to work on a project during the course. The successfull project development is crucial to pass the course.

The projects from previous semester are available here.