Studies in Computational Linguistics 1

Course Description

Recently deep learning approaches have obtained very high performance across many different computational linguistics or Natural Language Processing (NLP). This course provides an introduction to the Neural Network Model and deep learning methodologies applied to NLP from scratch. On the model side we will start from basic notions of Neural Networks such as linear/logistic regression, perceptrons, backpropagations, and parameter optimizations. Then we will cover actual Neural Network models including Feedforward, Convolutional, Recurrent, and Long Short Term Neural Networks. Along the lines of the models, various word/sentence/contextual embeddings and attention mechanism will also be dealt with in depth. The first part of the class focuses on the basics of neural network models and the second part covers actual implementations of NLP tasks such as sentiment analysis(movie/text classifications), and text generations. We will take advantage of modules from Python 3.x and PyTorch. Through lectures and programming assignments students will learn the necessary implementation tricks for making neural networks work on practical problems.

Updates

Please set up python, pytorch, and colab for class!

Useful Sites

Lectures

Deep Learning From Scratch source codes

PyTorch

Other Resources

Jupyter notebook

Jupyter notebook for beginners-A tutorial

Bring the best out of Jupyter notebooks for Data science-Enhance jupyter notebook’s productivity with these tips & tricks

Jump out of the Jupyter Notebook with nbconvert

Jupyter Notebook Extensions

Google Colabatory

Textbook and Sites

Deep Learning from Scratch (밑바닥부터 시작하는 딥러닝), by 사이토 고키, 한빛출판사. Deep Learning From Scratch source codes

DL wizard

Deep Learning Tutorials based on PyTorch

Syllabus

	Date	Topics	Related Materials and Resources	Assignments
1	9/1-9/7	Introduction to Computational Linguistics / Natural Language Processing Preliminaries: Machine Learning -Basics Machine Learning is Fun! (The world's easiest introduction to Machine Learning, part1~part8) Natural Language Processing is Fun! Recap of Linear Regression and Logistic Regression Linear Regression From Scratch with Python Linear Regression Simplified-Ordinary Least Square vs Gradient Descent Logistic Regression	PyTorch: Deep Learning With PyTorch: A 60 Minute Blitz Learning PyTorch with Examples Matrices Linear Regression With PyTorch Logistic Regression With PyTorch	Install Python 3.x and PyTorch 모두를 위한 머신러닝/딥러닝(홍콩과기대 김성훈 교수) Linear Regression의 개념:비디오, 강의 슬라이드 Linear Regression cost함수 최소화: 비디오, 강의 슬라이 여러 개의 입력(feature)의 Linear Regression: 비디오, 강의 슬라이드 Logistic Regression classification: 강의 슬라이드-Hypothesis 함수 소개: 비디오- cost 함수 소개: 비디오 How to Implement Simple Linear Regression From Scratch with Python Logistic Regression for Machine Learning
2	9/8-9/14	Introduction to a Neural Network Perceptron Neural Network: Activation Function(sigmoid, step, tanh, ReLU), Softmax Neural Network Training: Loss Function, Gradient Descent Method Backward Propagation of Errors	PyTorch: Feedforward Neural Network with PyTorch Gradient With PyTorch Forwardpropagation, Backpropagation and Gradient Descent with PyTorch	딥러닝 개념잡기 데이터셋 이야기 학습과정 이야기 학습과정 살펴보기 학습 조기종료 시키기 평가 이야기 학습 모델 보기/저장하기/불러오기
3	9/15-9/21	Introduction to a Neural Network Perceptron Neural Network: Activation Function(sigmoid, step, tanh, ReLU), Softmax Neural Network Training: Loss Function, Gradient Descent Method Backward Propagation of Errors	PyTorch: Feedforward Neural Network with PyTorch Gradient With PyTorch Forwardpropagation, Backpropagation and Gradient Descent with PyTorch	HomeWork Assignment 1: Backpropagation Calculation in Pytorch (refer to ETL)
4	9/22-9/28	Introduction to a Neural Network Parameter Optimization Weight Decay Batch Normalization DropOut Hyper-parameter Tuning Techniques in Deep Learning An Overview of Gradient Descent Optimization Algorithm New State of the Art AI Optimizer: Rectified Adam (RAdam) New Deep Learning Optimizer, Ranger: Synergistic combination of RAdam + LookAhead for the best of both	PyTorch: Derivative, Gradient and Jacobian Learning Rate Scheduling Optimization Algorithms Weight Initialization and Activation Functions
5	9/29-10/5	Introduction to a Neural Network Parameter Optimization Weight Decay Batch Normalization DropOut Hyper-parameter Tuning Techniques in Deep Learning An Overview of Gradient Descent Optimization Algorithm New State of the Art AI Optimizer: Rectified Adam (RAdam) New Deep Learning Optimizer, Ranger: Synergistic combination of RAdam + LookAhead for the best of both	PyTorch: Derivative, Gradient and Jacobian Learning Rate Scheduling Optimization Algorithms Weight Initialization and Activation Functions
6	10/6-10/12	Convolutional Neural Network Convolutional Neural Network 요약 Understanding Convolutional Neural Network for NLP Convolutional Neural Networks for Sentence Classification Character-Aware Neural Language Models	PyTorch: Convolutional Neural Network with PyTorch	HomeWork Assignment 2: Convolutional Neural Networks Paper 요약 (refer to ETL)
7	10/13-10/19	Reccurent Neural Network A Friendly Introduction to Recurrent Neural Network Long Short-Term Memory Neural Network and Gated Recurrent Unit	PyTorch: Recurrent Neural Network With PyTorch Long Short-Term Memory Neural Network With PyTorch
8	10/20-10/26	Encoder-Decoder Encoder-Decoder Long Short-Term Memory Networks A Gentle Introduction to LSTM Autoencoders Step-by-step Understanding LSTM Autoencoder layers	PyTorch: Sequence to Sequence Learning with Neural Networks Translation with Sequence to Sequence Network and Attention
9	10/27-11/2	Attention Model Neural Machine Translation By Jointly Learning to Align and Translate Attention: Illustrated Attention Attention and Memory in Deep Learning and NLP	PyTorch: Translation with Sequence to Sequence Network and Attention	HomeWork Assignment 3: Attention is All you need(refer to ETL)
10	11/3-11/9	Transformer Self Attention: Attention is All you need The Illustrated Transformer Seq2Seq Pay Attention to Self Attention: Part I Seq2seq Pay Attention to Self Attention: Part 2	PyTorch: Translation with Sequence to Sequence Network and Attention PyTorch-Transformers by Huggingface and Full Documentation
11	11/10-11/16	BERT (Bidirectional Encoder Representations from Transformers) The Illustrated BERT, ELMo, and co. (How NLP cracked Transfer Learning) Dissecting BERT Part 1: Understanding the Transformer Dissecting BERT Part2: BERT Specifics Dissecting BERT Appendix: The Decoder XLNet What is XLNet and why it outperforms BERT? XLNet - a clever language modeling solution XLNET -SOTA pre-training method that outperforms BERT XLM XLM - Enhancing BERT for Cross-lingual Language Model FastBert Introducing FastBert - A Simple Deep Learning Library for BERT Models RoBERTa RoBERTa: A Robustly Optimized BERT Pretaining Approach RoBERTa: An optimized method for self-supervised NLP systems Distilling BERT - How to achieve BERT performance using Logistic Regression Meet ALBERT: a new 'Lite BERT' from Google & Toyota With State of the ART NLP performance and 18x fewer parameters BERT, RoBERTa, DIstilBERT, XLNet - which one to use?	PyTorch: The Annotated Transformer BERT Fine Tuning BERT Fine-Tuning Tutorial with PyTorch Painless Fine-Tuning of BERT in Pytorch The Latest Breakthroughs and Developments in Natural Language Processing	HomeWork Assignment 4: BERT, XLNet, RoBERTa, DistilBERT Fine Tuning (refer to ETL)
12	11/17-11/23	Assignment4:BERT, XLNet, RoBERTa, DistilBERT Fine Tuning Presentation NLP Task 1: Sentiment Analysis PyTorch Sentiment Analysis (IMDB)	NLP Task 2: Sentiment Analysis for Korean Naver Movie Review Sentiment Analysis A Comprehensive Introduction to Torchtext Torchtext Github	Get ready for your presentation!
13	11/24-11/30	Embeddings (word embeddings) Sebastian Ruder의 On word Embeddings Part1, 2, 3, 4: Part 1: On word embeddings Part 2: Approximating the Softmax Part 3: The secret ingredients of word2vec Part 4: A survey of cross-lingual embedding models Part 5: Word Embeddings in 2017- Trends in Future directions A hands-on Intuitive Approach to Deep Learning Methods for Text Data - Word2Vec, Glove, and FastText The Current Best of Universal Word Embeddings and Sentence Embeddings Notebook	PyTorch: Word Embeddings: Encoding Lexical Semantics BERT Word Embedding	Guest Talk in Class on 11/26: Character Mining: Machine Comprehension By Prof. Jinho Choi at Emory University Linguistics Colloquium on 11/26 at 5:00: Emora: Social Chatbot that Cares For you By Prof. Jinho Choi at Emory University
14	12/8-12/14	NLP Task 2: Chatbot Chatbot Tutorial NLP Task 3: Generating and Classifying Names with a Character-Level RNN Generating Names Classifying Names
15	12/15-12/21	Final Project Presentation

108.413A: Studies in Computational Linguistics1

Course Description

Updates

Useful Sites

Textbook and Sites

Syllabus