Computational Linguistics

Course Description

Recently deep learning approaches have obtained very high performance across many different computational linguistics or Natural Language Processing (NLP). This course provides an introduction to the Neural Network Model and deep learning methodologies applied to NLP from scratch. On the model side we will start from basic notions of Neural Networks such as linear/logistic regression, perceptrons, backpropagations, and parameter optimizations. Then we will cover actual Neural Network models including Feedforward, Convolutional, Recurrent, and Long Short Term Neural Networks. Along the lines of the models, various word/sentence/contextual embeddings and attention mechanism will also be dealt with in depth. The first part of the class focuses on the basics of neural network models and the second part covers actual implementations of NLP tasks such as sentiment analysis(movie/text classifications), and text generations. We will take advantage of modules from Python 3.x and PyTorch. Through lectures and programming assignments students will learn the necessary implementation tricks for making neural networks work on practical problems.

Updates

코로나 19로 인해 3월 17일부터 Zoom을 사용한 온라인 수업을 시행합니다. https://zoom.us/ 에서 다운받으시길 바랍니다.
수업시작전에 ETL 공지사항에 클래스 초대 url을 명시할 예정입니다. 이 경로를 따라 들어오시면 됩니다.

Please set up python, pytorch, and colab for class!

Useful Sites

Lectures

Deep Learning From Scratch source codes

PyTorch

Other Resources

Jupyter notebook

Jupyter notebook for beginners-A tutorial

Bring the best out of Jupyter notebooks for Data science-Enhance jupyter notebook’s productivity with these tips & tricks

Jump out of the Jupyter Notebook with nbconvert

Jupyter Notebook Extensions

Google Colabatory

Textbook and Sites

Deep Learning from Scratch (밑바닥부터 시작하는 딥러닝), by 사이토 고키, 한빛출판사. Deep Learning From Scratch source codes

DL wizard

Deep Learning Tutorials based on PyTorch

Syllabus

	Date	Topics	Related Materials and Resources	Assignments
1	3/15-3/21	Introduction to Computational Linguistics / Natural Language Processing Preliminaries: Machine Learning -Basics Machine Learning is Fun! (The world's easiest introduction to Machine Learning, part1~part8) Natural Language Processing is Fun! Recap of Linear Regression and Logistic Regression Linear Regression From Scratch with Python Linear Regression Simplified-Ordinary Least Square vs Gradient Descent Logistic Regression	PyTorch: Deep Learning With PyTorch: A 60 Minute Blitz Learning PyTorch with Examples Matrices Linear Regression With PyTorch Logistic Regression With PyTorch PyTorch 실습	Install Python 3.x and PyTorch 모두를 위한 머신러닝/딥러닝(홍콩과기대 김성훈 교수) Linear Regression의 개념:비디오, 강의 슬라이드 Linear Regression cost함수 최소화: 비디오, 강의 슬라이 여러 개의 입력(feature)의 Linear Regression: 비디오, 강의 슬라이드 Logistic Regression classification: 강의 슬라이드-Hypothesis 함수 소개: 비디오- cost 함수 소개: 비디오 How to Implement Simple Linear Regression From Scratch with Python Logistic Regression for Machine Learning
2	3/22-3/28	Introduction to a Neural Network Perceptron Neural Network: Activation Function(sigmoid, step, tanh, ReLU), Softmax Neural Network Training: Loss Function, Gradient Descent Method Backward Propagation of Errors	PyTorch: Feedforward Neural Network with PyTorch Gradient With PyTorch Forwardpropagation, Backpropagation and Gradient Descent with PyTorch	딥러닝 개념잡기 데이터셋 이야기 학습과정 이야기 학습과정 살펴보기 학습 조기종료 시키기 평가 이야기 학습 모델 보기/저장하기/불러오기
3	3/29-4/4	Introduction to a Neural Network Perceptron Neural Network: Activation Function(sigmoid, step, tanh, ReLU), Softmax Neural Network Training: Loss Function, Gradient Descent Method Backward Propagation of Errors	PyTorch: Feedforward Neural Network with PyTorch Gradient With PyTorch Forwardpropagation, Backpropagation and Gradient Descent with PyTorch	Hw1: Logistic Regression: Diabetes Data Diabetes Data
4	4/5-4/11	Introduction to a Neural Network Parameter Optimization Weight Decay Batch Normalization DropOut Hyper-parameter Tuning Techniques in Deep Learning An Overview of Gradient Descent Optimization Algorithm	PyTorch: Derivative, Gradient and Jacobian Learning Rate Scheduling Optimization Algorithms Weight Initialization and Activation Functions	Hw2: How To Prepare Movie Review Data For Sentiment Analysis
5	4/12-4/18	Introduction to a Neural Network Parameter Optimization Weight Decay Batch Normalization DropOut Hyper-parameter Tuning Techniques in Deep Learning An Overview of Gradient Descent Optimization Algorithm	PyTorch: Derivative, Gradient and Jacobian Learning Rate Scheduling Optimization Algorithms Weight Initialization and Activation Functions	Hw3: IMDB_TwoLayerNet
6	4/19-4/25	Convolutional Neural Network Convolutional Neural Network 요약 Understanding Convolutional Neural Network for NLP	PyTorch: Convolutional Neural Network with PyTorch
7	4/26-4/25	Reccurent Neural Network A Friendly Introduction to Recurrent Neural Network Long Short-Term Memory Neural Network and Gated Recurrent Unit Mid-Term Test	PyTorch: Recurrent Neural Network With PyTorch Long Short-Term Memory Neural Network With PyTorch	MidTermProject: Korean News Data Torch FNN Korean News Data
8	5/3-5/9	NLP Task 1: Sentiment Analysis PyTorch Sentiment Analysis (IMDB) NLP Task 2: Sentiment Analysis for Korean Naver Movie Review Sentiment Analysis	A Comprehensive Introduction to Torchtext Torchtext Github
9	5/10-5/16	Encoder-Decoder Encoder-Decoder Long Short-Term Memory Networks A Gentle Introduction to LSTM Autoencoders Step-by-step Understanding LSTM Autoencoder layers	PyTorch: Sequence to Sequence Learning with Neural Networks Translation with Sequence to Sequence Network and Attention
10	5/17-5/23	Attention Model Neural Machine Translation By Jointly Learning to Align and Translate Attention: Illustrated Attention Attention and Memory in Deep Learning and NLP	PyTorch: Translation with Sequence to Sequence Network and Attention
11	5/24-5/30	Transformer Self Attention: Attention is All you need The Illustrated Transformer Seq2Seq Pay Attention to Self Attention: Part I Seq2seq Pay Attention to Self Attention: Part 2	PyTorch: Translation with Sequence to Sequence Network and Attention PyTorch-Transformers by Huggingface and Full Documentation
12	5/31-6/6	BERT (Bidirectional Encoder Representations from Transformers) The Illustrated BERT, ELMo, and co. (How NLP cracked Transfer Learning) Dissecting BERT Part 1: Understanding the Transformer Dissecting BERT Part2: BERT Specifics Dissecting BERT Appendix: The Decoder XLNet What is XLNet and why it outperforms BERT? XLNet - a clever language modeling solution XLNET -SOTA pre-training method that outperforms BERT XLM XLM - Enhancing BERT for Cross-lingual Language Model FastBert Introducing FastBert - A Simple Deep Learning Library for BERT Models RoBERTa RoBERTa: A Robustly Optimized BERT Pretaining Approach RoBERTa: An optimized method for self-supervised NLP systems Distilling BERT - How to achieve BERT performance using Logistic Regression Meet ALBERT: a new 'Lite BERT' from Google & Toyota With State of the ART NLP performance and 18x fewer parameters BERT, RoBERTa, DIstilBERT, XLNet - which one to use?	PyTorch: The Annotated Transformer BERT Fine Tuning BERT Fine-Tuning Tutorial with PyTorch Painless Fine-Tuning of BERT in Pytorch The Latest Breakthroughs and Developments in Natural Language Processing
13	6/6-6/13	Embeddings (word embeddings) Sebastian Ruder의 On word Embeddings Part1, 2, 3, 4: Part 1: On word embeddings Part 2: Approximating the Softmax Part 3: The secret ingredients of word2vec Part 4: A survey of cross-lingual embedding models Part 5: Word Embeddings in 2017- Trends in Future directions A hands-on Intuitive Approach to Deep Learning Methods for Text Data - Word2Vec, Glove, and FastText The Current Best of Universal Word Embeddings and Sentence Embeddings Notebook	PyTorch: Word Embeddings: Encoding Lexical Semantics BERT Word Embedding
14	6/14-6/20	Final Test

108.413A: Computational Linguistics

Course Description

Updates

Useful Sites

Textbook and Sites

Syllabus