108.413A: Computational Linguistics

Hyopil Shin (Graduate School of Data Science and Dept. of Linguistics, Seoul National University)

hpshin@snu.ac.kr, http://knlp.snu.ac.kr
Tue/Thur  11:00 to 12:15 in building 3 room 102

T.A: 이상아 (visualjan@snu.ac.kr), 장한솔 (624jhs@snu.ac.kr)


(http://www.theverge.com/2016/3/11/11208078/lee-se-dol-go-google-kasparov-jennings-ai) ( http://jalammar.github.io/illustrated-bert/)

Course Description

Recently deep learning  approaches have obtained very high performance across many different computational linguistics or Natural Language Processing (NLP). This course provides an introduction to the Neural Network Model and  deep learning methodologies applied to NLP from scratch. On the model side we will start from basic notions of Neural Networks such as linear/logistic regression, perceptrons, backpropagations, and parameter optimizations. Then we will  cover actual Neural Network models including Feedforward, Convolutional, Recurrent, and Long Short Term Neural Networks. Along the lines of the models, various word/sentence/contextual embeddings and attention mechanism will also be dealt with in depth. The first part of the class focuses on the basics of neural network models and the second part covers actual implementations of NLP tasks such as sentiment analysis(movie/text classifications), and text generations. We will take advantage of modules from Python 3.x and  PyTorch. Through lectures and programming assignments students will learn the necessary implementation tricks for making neural networks work on practical problems.  


  • 코로나 19로 인해 3월 17일부터  Zoom을 사용한 온라인 수업을 시행합니다. https://zoom.us/ 에서 다운받으시길 바랍니다.
  • 수업시작전에 ETL 공지사항에 클래스 초대 url을 명시할 예정입니다. 이 경로를 따라 들어오시면 됩니다.

  • Please set up python, pytorch, and colab for class!

Useful Sites

  • Lectures

Textbook and Sites


Deep Learning from Scratch (밑바닥부터 시작하는 딥러닝), by 사이토 고키, 한빛출판사. Deep Learning From Scratch source codes

DL wizard

Deep Learning Tutorials based on PyTorch


Date Topics Related Materials and           Resources
1 3/15-3/21

Introduction to Computational Linguistics / Natural Language Processing


Install Python 3.x and  PyTorch

모두를 위한 머신러닝/딥러닝(홍콩과기대 김성훈 교수)
  • Linear Regression의 개념:비디오, 강의 슬라이드
  • Linear Regression cost함수 최소화: 비디오, 강의 슬라이
  • 여러 개의 입력(feature)의 Linear Regression: 비디오, 강의 슬라이드
  • Logistic Regression classification: 강의 슬라이드-Hypothesis 함수 소개: 비디오- cost 함수 소개: 비디오

How to Implement Simple Linear Regression From Scratch with Python

Logistic Regression for Machine Learning

2 3/22-3/28 Introduction to a Neural Network PyTorch:
딥러닝 개념잡기
3 3/29-4/4 Introduction to a Neural Network PyTorch:
  Hw1: Logistic Regression: Diabetes Data

4 4/5-4/11

Introduction to a Neural Network

  • Parameter Optimization
  • Weight Decay
  • Batch Normalization
  • DropOut

Hyper-parameter Tuning Techniques in Deep Learning

An Overview of Gradient Descent Optimization Algorithm


Hw2: How To Prepare Movie Review Data For Sentiment Analysis
5 4/12-4/18

Introduction to a Neural Network

  • Parameter Optimization
  • Weight Decay
  • Batch Normalization
  • DropOut

Hyper-parameter Tuning Techniques in Deep Learning

An Overview of Gradient Descent Optimization Algorithm


Hw3: IMDB_TwoLayerNet

6 4/19-4/25

Convolutional Neural Network

Understanding Convolutional Neural Network for  NLP


7 4/26-4/25


Reccurent Neural Network
A Friendly Introduction to Recurrent Neural Network

Long Short-Term Memory Neural Network and Gated Recurrent Unit

Mid-Term Test


MidTermProject: Korean News Data Torch FNN

8 5/3-5/9 NLP Task 1: Sentiment Analysis
PyTorch Sentiment Analysis (IMDB)

NLP Task 2: Sentiment Analysis for Korean
Naver Movie Review Sentiment Analysis

A Comprehensive Introduction to Torchtext

Torchtext Github

5/10-5/16 Encoder-Decoder
Encoder-Decoder Long Short-Term Memory Networks
A Gentle Introduction to LSTM Autoencoders
Step-by-step Understanding LSTM Autoencoder layers




Attention Model
Neural Machine Translation By Jointly Learning to Align and Translate

Attention: Illustrated Attention
Attention and Memory in Deep Learning and NLP
Translation with Sequence to Sequence Network and Attention

11 5/24-5/30 Transformer
Self Attention: Attention is All you need

The Illustrated Transformer
Seq2Seq Pay Attention to Self Attention: Part I
Seq2seq Pay Attention to Self Attention: Part 2

Translation with Sequence to Sequence Network and Attention

PyTorch-Transformers by Huggingface and Full Documentation

12 5/31-6/6
BERT (Bidirectional Encoder Representations from Transformers)

What is XLNet and why it outperforms BERT?
XLNet - a clever language modeling solution
XLNET -SOTA pre-training method that outperforms BERT

XLM - Enhancing BERT for Cross-lingual Language Model

Introducing FastBert - A Simple Deep Learning Library for BERT Models

RoBERTa: A Robustly Optimized BERT Pretaining Approach

RoBERTa: An optimized method for self-supervised NLP systems

Distilling BERT - How to achieve BERT performance using Logistic Regression

Meet ALBERT: a new 'Lite BERT' from Google & Toyota With State of the ART NLP performance and 18x fewer parameters

BERT, RoBERTa, DIstilBERT, XLNet - which one to use?
The Annotated Transformer

BERT Fine Tuning
BERT Fine-Tuning Tutorial with PyTorch
Painless Fine-Tuning of BERT in Pytorch

The Latest Breakthroughs and Developments in Natural Language Processing

13 6/6-6/13

Embeddings (word embeddings)

Sebastian Ruder의 On word Embeddings Part1, 2, 3, 4:  A hands-on Intuitive Approach to Deep Learning Methods for Text Data - Word2Vec, Glove, and FastText

The Current Best of Universal Word Embeddings and Sentence Embeddings




Word Embeddings: Encoding Lexical Semantics

BERT Word Embedding

14 6/14-6/20 Final Test