Hyopil Shin (Dept. of Linguistics, Seoul National University)
hpshin@snu.ac.kr
http://knlp.snu.ac.kr/
Mon 1:00 to
3:45 in building room
T.A: TBA
(http://www.theverge.com/2016/3/11/11208078/lee-se-dol-go-google-kasparov-jennings-ai)
이 과목에서는
자연언어처리(Natural Language Processing) 또는
컴퓨터언어학(Computational Linguistics)의 이론적인 기초에서부터 최근의
Transformers, BERT 기반의 방법론을 학습한다. 강의 전반부에서는 N-gram,
Entropy, Embedding에 관한 내용이 다루어지며 후반부에는
Encoder-Decoder, Attention, Transformer를 학습하고
Huggingface의 Transformers의 사전학습모델과 모듈을 사용하여 자연언어처리에 활용하는
다양한 태스크를 실제 구현해 보도록 한다. 프로그래밍으로 Pytorch가 다루어지며 모든 과제는 토치를
기반으로 구현하도록 한다. 파이선 및 딥러닝 기본 지식이 요구된다. 이 수업을 통해 자연언어처리의
기본개념에서부터 최근의 방법론까지 학습하여 실제 언어처리에 활용할 수 있는 능력을 키우도록 한다.
Date | Topics | Related Materials and
Resources |
PyTorch |
|
1 | 9/5 |
Introduction to Natural Language Processing
Language Modeling 1- Statistical Language
Modeling: N-Grams |
Natural
Language Processing is Fun! Language Modeling and with N-Grams |
PyTorch: |
2 | 9/12 | Language Modeling
1- Statistical Language Modeling: Entropy and Maximum
Entropy Models (Chuseok Holiday- Zoom Class) |
Entropy is a Measure of Uncertainty | |
3 | 9/19 | Text Classification |
Text Classification |
|
4 | 9/26 | Vector Semantics Language Modeling II: Static Word Embedding
|
Vector Semantics and Embeddings | PyTorch: Linear Regression With PyTorch Logistic Regression With PyTorch |
5 | 10/3 | Language Modeling II:
Static Word Embedding (Holiday, Zoom Class) |
Vector Semantics
and Embeddings |
PyTorch:
|
6 | 10/10 |
Sequence to Sequence
Model: Encoder-Decoder |
PyTorch:
|
|
7 | 10/17 | Attention
Model Neural Machine Translation By Jointly Learning to Align and Translate
|
Attention: Illustrated Attention | PyTorch:
|
8 | 10/24 | Transformer Self Attention: Attention is All you need |
PyTorch: |
|
9 |
10/31 | Language
Modeling III:
Dynamic Word
Embedding : BERT
(Bidirectional Encoder
Representations from
Transformers) |
BERT Fine Tuning BERT Fine-Tuning Tutorial with PyTorch BERT Word Embeddings |
|
10 |
11/7 |
Pre-trained
Models and Transfer Learning
|
|
XLM-R:
Unsupervised Cross-lingual Representation Learning
at Scale XLNet: Generalized Autoregressive Pretraining for Language Understanding MASS: Masked Sequence to Sequence Pre-training for Language Generation BART:Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension GLM: All NLP Tasks Are Generation Tasks: A General Pretraining Framework SpanBERT: Improving Pre-training by Representing and Predicting Spans |
11 | 11/14 |
Pre-trained Models:
Interpretation and Theoretical
Analysis
|
A
Structural Probe for Finding Syntax in
Word Representations Linguistic Knowledge and Transferability of Contextual Representations What Does BERT Learn about the Structure of Language? Open Sesame: Getting Inside BERT's Linguistic Knowledge What Does BERT Look At? An Analysis of BERT's Attention Revealing the Dark Secrets of BERT |
|
12 | 11/21 | Sentence Embedding
With Transformers |
Sentence-BERT: Sentence Embeddings using
Siamese-Networks |
|
13 | 11/28 | Transformer-based
Applications Semantic Search with Transformers
Similarity
Search with FAISS Question-Answering
with Transformers |
|
Group Projects and Presentations |
14 | 12/5 | Transformers by Huggingface
For Korean Naver Sentiment Movie Corpus KorNLI KorQuAD KoreanNERCorpus Naver NLP Challenge NER NLP Challenge SRL |
|
Group Projects and Presentations |
15 | 12/12 | Final Test and Project Presentations |