108.535A: 컴퓨터언어학연구 I


Hyopil Shin (Dept. of Linguistics, Seoul National University)

hpshin@snu.ac.kr
http://knlp.snu.ac.kr/

Mon  1:00 to 3:45 in building  room

T.A: TBA

transformer

(http://www.theverge.com/2016/3/11/11208078/lee-se-dol-go-google-kasparov-jennings-ai)

Course Description

이 과목에서는 자연언어처리(Natural Language Processing) 또는 컴퓨터언어학(Computational Linguistics)의 이론적인 기초에서부터 최근의 Transformers, BERT 기반의 방법론을 학습한다.  강의 전반부에서는 N-gram, Entropy, Embedding에 관한 내용이 다루어지며 후반부에는  Encoder-Decoder, Attention, Transformer를 학습하고  Huggingface의 Transformers의 사전학습모델과 모듈을 사용하여 자연언어처리에 활용하는 다양한 태스크를 실제 구현해 보도록 한다. 프로그래밍으로 Pytorch가 다루어지며 모든 과제는 토치를 기반으로 구현하도록 한다. 파이선 및 딥러닝 기본 지식이 요구된다. 이 수업을 통해 자연언어처리의 기본개념에서부터 최근의 방법론까지 학습하여 실제 언어처리에 활용할 수 있는 능력을 키우도록 한다.

Updates

  • 9월 5일 첫수업은 수해로 침수된 강의실이 완벽히 복구되지 않아 부득이 줌으로 진행합니다. 추후에는 대면수업입니다.
  • 강의 줌 주소:  https://snu-ac-kr.zoom.us/j/92234443751

Useful Sites

  • Lectures


Textbook and Sites

speech and Language Processing 3rd
            Edition Drafts

                                                                                     

Speech and Language Processing (3rd ed. Draft)


huggingface transformers

Huggingface Transformers


DL wizard

Deep Learning Tutorials based on PyTorch

 

Syllabus


Date Topics Related Materials and Resources
PyTorch
1 9/5

Introduction to Natural Language Processing


Language Modeling 1- Statistical Language Modeling: N-Grams

Natural Language Processing is Fun!

Language Modeling and with N-Grams

PyTorch:
2 9/12 Language Modeling 1- Statistical Language Modeling: Entropy and Maximum Entropy Models

(Chuseok Holiday- Zoom Class)
Entropy is a Measure of Uncertainty
3 9/19
Text Classification
Text Classification  
4 9/26 Vector Semantics

Language Modeling II: Static Word Embedding


Vector Semantics and Embeddings PyTorch:

Linear Regression With PyTorch
Logistic Regression With PyTorch
5 10/3 Language Modeling II: Static Word Embedding

(Holiday, Zoom Class)
Vector Semantics and Embeddings


PyTorch:

Word Embeddings: Encoding Lexical Semantics

6 10/10

Sequence to Sequence Model: Encoder-Decoder





PyTorch:
  • pytorch-seq2seq
    • Sequence to Sequence Learning with Neural Networks
    • Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
    • Neural Machine Translation by Jointly Learning to Align and Translate
    • Packed Padded Sequences, Masking, Inference and BLEU
    • Convolutional Sequence to Sequence Learning
    • Attention is All You Need


A Comprehensive Introduction to Torchtext

Torchtext Github

7 10/17 Attention Model
Neural Machine Translation By Jointly Learning to Align and Translate

Attention: Illustrated Attention PyTorch:
  • pytorch-seq2seq
    • Sequence to Sequence Learning with Neural Networks
    • Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
    • Neural Machine Translation by Jointly Learning to Align and Translate
    • Packed Padded Sequences, Masking, Inference and BLEU
    • Convolutional Sequence to Sequence Learning
    • Attention is All You Need
8 10/24 Transformer
Self Attention: Attention is All you need

The Illustrated Transformer

 

PyTorch:
The Annotated Transformer

9
10/31 Language Modeling III: Dynamic Word Embedding : BERT (Bidirectional Encoder Representations from Transformers)

BERT Fine Tuning
BERT Fine-Tuning Tutorial with PyTorch

BERT Word Embeddings


10



11/7

Pre-trained Models and Transfer Learning




XLM-R: Unsupervised Cross-lingual Representation Learning at Scale

XLNet: Generalized Autoregressive Pretraining for Language Understanding

MASS: Masked Sequence to Sequence Pre-training for Language Generation

BART:Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

GLM: All NLP Tasks Are Generation Tasks: A General Pretraining Framework

SpanBERT: Improving Pre-training by Representing and Predicting Spans
11 11/14 Pre-trained Models: Interpretation and Theoretical Analysis
  • Knowledge of PTMs: Linguistic Knowledge


Transformers by Huggingface:

Quick Tour
Summary of Tasks : Sequence Classification, Extractive Question Answering, Language Modeling, Text Generation, Named Entity Recognition, Sumarization, and Translation


A Structural Probe for Finding Syntax in Word Representations

Linguistic Knowledge and Transferability of Contextual Representations

What Does BERT Learn about the Structure of Language?

Open Sesame: Getting Inside BERT's Linguistic Knowledge

What Does BERT Look At? An Analysis of BERT's Attention

Revealing the Dark Secrets of BERT
12 11/21 Sentence Embedding With Transformers

Sentence-BERT: Sentence Embeddings using Siamese-Networks

13 11/28 Transformer-based Applications

Semantic Search with Transformers
  • Introducing txtai, an AI-Powered Search engine on Transformers

Similarity Search with FAISS

Question-Answering with Transformers

 




Group Projects and Presentations
14 12/5 Transformers by Huggingface For Korean

Naver Sentiment Movie Corpus
KorNLI
KorQuAD
KoreanNERCorpus
Naver NLP Challenge NER
NLP Challenge SRL

Group Projects and Presentations
15 12/12 Final Test and Project Presentations