Notes

Attention and Memory in Deep Learning (Alex Graves, DeepMind x UCL DL Lecture 8)

19 May 2022

세계 최고의 머신 러닝 학회인 NIPS 에 Attention is All you Need 라는 이름으로 Transformer Network Architecture가 공개된 지 5년이란 시간이 흘렀습니다. 이제는 NLP부터 CV 까지 안 쓰이는 곳이 없는 General Purpose Network 가 되었는데요, 본 post에서는 Deepmind의 Research Scientist 인 Alex Graves 가 Transformer 의 핵심 요소인 Attention 에 대한 본인의 직관을 담은 영상을 리뷰해보려고 합니다.

< 목차 >

Overview

ucl_deepmind_lecture_8_slide_4 Fig. Slide 4.

말씀드린 것 처럼 주제는 Attention 과 딥러닝 입니다.

내용을

ucl_deepmind_lecture_8_slide_3 Fig. Slide 3.

요약

ucl_deepmind_lecture_8_slide_67 Fig. Slide 67.

Introduction

tmp

ucl_deepmind_lecture_8_slide_5 Fig. Slide 5.

ucl_deepmind_lecture_8_slide_6 Fig. Slide 6.

ucl_deepmind_lecture_8_slide_7 Fig. Slide 7.

ucl_deepmind_lecture_8_slide_8 Fig. Slide 8.

ucl_deepmind_lecture_8_slide_9 Fig. Slide 9.

ucl_deepmind_lecture_8_slide_10 Fig. Slide 10.

ucl_deepmind_lecture_8_slide_11 Fig. Slide 11.

ucl_deepmind_lecture_8_slide_12 Fig. Slide 12.

ucl_deepmind_lecture_8_slide_13 Fig. Slide 13.

ucl_deepmind_lecture_8_slide_14 Fig. Slide 14.

ucl_deepmind_lecture_8_slide_15 Fig. Slide 15.

ucl_deepmind_lecture_8_slide_16 Fig. Slide 16.

ucl_deepmind_lecture_8_slide_17 Fig. Slide 17.

ucl_deepmind_lecture_8_slide_18 Fig. Slide 18.

ucl_deepmind_lecture_8_slide_19 Fig. Slide 19.

ucl_deepmind_lecture_8_slide_20 Fig. Slide 20.

ucl_deepmind_lecture_8_slide_21 Fig. Slide 21.

Soft Attention

ucl_deepmind_lecture_8_slide_22 Fig. Slide 22.

ucl_deepmind_lecture_8_slide_23 Fig. Slide 23.

ucl_deepmind_lecture_8_slide_24 Fig. Slide 24.

ucl_deepmind_lecture_8_slide_25 Fig. Slide 25.

ucl_deepmind_lecture_8_slide_26 Fig. Slide 26.

ucl_deepmind_lecture_8_slide_27 Fig. Slide 27.

ucl_deepmind_lecture_8_slide_28 Fig. Slide 28.

ucl_deepmind_lecture_8_slide_29 Fig. Slide 29.

ucl_deepmind_lecture_8_slide_30 Fig. Slide 30.

ucl_deepmind_lecture_8_slide_31 Fig. Slide 31.

ucl_deepmind_lecture_8_slide_32 Fig. Slide 32.

ucl_deepmind_lecture_8_slide_33 Fig. Slide 33.

ucl_deepmind_lecture_8_slide_34 Fig. Slide 34.

ucl_deepmind_lecture_8_slide_35 Fig. Slide 35.

ucl_deepmind_lecture_8_slide_36 Fig. Slide 36.

ucl_deepmind_lecture_8_slide_37 Fig. Slide 37.

Introspective Attention

ucl_deepmind_lecture_8_slide_38 Fig. Slide 38.

ucl_deepmind_lecture_8_slide_39 Fig. Slide 39.

ucl_deepmind_lecture_8_slide_40 Fig. Slide 40.

ucl_deepmind_lecture_8_slide_41 Fig. Slide 41.

ucl_deepmind_lecture_8_slide_42 Fig. Slide 42.

ucl_deepmind_lecture_8_slide_43 Fig. Slide 43.

ucl_deepmind_lecture_8_slide_44 Fig. Slide 44.

ucl_deepmind_lecture_8_slide_45 Fig. Slide 45.

ucl_deepmind_lecture_8_slide_46 Fig. Slide 46.

ucl_deepmind_lecture_8_slide_47 Fig. Slide 47.

ucl_deepmind_lecture_8_slide_48 Fig. Slide 48.

ucl_deepmind_lecture_8_slide_49 Fig. Slide 49.

ucl_deepmind_lecture_8_slide_40 Fig. Slide 50.

ucl_deepmind_lecture_8_slide_51 Fig. Slide 51.

ucl_deepmind_lecture_8_slide_52 Fig. Slide 52.

ucl_deepmind_lecture_8_slide_53 Fig. Slide 53.

ucl_deepmind_lecture_8_slide_54 Fig. Slide 54.

ucl_deepmind_lecture_8_slide_55 Fig. Slide 55.

Further Topics

ucl_deepmind_lecture_8_slide_56 Fig. Slide 56.

ucl_deepmind_lecture_8_slide_57 Fig. Slide 57.

ucl_deepmind_lecture_8_slide_58 Fig. Slide 58.

ucl_deepmind_lecture_8_slide_59 Fig. Slide 59.

ucl_deepmind_lecture_8_slide_60 Fig. Slide 60.

ucl_deepmind_lecture_8_slide_61 Fig. Slide 61.

ucl_deepmind_lecture_8_slide_62 Fig. Slide 62.

ucl_deepmind_lecture_8_slide_63 Fig. Slide 63.

ucl_deepmind_lecture_8_slide_64 Fig. Slide 64.

ucl_deepmind_lecture_8_slide_65 Fig. Slide 65.

ucl_deepmind_lecture_8_slide_66 Fig. Slide 66.

ucl_deepmind_lecture_8_slide_67 Fig. Slide 67.

Reference