사전학습 언어모델의 기술 분석 연구


A Study on the Approaches of the Pre-trained Language Model

지인영, 김희동

피인용수 : 0(자료제공 : 네이버학술정보)



The pre-training language model, BERT has achieved a great success in natural language processing by transferring knowledge from rich-resource pre-training task to the low-resource downstream tasks. The model was recognized as a breakthrough or an innovative technology that changed the paradigm of natural language processing. In this paper, a number of studies have been analyzed to classify and compare the research directions. We examined a technical challenges after BERT. In the pre-training process, self-supervised learning is performed, which relies entirely on training data. If we introduce the linguistic knowledge in the course of the training, it would be possible to get better result more effectively. Therefore, it is necessary to develop a method to insert external knowledge such as linguistic information in the training process. The mask language model and the next sentence prediction are being used in BERT's pre-training tasks. Though, to get much deeper understanding of the natural language, some other effective methods are to be studied and developed. Lastly, we should aim to develop eXplainable Artificial Intelligence (XAI) technology in natural language processing, helping us look into the transparent processing. The pre-trained language model focuses on the development of skills that can be used for all the tasks of natural language understanding. A lot of researches are focused on how to adapt language model effectively to downstream tasks based on common language models, even with the case with little data. It is also hoped that the technical analysis reviewed in this study will provide linguists and computer researchers with an opportunity to understand recent technological achievements in the field of natural language processing and to seek joint research.


1. 서론
2. 언어모델의 종류
2.1 언어모델의 확률표현
2.2 통계적 N-gram 모델
2.3 단어의 분산표현(Distributive Representation) 언어모델
2.4 부호기 복호기(Encoder-Decoder) 방식의 언어 모델
2.5 주의 기구(Attention Mechanism)를 이용한 모델
3. 전이 학습(Transfer Learning)과 사전 학습(Pre-trained Learning)
3.1 전이 학습 (Transfer Learning)
3.2 자기지도 학습(self-supervised learning)
3.3 사전학습 언어모델
4. 사전학습 언어모델의 응용과 기술 분석
4.1 연구 동향
4.2 사전학습 언어모델에 대한 연구과제
5. 결론


  • 지인영 In-Youn Jhee. 한국체육대학교 교양교직학부, 교수
  • 김희동 Hee-Dong Kim. 한국외국어대학교 공과대학 정보통신공학과, 교수


자료제공 : 네이버학술정보

    함께 이용한 논문

      ※ 기관로그인 시 무료 이용이 가능합니다.

      • 6,000원

      0개의 논문이 장바구니에 담겼습니다.