Penerapan Arsitektur BERT2BERT pada Model Respon Dialog Chatbot Berbahasa Indonesia dengan Dataset Berukuran Terbatas

Abstract

The efficiencies offered by chatbots have made them a popular solution to a variety of problems. For example, in the context of academic services, chatbots can save the resources needed to provide answers to routine questions, but are still difficult to implement due to lack of data. Popular architectures for text generation tasks include encoder-decoder and decoder-only. Using the encoderdecoder architecture, BERT2BERT leverages the BERT pre-trained model which is used to initialize the encoder and decoder components (Rothe et al., 2020). This architecture has been used in research to build conversational models in Arabic and is a good solution to the problem of lack of datasets (Naous et al., 2021). This research tries to use the BERT2BERT architecture to build an Indonesian chatbot response model with limited data and for academic service purposes. The result of this research is that the BERT2BERT architecture provides less performance with the perplexity score obtained is 573.0654, while the perplexity score of the GPT2 model is 61.2645. The cosine similarity score is also lower at 0.4009 compared to the GPT2 model with a cosine similarity score of 0.7037. However, the BERT2BERT model provides better results compared to a similar encoder-decoder model, BERT2GPT. Models with encoder-decoder architectures such as BERT2BERT did not excel in this study possibly due to the difficulty of obtaining context from data of limited size.