Perbandingan Variasi Prompt terhadap Kualitas Puisi Menggunakan Model GPT-2 Fine-tuned

Abstract

Abstract and creative thinking are key characteristics of human intelligence that enable the creation of works of art such as poetry, which have complex linguistic structures and depths of meaning. The development of Natural Language Generation (NLG) has encouraged the use of generative language models such as GPT-2 in the generation of Indonesian-language poetry, although the quality, naturalness, and diversity of the results are still debatable, especially in Indonesian, which is classified as a low-resource language. This study aims to compare the influence of prompt variations (zero-shot, one-shot, and few-shot) and hyperparameter settings on the quality of Indonesian-language poetry generated by a fine-tuned GPT-2 model, as well as to determine the best combination of model and prompt based on the evaluation metrics used. The research method involved three GPT-2 models, namely GPT-2 Small Indonesia (pre-trained), GPT-2 Small Indonesia Fine-Tuning Poem (pre-trained), and GPT-2 Small Indonesia Fine-Tuning Poem, which had undergone a fine-tuning process using SFTTrainer and a poetry dataset from two books by Goenawan Mohamad. The evaluation was conducted using the perplexity metric as a measure of text naturalness and lexical diversity through distinct-1 and distinct-2. The results showed that the GPT-FT model achieved the lowest average perplexity of 1.907 on a one-shot prompt with specific hyperparameter configurations. The highest distinct unigram value of 0.9169 was obtained by GPT-2 Small Indonesia FineTuning Poem on a one-shot prompt, and the highest distinct bigram value of 0.9809 was achieved by GPT-2 Small Indonesia on a one-shot prompt. These findings indicate that fine-tuning and prompt variation have a significant effect on the naturalness and diversity of poetry. In future research, it is recommended to conduct human evaluation to comprehensively measure aesthetic quality.