Analisis Perbandingan Pengaruh Metode Praproses Data dan Fine-Tuning Terhadap Performa BERT dalam Klasifikasi Teks

Abstract

Transformer-based models, such as Bidirectional Encoder Representations from Transformers (BERT), have achieved significant advancements in natural language processing by understanding deeper connections between words and their context. This advancement has relied on the model’s architecture and its capacity to capture linguistic nuances, which are essential for tasks like text classification. Fine-tuning on domain-specific datasets has been crucial for optimal performance in text classification. Additionally, data preprocessing has significantly improved BERT’s performance by emphasizing relevant information. This research aimed to assess how data preprocessing and fine-tuning affect BERT’s performance in text classification. The study involved several stages: collecting benchmark datasets (SST-2, AG News, Enron Spam), designing preprocessing methods, fine-tuning BERT using domain-specific models, hyperparameter tuning, feature-based approaches, and classifier algorithms. The findings revealed that pre-processing improved performance for complex datasets while reducing it for simpler ones. Domain-specific BERT models outperformed base and large versions. The optimal hyperparameter configuration was found to be learning rate 2e-5 and batch size 32, and the last encoder layer provided the best feature representation to be fed to BERT’s classification head.