Penyelarasan Large Language Model terhadap Respon Tidak Etis menggunakan Reinforcement Learning from Human Feedback (RLHF)

Abstract

LLM or Large Language Model have demonstrated remarkable capabilities in various natural language processing tasks, yet they continue to face challenges related to ethical aspects and safety of generated outputs. Supervised FineTuning (SFT), commonly used to address these issues, has limitations in terms of flexibility and generalization ability to new contexts. This research aims to implement Reinforcement Learning from Human Feedback (RLHF) method to improve the alignment of Indonesian large language models with human preferences and values. RLHF implementation was conducted through three stages: Supervised Fine-Tuning (SFT), Reward Model training, and policy optimization using Proximal Policy Optimization (PPO). Evaluation results show that the Reward Model achieved Pairwise Accuracy of 0.80 and Kendall Tau of 0.60 with good generalization capability without overfitting. The PPO-trained model achieved a Harmlessness value of 0.75 with 0% Refusal Rate, indicating that the model no longer provides explicit refusals as in previous research, but instead generates informative, contextual responses that guide users toward ethical and safe alternatives. SHAP visualization confirms the shift in response characteristics from explicit refusals with tokens such as “cannot answer” to informative guidance with tokens such as “alternative” and “legal platform”. Comparison with baseline model shows that the RLHF-trained model generates responses with higher Reward values in most evaluation scenarios. Nevertheless, the model still shows weaknesses in some cases, particularly in refusing requests related to sensitive personal information. Overall, RLHF implementation successfully improved model alignment with safety and helpfulness values without sacrificing constructive interaction quality. Keywords: Large Language Model, Reinforcement Learning from Human Feedback, RLHF, Proximal Policy Optimization, Reward Model, Alignment, Indonesian Language