Evaluating alignment techniques for enhancing LLM performance in a closed-domain application: a RAG bench-marking study

In the field of natural language processing (NLP), Large Language Models (LLMs) such as MISTRAL have been pre-trained on diverse tasks and datasets to achieve comprehensive language understanding. Nevertheless, fine-tuning small closed-domain datasets presents limitations, including overfitting, restricted generalization, data representation bias, and catastrophic forgetting issues. To tackle these challenges and align LLMs with ethical standards and specific requirements within closed-domain datasets, we apply a methodology that involves training four prominent LLMs - GEMMA, LLaMA, MISTRAL, and SOLAR - on a small closed-domain dataset using three distinct optimization techniques: Contrastive Preference Optimization (CPO), Direct Preference Optimization (DPO), and Kahneman-Tversky Optimization (KTO). We evaluate their proficiency by employing the Retrieval Augmented Generation (RAG) benchmark system to assess accuracy. Our research provides researchers with comprehensive insights into the effectiveness of these training methods in improving language model performance within closed-domain datasets while addressing the aforementioned challenges faced by LLMs in specific domains with limited datasets