SFT-as-Context Mitigates Forgetting in Supervised Fine-Tuning
Supervised fine-tuning (SFT) equips large language models (LLMs) with specialized capabilities, but often comes at the cost of forgetting the general capabilities of their parent models (i.e., the pretrained models before fine-tuning).
arXiv - cs.CL (NLP)2 min read