COFFEQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answering

Knowledge distillation aims to train a smaller student model to mimic a larger teacher model for resource-constrained environments. However, the process of distillation is typically data-intensive and computationally expensive, particularly for generative tasks.