Post Hoc Explanations of Language Models Can Improve Language Models

Large Language Models (LLMs) have demonstrated remarkable capabilities in performing complex tasks. Moreover, recent research has shown that incorporating human-annotated rationales (e.g., Chain-of-Thought prompting) during in-context learning can significantly enhance the performance of these models, particularly on tasks that require reasoning capabilities. However, incorporating such rationales poses challenges in terms of scalability as this requires a high degree of human involvement. In this work, we present a novel framework, Amplifying Model Performance by Leveraging In-Context Learning with Post Hoc Explanations (AMPLIFY), which addresses the aforementioned challenges by automating the process of rationale generation. To this end, we leverage post hoc explanation methods which output attribution scores (explanations) capturing the influence of each of the input features on model predictions. More specifically, we construct automated natural language rationales that embed insights from post hoc explanations to provide corrective signals to LLMs. Extensive experimentation with real-world datasets demonstrates that our framework, AMPLIFY, leads to prediction accuracy improvements of about 10-25% over a wide range of tasks, including those where prior approaches which rely on human-annotated rationales such as Chain-of-Thought prompting fall short. Our work makes one of the first attempts at highlighting the potential of post hoc explanations as valuable tools for enhancing the effectiveness of LLMs. Furthermore, we conduct additional empirical analyses and ablation studies to demonstrate the impact of each of the components of AMPLIFY, which, in turn, leads to critical insights for refining in-context learning.

Summary Notes

Improving Language Models with Automated Rationales: The AMPLIFY Framework

Overview of AMPLIFY

• Choosing a Proxy Model: It starts with a smaller model like GPT-2 or BERT to make generating explanations feasible.

• Identifying Misclassified Samples: It then focuses on samples that LLMs misclassify, especially those the proxy model is confident it has misclassified.

• Automating Rationale Generation: By applying explainability techniques, AMPLIFY creates natural language rationales.

• Creating Few-shot Prompts: These automated rationales are then used to construct prompts that significantly boost the LLM’s task performance.

Testing AMPLIFY

• Proxy Model Choice: The selection is crucial, though fine-tuning the proxy model offered minimal benefits.

• Strategy for Sample Selection: Selecting samples based on a High Misclassification Confidence Score was most effective.

• Methods for Generating Explanations: Techniques like Gradient x Input and its contrastive version were most useful.

• Tailoring Rationale Templates: Customizing templates for specific tasks slightly improved outcomes.

Key Takeaways

• AMPLIFY outperforms traditional prompting methods, underscoring the value of automated rationales.

• The default capabilities of proxy models are generally adequate, with fine-tuning offering little extra benefit.

• Focusing on samples where the proxy model is highly confident in its misclassification is a winning strategy.

• The choice of explanation method and the customization of rationale templates are important for maximizing performance.

Conclusion

Acknowledgments and Funding

Appendices

• Appendix A: Offers a deep dive into proxy model performance and the effects of fine-tuning.

• Appendix B: Discusses the limitations and broader impacts of LLMs and post hoc explanations.

• Appendix C: Explores further experiments with larger proxy models and additional tasks.

Reporting Errors

How Athina AI can help