10 Rules for LLM Evaluation: What we learned after a year of building

1. Eval metrics unlock 10x faster iteration than eyeballing responses.

2. Other than human evaluators, LLMs are the most effective evaluators for most tasks.

3. LLM graded evaluators are NOT perfect (but they can be quite good).

4. The key is to break down evaluations into very simple subtasks - ambiguity, unclear instructions, or complex tasks will lead to poor eval performance.

5. Use evals to measure your retrievals as well, not just your responses.

6. LLM graded evals can serve as effective quality metrics, but rarely as guardrails.

7. Enforce a consistent output format.

8. Use GPT-4 for complex eval tasks. Switch to cheaper models for simpler tasks

9. Run the same evaluations in development and production.

10. Your eval metrics should be closely track your product metrics

Why do you need evals?

• Iteration: Eval metrics unlock 10x faster iteration cycles than eyeballing responses

• Production: Understand model performance in production - production queries are not always the same as the queries in your development dataset.

Why do LLM evals work?

“Why would LLM evaluation work if my own inference failed?”

• your application’s prompt might be: “Answer this question: {question} based on this information: {context}”

• But your evaluation prompt might be something like: “Can you infer this response {response} from this context {context}. Answer in Y/N with a reason”

How to think about LLM evals

• LLM evals are not perfect. But they are the state-of-the-art for many use cases and the closest thing we have to human evaluators.

• LLM evals can serve as effective quality metrics, but not as guardrailsFor most applications (especially chat apps), the latency of LLM evals (a few seconds) is too high to run in real time. So you need to think of these evals as quality metrics, not as real-time guardrails.If latency is not an issue, then theoretically you could run them in real time, but keep in mind that LLM evals will never be perfect, so you’d be adding an additional layer of non-determinism into your product.

• LLM eval metrics are especially good for comparisons:The eval might mess up from time to time, but the overall metrics will give you a very good signal of how things are changing over time, and an objective way to compare performance across different models / prompts / retrieval strategies.

Rules for effective, high-performing evaluators

1. Break the evaluations down into simple subtasks: The simpler the subtask, the more effective the evaluator. The most effective ones will break down the task into very small, discrete subtasks (something that's very hard for an LLM or a 5th grader to get wrong).Example: Suppose you want to measure groundedness (or faithfulness) of a response to the context.For example: instead of providing it the entire response and context at once, you can pass it one sentence at a time and ask it to determine if that particular sentence has evidence in the context.Then Groundedness score = # of sentences with context / total # of sentencesHere’s a link to an evaluator that uses this approach. Ragas metrics are particularly good at this.

2. Enforce a consistent output format: Force the LLM to return a structured output (ex: JSON). This will create standardized eval results that are much easier to compare.Try OpenAI JSON mode, or Marvin, (or just ask for JSON and write a simple extract function).

3. Use chain-of-thought prompting wherever reasoning is required. This is an easy win to improve evaluation performance.

4. Measure your retrievals as well, not just your prompts and responses Bad retrieval ⇒ bad outputMake sure to set up some evals that measures your retrieval accuracy. For example: Athina’s Context Sufficiency eval, or Ragas’ Context Relevancy eval.

5. Use GPT-4 for complex tasks. Use cheaper models for simpler tasks. There might be a slight bias in the eval from using the same model, but since the evaluation task is fundamentally different from your main inference prompt, it’s still fine to use the same model.

6. Consider multi-step evals Just like your LLM inference pipeline has multiple steps, an effective eval might also require multiple steps to draw a meaningful conclusion.For example, for a Summary Accuracy eval metric, here’s a multi-step eval.

7. Step 1: Generate a list of closed-ended (Y/N) question and answer pairs from the summaryStep 2: Ask an LLM to answer these questions based ONLY on the source documentThen compare the responses. You can read more about this technique here.

8. Run evals in production To help you understanding how your model performs in the wild, run evals in the wild.You don’t need to run evals on every production log - running on a representative sample would still give you good indicators of model performance and response quality.

9. Use the same evals in development AND production This helps you identify how your production data is different from your dev dataset, and make better improvements to your model and dataset.

10. Start with preset evals. Then add your own custom evals. Preset evals / metrics are great to set up quickly and detect common issues in RAG applications.Ultimately, evals should be as closely aligned to your quality criteria as possible. This usually requires you to write your own custom evals.Using a custom eval is likely the best way to tailor your eval to work perfectly for your use case.

11. Your eval metrics should closely track your product success metrics Eval metrics are best if they can directly help you improve your product.Pick evals that are closely related to your product success metrics and – if possible, join eval results with your product analytics tool to get the complete picture.