How would you build an evaluation set that represents real client work?

Instruction: Design the evaluation set, label process, and separation from development examples. Explain what you would report before a pilot.

Context:

A fictional procurement team wants an assistant to summarize supplier proposals. It processes routine renewals, unusual pricing terms, incomplete proposals, and documents in two supported languages. The development team has already tuned prompts using 60 example proposals. A sponsor offers 20 successful demonstrations as the launch evaluation.

Updated

Official answer available

Read the opening below, then unlock the full answer and practical guidance.

I would keep the demonstrations as examples, not treat them as an independent launch test. They were selected for success, and the 60 development proposals have already influenced the system...

Upgrade to view official answer

Related Questions