Evaluate a vendor’s AI benchmark and purchase proposal.

Instruction: Design a fair purchase evaluation and explain how you would reach a decision from its results.

Context:

A fictional vendor reports 95% accuracy on forty demonstration questions that it selected. Your client needs an internal policy assistant with document-level access controls and citations. The client can provide 300 representative, authorized test cases, including twenty scenarios where an unauthorized disclosure or unsupported high-impact instruction is unacceptable. Evaluation criteria and expected evidence can be reviewed by internal specialists. The vendor must not receive the hidden expected answers before the locked evaluation.

Updated

Official answer available

Read the opening below, then unlock the full answer and practical guidance.

The vendor's 95% does not establish performance in our workflow. I'd agree the client's evaluation criteria before testing: supported answers, useful citations, appropriate abstention, access-control behavior, latency, and total operating cost...

Related Questions