How would you evaluate an LLM’s reliability on logical reasoning tasks?
Instruction: Give a clear explanation, a practical example, and the main limitation.
Context: Evaluate reasoning on realistic variations and verify results with an independent, validated checker.
Updated
Official answer available
Read the opening below, then unlock the full answer and practical guidance.
I would evaluate the actual class of reasoning the product needs rather than infer ability from a fluent explanation. I would test unfamiliar cases, paraphrases, changed numbers, extra irrelevant information, and combinations of constraints...
Related Questions
-
easy
-
medium
-
medium
-
hard