An AI response includes a factual claim you cannot verify from the supplied sources. What would you do?

Instruction: Apply guideline version 3 to each claim in the fictional packet. Return source-linked evidence statuses, the grounding acceptance decision and a rationale of no more than two sentences.

Context:

Classify four claims against a source packet, separating contradiction, absent evidence and partial support. Produce inspectable evidence records and a concise grounding rationale.

Fictional practice task

Evaluate this four-claim summary of the Tarin Pilot Grant using only the approved packet. The fictional project uses guideline version 3. External research is not allowed, and the excerpts do not establish facts they omit.

Approved source packet:

  • S1: Organizations with 5–50 employees, inclusive, meet the grant’s staff-size eligibility condition. This excerpt establishes only staff-size eligibility.
  • S2: The maximum grant is $2,000.
  • S3: Equipment purchases are permitted expenses. This excerpt provides no information about travel expenses or application review times.

AI response, split into claims:

  • C1: An organization with 12 employees meets the staff-size eligibility condition.
  • C2: The maximum grant is $3,000.
  • C3: Applications are reviewed within seven days.
  • C4: The grant can pay for equipment and travel expenses.

Allowed evidence_status labels:

  • supported: every factual part is established by the packet, with no contradiction.
  • contradicted: at least one factual part is explicitly incompatible with the packet. This takes precedence over partial or absent support.
  • absent: no factual part is supported or contradicted by the packet.
  • partial: at least one factual part is supported and another is unaddressed, with no contradiction.

Grounding acceptance rule: Mark the response acceptable only if every claim is supported; otherwise mark it not_acceptable. This is a grounding judgment. An absent claim is not thereby proven false.

Produce: One record per claim containing its ID, evidence status, decisive source IDs and a short reason. For absent evidence, use an empty decisive-source list and name the sources checked. For a partial claim, identify the supported and unaddressed parts. Then give the overall rating and a submitted rationale of no more than two sentences.

Updated

Prepare a stronger answer

I’d mark C1 supported: an organization with twelve employees falls within S1’s five-to-fifty range. C2 is contradicted because S2 gives a two-thousand-dollar cap, not three thousand...

This member answer includes:

  • • A complete, copyable sample answer
  • • Guidance for adapting the answer to your experience
  • • A practical walkthrough
  • • Common mistakes and how to avoid them
  • • Answered interviewer follow-ups
  • • Strong, adequate and weak assessment criteria
Unlock the full answer and preparation guide

One payment for one year of full access. No automatic renewal.

See pricing and everything included

Your preparation path

Choose the track that matches the role. Work through its questions in order, then explain each answer in your own words.

1. Entry level annotation

Apply guidelines, label text and spans, and explain a small practice project.

  1. How would you explain the data annotator role and the kind of work you would expect to do? Free sample
  2. How would you learn a new annotation guideline before starting your first batch? Member answer
  3. Apply a sentiment guideline to four short comments. Which labels would you choose, and why? Free sample
  4. Mark two location mentions using the exact character-offset contract. How would you check your result? Free sample
  5. Walk me through an annotation or quality-checking project you can discuss, including your own contribution and limits. Member answer

2. AI response evaluation

Compare responses using separate criteria for correctness, instruction following, and writing quality.

  1. Compare two AI responses against a supplied fact sheet. Which response is better under the rubric? Free sample
  2. One response is accurate but breaks the required format; another follows the format but contains a false claim. How would you rate them? Member answer
  3. Write a short rating rationale that identifies the decisive error without restating both responses. Member answer
  4. An AI response includes a factual claim you cannot verify from the supplied sources. What would you do? Member answer
  5. Two responses have different strengths and neither clearly wins. How would you apply the ranking rules? Member answer

3. Senior review and quality

Work through disagreement, missed critical cases, changing guidelines, review capacity, and reviewer calibration.

  1. Two experienced reviewers disagree repeatedly, and the deadline leaves little time for adjudication. What would you recommend? Free sample
  2. A batch has 98% accuracy against reviewed references but misses every critical item. Would you accept it? Member answer
  3. A labeling rule changes halfway through a delivery. Would you relabel old work, split the dataset or delay the release? Member answer
  4. The delivery requires review of every item, but the available reviewers cannot finish by the deadline. What would you change? Member answer
  5. A reference answer appears to contradict the written rule, and workers are being penalized for disagreeing with it. What would you do? Member answer

Try the 20 minute mock assessment. Use the fictional cases to practice; the self-check is not an employer's hiring benchmark.

Related Questions