How To Debug a Failing RAG System or AI Agent in an Interview

Quick summary

Summarize this blog with AI

An interviewer who says, “The RAG assistant returns wrong answers” or “The agent sometimes fails” is giving you a debugging exercise, not a greenfield design prompt. Show that you can define the failure, protect users, and isolate the earliest broken layer without randomly changing prompts or models.

A strong answer traces the request from input to outcome and finds the first point where actual behavior diverges from expected behavior; later failures may only be symptoms. Review retrieval-augmented generation if needed, but keep your interview answer on diagnosis.

Start by turning the complaint into an incident

“Bad answer” is not yet a failure definition. Ask for one concrete request, the expected result, the actual result, the trace ID, the time window, and the affected user or tenant. Then clarify the failure class: unsupported claim, missing evidence, stale source, wrong tool, invalid arguments, repeated action, excessive latency, or permission breach. Establish frequency and blast radius by slicing failures by model version, prompt version, document version, language, tenant, tool, and release.

“I’ll first define the failure in observable terms and contain any unsafe actions. Then I’ll reproduce one case, inspect its end-to-end trace, locate the earliest divergence, and run the smallest experiment that separates my leading hypotheses. I’ll fix that layer, replay the failure set, and canary the change with regression monitoring.”

If the agent can send money, delete data, or contact customers, containment comes before analysis. Pause the risky action, require approval, or route to a safe deterministic fallback. Preserve traces and version metadata before rollback so that the evidence is not lost. Logs should be access-controlled and redact secrets and sensitive user data.

Use the same four questions at every layer

For each layer, state a symptom, choose a measurement, form a falsifiable hypothesis, and run one controlled experiment. This structure keeps an interview answer specific and prevents “try a better model” from becoming the entire plan.

Layer	Symptom	Measurement	Hypothesis	Experiment
Input and scope	Only some languages, tenants, or long conversations fail.	Failure rate by slice; normalized query and session state.	Query rewriting drops a constraint, or stale conversation state changes intent.	Replay the same request with rewriting or prior-turn context disabled.
Ingestion and index	The answer exists in the source but cannot be retrieved.	Extracted text, chunk boundaries, metadata, index alias, and document version.	Parsing lost a table, chunking separated a qualifier, or the new index is incomplete.	Search the raw extracted corpus for a known answer and compare old and new index snapshots.
Retrieval and ranking	Relevant evidence exists but does not reach the prompt.	Recall at k, candidate ranks, filter drop-off, reranker scores, and duplicate rate.	A metadata filter excludes the document, or the reranker favors topical but non-answer-bearing text.	Bypass filters or reranking one at a time and force the known-good document into candidates.
Context assembly	The right candidate was retrieved but omitted, truncated, or mislabeled.	Exact final prompt, token budget, passage order, citations, and provenance IDs.	Context packing drops the decisive passage or mixes source ownership.	Insert the gold passage in a fixed position and compare the generated answer.
Model and prompt	Correct evidence is present, but the answer contradicts or ignores it.	Claim-to-source support, abstention behavior, prompt/model versions, and decoding settings.	The instruction hierarchy is ambiguous, or the model follows prior knowledge over evidence.	Run the identical saved context through the previous prompt or model and through a minimal grounded prompt.
Planning, routing, and memory	The agent chooses the wrong tool, skips a step, loops, or follows stale memory.	Step trace, plan revisions, router decision, state transitions, and stop reason.	The planner created a bad subgoal, the router saw incomplete state, or memory outranked fresh evidence.	Replay with a fixed plan, memory disabled, or a forced route while keeping tool outputs constant.
Tool execution and recovery	Arguments are invalid, results are misread, or an action happens twice.	Validated arguments, tool response, timeout timing, retry count, idempotency key, and side-effect ledger.	A schema mismatch corrupts arguments, or an ambiguous timeout triggers an unsafe retry.	Stub a known tool response and fault-inject a timeout after the server commits the action.
Output and runtime	The trace is correct but the UI is wrong, or failures correlate with load.	Post-processing output, streaming events, cache keys, queue time, per-stage latency, and error rates.	A citation mapper uses the wrong ID, a cache leaks stale state, or a deadline cuts off a required step.	Compare the raw service response with rendered output and replay under controlled latency.

Debug RAG by following the evidence

For a failing RAG request, ask where the required evidence disappears. Search the extracted corpus first. If it is absent, investigate parsing, OCR, chunking, freshness, and index publication. If it is present but not in the candidate set, inspect query transformation, access filters, embedding or lexical retrieval, and index selection. If it appears in candidates but loses during reranking, inspect rank features and candidate diversity. If it reaches the final context but the answer is unsupported, the likely boundary has moved to context instructions, generation, or post-processing.

Use metrics that match the suspected layer. With labeled relevant passages, recall at k tells you whether retrieval surfaced evidence; reciprocal rank or nDCG describes where useful evidence appears. Context precision reveals how much distractor material was packed. Answer-level correctness does not prove retrieval quality, and a similarity score is not calibrated confidence that the evidence supports the claim.

The highest-value experiment is often an oracle-context test: give the current generator the known-good passage. If it still fails, retrieval is not the primary blocker. If it succeeds, walk backward through packing, reranking, filters, and ingestion. Also replay the exact saved context against the previous model and prompt. Change one variable at a time.

“The policy is present in the corpus, so I would not swap embedding models yet. I’ll verify whether it survives tenant and effective-date filters, inspect its candidate and reranker ranks, and then inject that passage into the current final prompt. Those checks distinguish an index/filter problem from ranking and generation problems.”

Debug an agent by following the trajectory

An agent can produce a plausible final response after taking a dangerous path, or a poor response after every tool behaved correctly. Evaluate the trajectory: input, plan, selected tool, validated arguments, tool result, state update, next decision, retry, stop condition, and user-visible outcome. The agentic AI systems patterns matter here because planning, orchestration, memory, and tools have different failure boundaries.

If the first plan is wrong before any tool runs, test planning instructions and task decomposition.
If the plan is correct but routing is wrong, replay with the intended route and inspect what state the router received.
If tool selection is correct but arguments are invalid, validate the schema boundary and compare generated arguments with the user’s request.
If the tool result is correct but later state is wrong, inspect parsing and the state transition rather than the tool.
If an action repeats after a timeout, treat the outcome as unknown until reconciled. Use a stable operation-level idempotency key; do not create a new key for each retry.
If the agent loops, inspect repeated state, progress signals, retry budgets, and stop reasons. A larger model is not a loop-control mechanism.

“For the duplicate action, I’ll correlate the orchestration trace with the downstream transaction ledger. If the server committed before the client timed out, the retry policy—not the planner—is the failure. I’ll reproduce that boundary with fault injection, then verify reconciliation and idempotency under repeated delivery.”

Make hypotheses compete

State two or three ranked hypotheses and the cheapest discriminating test for each. Suppose the assistant cites a stale policy. Plausible causes include an index alias still pointing to an old snapshot, an effective-date filter being lost during query rewriting, or agent memory overriding fresh retrieval. Inspecting version metadata and candidates separates the first two; replaying without memory tests the third. “Tune the prompt” does not distinguish any of them.

A reusable interview talk track

Define: State expected versus actual behavior and choose a severity-sensitive metric.
Contain: Disable or gate irreversible actions and preserve diagnostic evidence.
Scope: Measure onset, frequency, blast radius, and affected slices.
Trace: Follow one request end to end and identify the earliest divergence.
Test: Rank hypotheses and run controlled replays, ablations, oracle inputs, or fault injection.
Fix: Change the responsible layer, including its failure handling—not merely the visible symptom.
Verify: Replay incidents, run regression slices, canary the release, and monitor both quality and safety.

Connect the last step to an AI evals, observability, and reliability loop. A production incident should become a small, durable regression set with the responsible layer labeled. Add deterministic checks where possible, such as permission filters, schema validation, citation IDs, idempotency, and tool invariants; reserve model graders and human review for judgments that genuinely require semantics.

Common mistakes interviewers notice

Prompt-first debugging: changing wording before proving the prompt is the first failing boundary.
Shotgun changes: replacing the model, retriever, chunk size, and top k together, making the result impossible to attribute.
Average-only analysis: hiding severe failures in a language, tenant, document type, or long-tail workflow.
Final-answer-only logging: omitting candidate sets, final context, tool arguments, state transitions, retries, and version IDs.
Unsafe replay: debugging against live side-effecting tools instead of a sandbox or recorded stubs.
Stopping at the fix: failing to add regression coverage, fault tests, canary criteria, and a rollback threshold.

Practice drill: two failures after one release

Scenario: A customer-support agent now produces an incorrect refund-policy summary in 8% of sampled conversations, duplicates refund actions in 2%, and has a 40% increase in p95 latency. The release changed the query rewriter, model, and timeout retry logic.

Round 1: triage

Say what you would do before asking for more evidence. A strong response pauses automatic refunds or requires approval, preserves traces, separates answer-quality incidents from action-integrity incidents, and slices both by release version, jurisdiction, policy version, tool, and retry count. Do not assume one root cause just because the symptoms started together.

Round 2: evidence reveal

Wrong summaries occur only for jurisdiction-specific questions. The rewriter removes the jurisdiction phrase, and retrieval ranks a global but superseded policy first.
Duplicate refunds show a successful downstream transaction followed by a client timeout. The orchestrator retries with a new idempotency key.
The latency increase is concentrated in requests that retry; ordinary single-pass generation latency is unchanged.

Round 3: diagnosis and validation

Identify two root causes. For summary quality, preserve structured jurisdiction constraints through rewriting, enforce validity metadata in retrieval, and add version-sensitive retrieval cases. For duplicate actions, derive a stable key from the logical refund operation, record its status, and reconcile an ambiguous timeout before retrying. The latency symptom should improve when retries stop, but verify it rather than assuming.

Replay every affected trace against a sandbox. Run retrieval tests across jurisdictions and effective dates, plus fault injection where the refund service commits and the response is delayed or dropped. Then canary the change with thresholds for stale-policy retrieval, unsupported claims, duplicate-operation rate, retry rate, and p95 latency. Roll back or re-enable approval if any safety threshold regresses.

Final diagnostic checklist

Can I state one expected result and one observed result?
Did I contain irreversible or unauthorized behavior first?
Do traces include the exact model, prompt, index, schema, and workflow versions?
Where is the earliest divergence from a known-good path?
What measurement belongs to that layer?
What experiment would falsify my leading hypothesis?
Am I changing only one independent variable?
Did I test edge slices, retries, timeouts, stale data, and permission boundaries?
Did I verify the raw outcome and the user-visible result?
Will this incident become regression coverage and a monitored release gate?

The best interview answer is not the one that names the most AI techniques. It is the one that makes the system explainable: define the failure, protect users, trace the first divergence, run a discriminating experiment, repair the correct boundary, and prove the failure stays fixed.