How would you evaluate whether a safety filter is overblocking?

Instruction: Describe how you would tell whether a guardrail is blocking too much legitimate work. Use the supplied reviewed sample to calculate the false-positive rate, false-negative rate, and fraction of blocked requests that were permitted. Explain what the sample cannot establish.

Context:

A reviewed evaluation sample has 900 permitted requests: 45 blocked and 855 allowed. It also has 100 prohibited requests: 20 allowed and 80 blocked. Treat the labels as correct for this exercise.

Updated

Example Answer

I would compare blocked cases against legitimate workflow goals, not just against abstract harmlessness. A filter is overblocking when it prevents the product from doing valid jobs too often or in the wrong places.

That means reviewing false positives by segment, language, workflow type, and business criticality. A filter that is acceptable on casual traffic can be unacceptable on enterprise workflows if it blocks common legitimate actions or content.

I also want user and operator behavior in the picture. Workarounds, support complaints, and manual overrides are often strong signals that the filter is too blunt.

Concrete example

Have reviewers label permitted and prohibited cases under the same policy. Track the share of truly permitted cases that the filter blocks, and inspect which blocked workflows users actually need to complete.

Limit to explain

Blocked traffic alone cannot reveal missed unsafe cases. Review allowed traffic too, and record the denominator and sampling method behind each rate.

Follow-up to practice

How would you detect underblocking while reducing false positives?

Worked practice case

The false-positive rate is 45 / 900 = 5% of truly permitted requests. The false-negative rate is 20 / 100 = 20% of truly prohibited requests. Of 125 blocked requests, 45 / 125 = 36% were permitted; that is a different denominator from the false-positive rate.

The sample has a 10% prohibited-request prevalence by construction. Its 36% blocked-request error fraction need not hold in production with a different traffic mix. Report sample sizes and uncertainty, inspect severity and workflow/language slices, and include review of allowed traffic so missed prohibited requests are observable. These aggregate numbers do not establish whether one severe failure is acceptable.

Your preparation path

Work through these questions in order. Read the answer aloud, then explain it in your own words.

1. Start with the foundations

Build the vocabulary and explain the core decisions.

  1. Why is prompt injection different from ordinary bad input? Free sample
  2. How do you separate policy enforcement from model style instructions? Free sample
  3. What information should never flow from untrusted content straight into tool execution? Free sample

2. Apply it to a real workflow

Practice diagnosis, validation, and everyday tradeoffs.

  1. How do you decide where to place guardrails in a multi-step agent workflow? Free sample
  2. A summarization workflow passes unsafe content from retrieval into a downstream action. How would you fix it? Member answer
  3. A reviewer approves a high-risk action because the model framed it confidently. How would you reduce that risk? Member answer

3. Prepare for senior discussions

Explain failure boundaries, recovery, and production choices.

  1. A multi-agent system routes around one guardrail because another agent has broader permissions. How would you fix it? Member answer
  2. OAuth discovery metadata points your agent at an internal URL. How would you prevent SSRF? Member answer
  3. Why can a localhost MCP server still be exposed to attacks from a website? Member answer

Related Questions