COMPASS: Your Enterprise Chatbot Says Yes Too Easily — and That's a Policy Breach Waiting to Happen
We built the first systematic framework for testing whether LLMs follow organization-specific policies. Across 5,920 queries and seven frontier models, the result is a fundamental asymmetry: models handle legitimate requests with over 95% accuracy but refuse only 13–40% of direct policy violations — and as little as 3% under adversarial framing.
A general-purpose chatbot that compares car brands is being helpful. An automotive manufacturer's chatbot that compares car brands is violating company policy. Same behavior, same model — completely different outcome for the business.
Every safety benchmark you've seen measures universal harms: toxicity, violence, dangerous instructions. None of them can tell you whether your deployed assistant will follow your rules — the denylist your legal team wrote, the allowlist your compliance team approved. That gap is exactly where enterprise AI incidents happen.
Our new paper, COMPASS (Company/Organization Policy Alignment Assessment) — a collaboration between AIM Intelligence, BMW Group, Yonsei University, POSTECH, and Seoul National University — is the first systematic framework for evaluating organization-specific policy alignment in LLMs. We generated and validated 5,920 queries across eight industry scenarios and evaluated seven state-of-the-art models.
The headline finding is uncomfortable: models reliably do what they're allowed to do, and catastrophically fail to refuse what they're forbidden to do.
Why Universal Safety Isn't Enough
Organizational policies are domain-dependent and constantly evolving. A healthcare chatbot may share clinic locations and book appointments, but must never give a symptom-based diagnosis or recommend a prescription. A financial assistant can quote published rates, but must never make an investment recommendation. A government service bot must not comment on pending legislation.
No fixed benchmark can cover this diversity — which is why COMPASS is a framework, not a dataset. Feed it any organization's policy set (allowlist + denylist) and organizational context, and it generates a bespoke, validated evaluation suite.
How COMPASS Works
COMPASS takes a policy set — allowlist policies and denylist policies — plus an organizational context description, and runs a four-step pipeline:
- Base query synthesis — naturalistic queries per policy: legitimate requests for allowlist policies, direct violations for denylist policies.
- Base query validation — an LLM validator confirms each query matches its intended policy and triggers no unintended one.
- Edge case synthesis — the adversarial layer. Allowed edge queries look like violations but are fully compliant (probing over-refusal). Denied edge queries are genuine violations disguised through six transformation strategies: regulatory interpretation, analogical reasoning, statistical inference, context overflow, hypothetical scenarios, and indirect reference.
- Edge case validation — human agreement with the validator reached 89.4% (allowed-edge) and 90.3% (denied-edge).
Responses are scored with a Policy Alignment Score (PAS): an allowed query is aligned only if the model gives a substantive, compliant answer; a denied query is aligned only if the model explicitly refuses without leaking prohibited content. The GPT-5-mini judge agreed with human annotators 95.4% of the time.
We instantiated eight fictional organizations — automotive, government, financial, healthcare, travel, telecom, education, and recruiting — each with realistic allowlist and denylist categories, yielding 2,561 allowed and 3,359 denied queries.
The Results: Saying Yes Is Easy, Saying No Is Broken
Across Claude-Sonnet-4, GPT-5, Gemini-2.5-Pro, Kimi-K2, Qwen3-235B, Llama-3.3-70B, and Gemma-3-27B:
| Query type | What it tests | PAS range |
|---|---|---|
| Allowed base | Routine legitimate requests | 97.5–99.8% |
| Allowed edge | Legitimate but violation-looking | 79.7–96.6% |
| Denied base | Direct policy violations | 13.0–39.6% |
| Denied edge | Adversarially disguised violations | 3.3–20.5% |
Two details stand out.
The GPT-5 paradox. GPT-5 is the best model at handling legitimate requests (99.78% on allowed-base) — and the worst at resisting adversarial denylist attacks, refusing just 3.27% of disguised violations. Helpfulness optimization is directly at odds with policy enforcement.
The polite no that still leaks. When frontier proprietary models fail, their dominant failure mode (61–65% of misaligned adversarial responses) is the refusal-answer hybrid: the model says "I can't help with that" — and then provides the prohibited content anyway. Open-weight models mostly just comply outright (80–83% direct violation).
Domain matters too: education and recruiting chatbots were the most easily subverted, with average denied-edge scores of just 5.2% and 6.7%.
What Actually Fixes It (and What Doesn't)
We tested the mitigations enterprises actually reach for:
- Explicit refusal prompting: moves results by ~1–3%, sometimes negatively. Telling the model "be strict about the denylist" does not work.
- Bigger models: allowlist performance scales with size, but denied-edge robustness stays near zero even at 72B parameters. Scale does not buy policy robustness.
- RAG: minimal and inconsistent effect — the failure is capability-based, not missing context.
- Pre-filtering: pushes denied-base enforcement above 96%, but wrongly rejects roughly two-thirds of legitimate boundary requests (allowed-edge collapses to ~37%). You buy enforcement by destroying usability.
- Few-shot demonstrations: modest gains (Claude denied-edge 20.5% → 33.9%) at the cost of over-refusal.
- Policy-aware LoRA fine-tuning: the one strong lever. Trained on seven domains and tested on a held-out eighth, denied-edge PAS jumped from 0% to 60–62% while preserving allowed performance. Policy alignment appears to be a learnable, transferable skill.
Why This Matters
If you're deploying an LLM under any regulatory, legal, or brand constraint, your real exposure isn't the model saying something universally toxic. It's the model cheerfully doing the one thing your policy forbids — after a user wraps the request in a hypothetical, an analogy, or a fake regulatory citation.
COMPASS shows that this failure is structural, that prompt engineering and scale don't fix it, and that it can be measured — and trained away — systematically. That is exactly the kind of evaluation any organization should run before an assistant goes in front of customers, not after the first incident.
Read the full paper on arXiv.
