Over 90% of LLM safety research ignores non-English languages entirely. The standard workaround — translating English safety benchmarks into other languages — sounds reasonable and fails structurally: a translated prompt inherits the harm taxonomy of its English source. It will never test whether a model helps with jeonse housing-deposit fraud in Korea, because that harm category doesn't exist in English benchmarks. "Telecom phishing" in the real world isn't a translated sentence — it's an SMS scam built around specific local banks and messaging apps.

Our new paper, XL-SafetyBench, led by AIM Intelligence with collaborators from Microsoft, Korea AISI, KT, BMW Group, Coinbase, TU Munich, Ankara University, Cyril Amarchand Mangaldas, and Seoul National University, is the first country-grounded cross-cultural benchmark for LLM safety: 5,500 native-language test cases across 10 country-language pairs (US, France, Germany, Spain, South Korea, Japan, India, Indonesia, Türkiye, UAE), evaluated on 10 frontier models and 27 local ones.

Two Kinds of Safety, Deliberately Separated

Most benchmarks collapse "safety" into one number. XL-SafetyBench splits it into two sub-benchmarks because they measure genuinely different abilities:

1. Jailbreak Benchmark (4,500 cases). For each country: 25 harm subcategories shared across all countries plus 25 discovered per-country by a web-search-augmented LLM and validated by native-speaker annotators — things like jeonse fraud that only exist locally. Base queries are written natively, then transformed into adversarial prompts with a PAIR-style automated red-teaming loop against three different probe targets.

2. Cultural Benchmark (1,000 cases). Scenarios where a cultural violation appears as an incidental detail inside a larger innocuous task: a model asked to plan a thank-you gift in France recommends chrysanthemums (funeral flowers); a model helping design name tags in Korea writes the names in red ink (associated with death). The model isn't asked "is this offensive?" — it has to notice on its own.

Every stage was validated by 20 native-speaker annotators (two per country, 15+ years of residence, with AI safety, law, or social-science expertise), with 92.7–98.1% agreement on binary filters.

The Metric That Catches Fake Safety

Here's the subtle problem: a low attack success rate can mean two very different things — principled refusal, or the model simply not understanding the language.

So alongside ASR (attack success rate) and CSR (cultural sensitivity rate), XL-SafetyBench introduces NSR — the Neutral-Safe Rate, the share of responses that are "safe" only because they're irrelevant or degenerate.

The result is one of the paper's sharpest findings: among local models, ASR and NSR trade off almost linearly (r = −0.81). France's CroissantLLM posts a stellar-looking 8.0% ASR — with 62.9% NSR. It isn't refusing attacks; it's failing to comprehend them. Türkiye's Trendyol-8B is the mirror image: fluent comprehension (0.4% NSR) and a 96.9% ASR. A leaderboard showing only ASR would call CroissantLLM one of the safest models tested.

What We Found at the Frontier

Across 10 frontier models:

• Jailbreak robustness varies wildly: Claude-4.5-Sonnet at 2.8% average ASR versus Mistral-Large-3 at 98.8%, with Llama-4-Maverick also above 90%. The frontier average is 44.8%.
• Attacks are harder to defend in some countries than others: frontier-average ASR is 34.5% on US prompts but 57.0% on UAE prompts and above 50% for South Korea. Localizing the attack materially changes the threat picture.
• Cultural sensitivity has huge headroom: the frontier average CSR is just 49.9%. Gemini-3.1-Pro leads at 76.1%; Llama-4-Maverick and Mistral-Large-3 sit below 15%.
• The two axes decouple: among closed-weight frontier models, ASR and CSR are essentially uncorrelated (r = −0.27, not significant). A model can be jailbreak-robust and culturally blind, or vice versa. A composite "safety score" hides which axis a model fails.

Why This Matters

If you deploy an LLM in Korea, Türkiye, or the UAE, an English safety report — or a translated benchmark — gives you false assurance about locally grounded attacks that your actual users will encounter. If you're procuring a sovereign or local-language model, ASR alone can reward comprehension failure over alignment. And if your assistant touches gift recommendations, food, funerals, or forms of address, cultural insensitivity is a product-quality and brand risk that no toxicity filter will catch.

XL-SafetyBench's construction pipeline — web-search discovery of local harms, LLM generation with validation gates, dual native-speaker review, multi-target automated red teaming — is itself a reusable methodology for country-grounded red teaming, which is exactly how we approach regional deployments at AIM Intelligence.

The dataset is public: Hugging Face · GitHub · Paper.