Not All Needles Are Found: How Fact Distribution and Don't...

When context windows lie: distributional collapse and the safety tax in long-context LLMs

I work with teams building AI systems where correctness matters. A lot of failure modes I see come from assumptions about what a model will use from a long context. The paper "Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs" (arXiv:2601.02023) is a useful, practical look at one class of those failures. It does not promise to fix everything. What it does is make an operational problem visible and measurable, and that matters.

Technical summary

The authors build a model-agnostic benchmark they call an extended needle-in-a-haystack. The idea is simple but valuable: vary where facts live in a large corpus and measure three separate behaviors — literal extraction, logical inference, and hallucination risk — so you can see whether a model fails to find facts, fails to reason with facts it has, or invents answers out of thin air.

They test several closed-source models: Gemini-2.5-flash, ChatGPT-5-mini, Claude-4.5-haiku, and Deepseek-v3.2-chat. Their two headline failure modes are what they call Distributional Collapse and the Safety Tax. Distributional Collapse is when model performance drops sharply as evidence is dispersed across many documents rather than concentrated in one place. The Safety Tax is the opposite problem: anti-hallucination prompts, like "Don't Make It Up," reduce hallucinations but also cause models to refuse or abstain even when the evidence is explicitly present, which lowers accuracy.

Their experiments suggest many failures are less about logical reasoning and more about ineffective usage of the available context. Even when the facts are present somewhere in the window, models often do not prioritize them properly and either ignore evidence or play it safe and decline to answer.

My take

I like the paper because it speaks to the kinds of failures I see in production RAG and agent systems. The extended needle-in-a-haystack framing matches real operational patterns: corpora are not uniform, facts can be rare, documentation is fragmented, and evidence often sits spread across many short chunks. The paper pushes beyond simple retrieval accuracy tests and separates extraction, inference, and hallucination. That separation is the right move because each implies different fixes.

There are limitations and things I wish were clearer. The experiments use proprietary models only. That makes the headline numbers less actionable for teams using open models or custom fine-tuned stacks. The paper does not fully disentangle retrieval failures from attention or reasoning failures. If evidence is distributed, is the model failing because the retriever did not surface the relevant chunks, because the model cannot integrate several small snippets, or because token ordering and window layout confuse the attention? Those mechanisms matter for engineering responses.

The Safety Tax finding aligns with my experience. Prompt-based refusal is blunt. Telling a model not to hallucinate is useful in the absence of a retrieval stage, but it is a blunt instrument when you do have evidence. It trades false positives for false negatives. In customer-facing systems that need to supply accurate answers or provide traceable abstentions, an uncalibrated refusal is not acceptable. You need calibrated uncertainty and provenance, not a binary "I refuse because I might be wrong."

The Distributional Collapse result is also unsurprising if you think about classic IR problems. Dense embeddings and retrieval have tokenization and chunking effects. A fact split across small chunks may have low retrieval score per chunk even though together they would reconstruct the fact. Models with large context windows do not automatically solve that. Long windows expand capacity but not necessarily the model's ability to prioritize sparsely distributed signals.

Implications for production systems

If you are building systems that rely on long context windows, treat this paper as a warning and a checklist.

First, test for distributional collapse explicitly. When you build evaluation suites, include cases where evidence is concentrated and where it is fragmented. Measure not just final answer accuracy but retrieval coverage, attention attribution where possible, and whether the model cites supporting passages.

Second, stop thinking of anti-hallucination prompts as a free safety layer. They change operating characteristics. If you use "Don't Make It Up" language, measure the safety tax: how often does the model refuse when evidence exists? Calibrate abstention thresholds and provide fallback flows. For example, have a verifier model that can inspect the cited evidence and produce a calibrated confidence. Use refusal with a clear remediation: say you cannot find the evidence and open a human-in-loop review rather than leaving the user hanging.

Third, fix the retrieval pipeline before blaming the LLM. Hybrid retrieval that combines BM25 and dense embeddings, chunking strategies that preserve context for moderately sized facts, and re-ranking layers for cross-chunk aggregation are practical fixes. In many cases the right engineering is to change how you index and present context to the model, not to expect the model to magically integrate dozens of tiny fragments.

Fourth, instrument. Observability is not optional. Track where evidence is found, which chunks were shown, model justification tokens, refusal rates, and the mismatch between retrieved evidence and model output. I work on observability precisely because these are the signals you need to diagnose Distributional Collapse versus hallucination.

Finally, accept trade-offs. You cannot optimize for zero hallucination and zero refusal without improving your retrieval and verification stack. The choice is operational. For high-stakes outputs you want higher precision and explicit provenance. For exploratory use you may tolerate more hallucination.

Bottom line

The paper makes a practical point: having a big context window is not the same thing as using available evidence well. Distribution of facts and prompt policies meaningfully change system behavior. That fact is not glamorous but it is critical for anyone running LLMs in production. I appreciate that the authors focus on measurable, model-agnostic failure modes. The next steps are engineering: better retrieval, explicit verification, calibrated abstention, and the observability to tell the two failure modes apart in the wild.