Does Giving AI More Documents Improve Its Answers? Designing Input Selection

An employee rushing to prepare an executive response selects 80 seemingly relevant documents from a shared folder and gives them all to a generative AI system. The returned prose is polished, but tracing the evidence reveals that a retired procedure and the current policy were treated with equal weight. The reassurance of supplying more material made the version conflict harder to see.

Even as more models accept long inputs, “how much fits” is not the same as “how much usable evidence the business has.” The first design question is not the maximum character count, but what to include for each question, what to exclude, and which version takes priority. This article calls that selected input set an “evidence pack.”

Giving the model everything hides decisions in exchange for skipping search

The benefits of sending every document are obvious: less anxiety about missed search results and faster prototyping. But when duplicate files, obsolete versions, reference-only documents, and information with different access levels share one input, the answer does not reveal which material was treated as authoritative.

The long-context study “Lost in the Middle” reports that in tasks such as multi-document question answering, performance can change substantially when only the position of relevant information changes, with a tendency to decline when information appears in the middle of the input.[1] This does not claim that every current model is affected to the same degree. It does show that a large context window alone is not evidence that the required material will be used reliably.

Anthropic’s official guidance also recommends structuring documents and source information and asking for relevant quotations before an answer when working with long documents.[2] Although product-specific, the operational implication is broader: before the final answer, create a stage where reviewers can see which parts of which documents were selected as candidate evidence.

Another option is to provide a small fixed set of documents. That is easier to manage but becomes insufficient as the question scope expands. An evidence pack sits between these extremes: collect what is needed for the question type, verify versions and permissions, and only then send it to the model.

Select an evidence pack along four axes

1. Authority — official document or reference material

First, establish document rank. Internal policies, approved procedures, contracts, FAQs, and meeting notes should not be treated equally. The business team must define priority when sources conflict—for example: “contracts and law,” “internal policies,” “approved procedures,” “reference material,” and “personal notes.” If AI infers this hierarchy, it may mistake detail or recency for authority.

2. Version and effective period — the latest version is not always enough

For questions about current operations, filter documents by effective and expiry dates. For an audit of an older case, however, the required version is the one effective when the case was handled, not today’s latest version. A single “use the newest version” rule is insufficient. Record the question’s reference date, document version, and applicable scope together.

3. Relevance — collect decision fields, not merely similar words

Keyword matches do not guarantee that a document contains what is needed for the answer. For a travel-expense question, do not stop at the phrase “transportation expense.” Check that the source covers eligible people, routes, limits, exception approval, and effective date. Add a missing-field check instead of passing top search results directly to the model.

4. Disclosure scope — distinguish what can be read from what should be supplied

A user’s permission to read a document does not mean its full contents are necessary for a specific answer. For personal data, trade secrets, or customer-specific terms, compare options such as extracting only the relevant passage, replacing identifiers, or using a lower-classification summary. The NIST Generative AI Profile identifies privacy risks from models revealing sensitive information or inferring it by combining multiple pieces of information.[3]

Add provenance before displaying citations

Each fragment in the evidence pack should carry a document ID, title, owning department, version, effective period, classification, link to the original, and extraction location. Citation numbers in the answer do not help a reviewer judge correctness if the provenance of the supplied material is unclear.

Version 2.0 of Japan’s Digital Agency guidelines for procuring and using generative AI gives provenance metadata as an example safeguard for information selected at runtime and inserted into a prompt. Its procurement checklist also covers managing the accuracy, confidentiality, integrity, structure, and provenance of data that affects generative AI.[4] Although intended for government procurement, these fields can inform how companies manage evidence packs.

Split processing into two stages. The first collects candidate documents based on question type, reference date, and user permissions. The second checks duplicates, expiry, authority conflicts, and missing required fields, while recording reasons for inclusion and exclusion. Only fragments that pass the second stage go to the model.

When a conflict is detected, do not let the model conveniently merge it away. Return the difference—for example, “Policy A sets limit X, while Procedure B sets Y”—and ask the document owner to resolve it. NIST warns that generative AI may produce not only incorrect content but plausible reasoning and fabricated citations that encourage overreliance.[3] Verification must therefore cover not just whether a citation exists, but whether the source exists and the cited passage supports the claim.

Measure answer quality and input quality separately during trials

For each evaluation question, store not only the expected answer but also “documents that must be used,” “obsolete versions that must not be used,” and “required decision fields.” Measure answer correctness, correct-source selection, obsolete-version contamination, missing fields, and unauthorized-information contamination. An answer that happens to be correct still fails if it relies on the wrong version.

Consider a hypothetical monthly policy inquiry with 100 questions divided into current rules, historical points in time, exception requests, and conflicts across policies. First evaluate search and selection alone; then generate answers from the selected evidence packs. If the first half fails, tuning prompt wording alone targets the wrong problem. Actual case counts should reflect the work and its risk.

Dynamic evidence packs are not suitable when there are few documents, no version conflicts, a single classification level, and an expert always reviews the final output closely. Under those conditions, the selection infrastructure may not justify its cost. Conversely, frequent revisions, historical answers, or department-level access controls make evidence-pack management essential.

Practical implications

Recommendation 1: Do not center requirements on model context limits. Define document authority, version, relevance, and disclosure scope first.

Recommendation 2: For each question, define a reference date and required decision fields, and record reasons for exclusion as well as inclusion.

Recommendation 3: In addition to citations in the answer, implement provenance tracking with links to originals, effective periods, owning departments, and extraction locations.

Recommendation 4: In FDE (Forward Deployed Engineering), use the questions and source conflicts that actually confuse frontline staff to test retrieval and selection separately from answer generation. What AI is allowed to read is a business decision, not merely a technical setting.

References

1. Liu et al., “Lost in the Middle: How Language Models Use Long Contexts”
2. Anthropic, “Prompting best practices”
3. NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile”
4. Digital Agency of Japan, “Guidelines for the Procurement and Use of Generative AI for Administrative Evolution and Innovation, Version 2.0”

This article was created with AI assistance and automatically reviewed before publication. Information checked on: 2026-09-26.