An Average Alone Cannot Set the Pass Line: Repeat Evaluation for Generative AI Variability

In a validation session, a generative AI system answered a prepared question correctly. The owner recorded the result in a spreadsheet and calculated the accuracy rate. Yet when the same question was entered again the next day, an important condition was missing. If only the first result had been saved, that variation would have been hidden behind a “pass” score.

A production decision for generative AI must consider not only the average score but also the variation across repeated runs under the same conditions and the business conditions in which failures cluster. What organizations need is not an academic benchmark for ranking models, but repeat evaluation that determines whether their business decisions can be protected.

One correct answer is an observation, not a capability

Consider a hypothetical evaluation that classifies 100 contract clauses and gets 92 correct. The 92% figure alone does not tell us whether rerunning the same 100 cases produces a similar result, whether the same eight difficult cases always fail, or whether different items are missed each time. Evaluation volume and tolerance must be set according to business risk.

NIST’s “AI 800-3” distinguishes benchmark accuracy on a fixed problem set from generalization accuracy across the broader population of similar problems. It also explains that uncertainty in evaluation results comes not only from problem selection but also from the nondeterministic sampling of LLMs.[1] A single run is not the expected performance itself; it is one observation drawn from a distribution.

Version 2.0 of Japan’s Digital Agency guidelines for procuring and using generative AI likewise gives examples of robustness checks: consistency when the same test data is entered repeatedly, consistency across semantically similar inputs, and stable behavior with typing errors or notation variations.[2] Japan is rolling out the government AI system “Gennai” across all ministries, aiming to provide access to about 180,000 government employees during fiscal 2026.[3] As usage expands, a one-time test passed by one representative cannot cover variation in inputs.

Measure variation in three parts

Repeating an identical input

First, fix the model, instructions, reference data, tools, and parameters, then run the same input several times. Record more than average accuracy. Preserve the share of runs containing every required field, the number of cases that ever triggered a prohibited outcome, and the worst-run result. Even if four out of five runs are correct, the workflow may still fail the business gate if the incorrect run is sent automatically.

Separate work in which a small wording change can alter the conclusion from work that allows multiple valid phrasings. “Exact text match” scoring marks semantically correct wording differences as failures. Instead, structure and score the elements that must remain fixed for the business: amount, date, category, supporting source, or action performed.

Paraphrasing while preserving meaning

Next, preserve the question’s meaning while changing word order, formality, abbreviations, typographical errors, and input length. The goal is not to test whether AI can polish writing, but whether differences in user expression reverse a business decision. Group the original and variants together, then compare the conclusion, required evidence, and rejection conditions.

The same answer is not always correct for every paraphrase. A shorter question with missing information should prompt a request for clarification rather than a guess. Evaluate whether the system followed the same decision rule, not merely whether it produced the same conclusion.

Failure distribution by business condition

Finally, do not collapse results into one overall average. Split them by business condition: routine processing, exceptions, boundary values, conflicts between rules, missing reference data, and requests beyond the user’s authority. Even with a high total score, concentrated failures immediately before a cancellation deadline or only on high-value cases should block automation for that use.

The NIST AI RMF Playbook states that documenting test sets, metrics, and the tools and procedures used for evaluation and validation supports repeatability and consistency in measurement.[4] Along with repeat count, record the model version, execution time, instructions, reference-data version, and scoring rules as one package.

Build the pass criteria from the average, worst case, and prohibited failures

A pass/fail decision needs at least three gates. The first is average performance across the target work. The second is the worst condition: whether critical business groups remain above their minimum threshold. The third covers prohibited failures, such as unauthorized actions, exposure of personal information, unsupported definitive statements, or incorrect amounts and recipients.

An average can be raised simply by adding more frequent, easy questions. The scorecard should therefore show both case counts and results for each business group. For items important enough to change the pass/fail result, provide positive and failing examples instead of leaving the decision entirely to scorer discretion.

OpenAI’s official evaluation guide recommends designing evaluations in the order of objective, dataset, metrics, comparison, and continuous evaluation, then running them after each change and adding newly observed nondeterministic cases.[5] This is vendor implementation guidance, but the practice of returning changes and operational cases to the evaluation set—rather than ending with a one-time acceptance test—can be considered regardless of product.

Repeat evaluation is not suitable when output is only read by a person, errors can be corrected immediately, results do not feed external processing, and reproducing the same answer is unnecessary. Under those conditions, the cost of building repeat infrastructure may outweigh the benefit. However, when AI output feeds classification, approval, notification, or registration, worst-run outcomes and prohibited failures cannot be ignored.

Practical implications

Recommendation 1: Do not score a question only once. Separate identical-input repetition, meaning-preserving paraphrases, and aggregation by business condition.

Recommendation 2: In addition to an average score, define minimum thresholds for critical business groups and prohibited failures that cause an immediate fail if they occur even once.

Recommendation 3: Record the model version, instructions, reference data, execution time, and scoring rules together so every change in an evaluation can be traced.

Recommendation 4: In FDE (Forward Deployed Engineering), collect real input variations and exceptions that make frontline staff hesitate, then return failures to the evaluation set. A production gate should determine not “what score did it get?” but “under which conditions can we depend on it consistently?”

References

1. NIST, “Expanding the AI Evaluation Toolbox with Statistical Models”
2. Digital Agency of Japan, “Guidelines for the Procurement and Use of Generative AI for Administrative Evolution and Innovation, Version 2.0”
3. Digital Agency of Japan, “Government AI ‘Gennai’”
4. NIST AI RMF Playbook, “Measure”
5. OpenAI, “Evaluation best practices”

This article was created with AI assistance and automatically reviewed before publication. Information checked on: 2026-09-27.