What Should AI’s Impact Be Compared Against?

Imagine a meeting after a generative AI trial in which users report, “Preparing documents is faster.” Even if they ask for a larger usage allowance, management still lacks enough information to decide. Was the time saved on completing the same document, or because there were more simple cases? Can the difference from people who did not use AI be called an implementation effect?

This article argues that measuring generative AI’s impact starts by deciding “what to compare with what,” before choosing metrics to aggregate. Keep the people, work, and completion criteria comparable, and state the differences that remain. Only then can the numbers inform a decision to expand usage.

Do not turn published savings rates into your company’s target

There is evidence from actual operations. CyberAgent says its generative AI creative platform “AI SCREAM” supports its own business and creative work for client companies, reducing production time by about 30% compared with conventional methods. This is a figure published by the provider, not an outcome independently verified by a third party. [1]

A survey by Persol Research and Consulting, on the other hand, reports that time saved with generative AI was reinvested in work. Its method was an online quantitative survey. Self-reported usage patterns should be distinguished from experimental results that measure task time under the same conditions. [2]

At your company too, treat “users felt it was convenient,” “time per task decreased,” and “the department’s throughput increased” as separate questions. Satisfaction helps explain why people continue using it; time records help improve a process step; throughput helps assess overall business results. Do not try to explain all three with one number.

Fix the scope of work first, not the list of people

A measurement plan should state the target process, start point, completion criteria, and comparison period. “Document creation” is too broad. For example, “creating an explanation with specified fields from designated internal materials, through the owner’s review” defines the scope of inputs and deliverables.

Next, classify ordinary work by difficulty or type and register in advance which cases are included. Do not design a process that selects only cases that seem easy for AI after the fact. If exclusions are necessary, record the reason and number, such as confidential information or an environment that has not been approved for handling it. Avoid applying the same savings rate to work outside the measured scope.

This caution also appears in research practice. In a replication study with developers, METR reported that some people declined to participate because they did not want to work without AI, while some did not submit certain tasks for testing. The organization says participant and task selection were biased, making it difficult to interpret the resulting estimates as measures of real-world productivity impact. [3]

There is no need to apply that study’s effect size to general work at Japanese companies. What is useful is the perspective of checking for people and tasks that do not appear in aggregate tables. If a trial includes only volunteers, keep the conclusion limited to those participants and target tasks. If deciding on a company-wide rollout, the plan should include workers with different levels of experience.

Choose a comparison method to match the question

What self-reports and before-and-after comparisons can tell you

Surveys work well for asking about perceived burden or difficulty of use. However, answers such as “how long it would have taken without AI” should be stored as estimates and not combined with measured time. When converting implementation impact to monetary value, do not rely only on survey responses.

When comparing before and after deployment, also record case volume, difficulty, owner, busy periods, and simultaneous process changes. For example, if AI is introduced right after document templates are reorganized, it is not valid to attribute all time saved to AI. Decide in advance whether to evaluate template changes and AI as a combined initiative or separate their implementation timing.

If comparing at the same time, record assignment and attrition

When similar cases are available, this article recommends comparing cases where AI can be used with cases handled conventionally during the same period. Choose target cases in advance and assign them so difficulty and operator experience are not concentrated on one side. If possible, randomize; also record any cases switched to AI midway alongside their original assignment.

Having the same person perform the same task twice can let them use what they learned the first time in the second run. And if a neighboring operator can receive an AI-generated draft, that also affects the conventional condition. Choose a comparison design that the team can maintain, such as using different but equivalent cases or splitting teams at a shared-work unit. If the split means operator skill differs, include that limitation in the report.

Aggregating only completed cases is another design to avoid. Count incomplete cases, early stops, and missing records separately. State which cases are used to compare time and which to judge completion, and check whether difficult cases dropped out of one condition.

Separate savings on existing work from newly created work

METR discusses productivity impact in three parts: time saved on work done before adoption, time saved on work chosen after adoption, and value added through work redesign. Since the composition of work changes, these three measurement approaches do not necessarily produce the same result. [4]

In practice, consider splitting the report in two. The first compares time and quality for work with the same completion criteria as before. The second tracks work newly undertaken because AI is available, asking what it was used for and which decisions it supported.

For example, if many explanatory materials are produced that were not made before, do not add the hypothetical time humans would have needed to make all of them as “time saved.” If the additional materials have value, evaluate them in connection with proposals adopted or decisions they informed. This distinction prevents equating output volume with value.

Time records should also separate the operator’s hands-on time from elapsed time between request and completion. The measurement endpoint should be when the defined completion criteria are met, not when a draft is generated. If converting a worker’s saved time into money, state whether it represents reduced spending or capacity available for other work.

Do not constrain frontline work too much for the sake of comparison

This approach is unsuitable when a process occurs only once, comparable work is unavailable, and completion criteria change during execution. Rather than pretending to have a rigorous comparison, record the decision process, deliverables, and time used, then use them to decide what to check in the next trial.

Also, when removing AI is incompatible with deadlines or safety requirements, a comparative trial should not take priority. It may be necessary to continue observing within a limited scope and defer claims about impact. Decide the measurement cost and recordkeeping burden in proportion to the investment decision at stake.

With a Forward Deployed Engineering (FDE) approach, working with customers from planning and prototyping through implementation and adoption, align comparison conditions when the trial begins. Development defines usage logs, the business side defines target cases and completion criteria, and management defines the difference needed for an expansion decision.

Practical implications

Recommendation 1: Before additional investment, put the comparison plan on one page. Decide participants, case selection, completion criteria, measurement period, exclusion reasons, and required improvement before viewing the results.

Recommendation 2: Count cases where AI was used and could not be used. Report incompletions and midstream condition changes, and limit the scope to which results apply.

Recommendation 3: Report existing-work savings separately from new outcomes. Separate results from comparing the same work from value created through work redesign, and explain the evidence for expanding usage.

References

1. CyberAgent, “AI SCREAM, the production platform for AI Creative BPO, now available to businesses” (published 2026-07-31; checked 2026-09-22)
2. Persol Research and Consulting, “Survey on Generative AI and Ways of Working,” press release (published 2026-02-03; checked 2026-09-22)
3. METR, We are Changing our Developer Productivity Experiment Design (published 2026-02-24; checked 2026-09-22)
4. METR, Task Substitution and Uplift (published 2026-05-08; checked 2026-09-22)

This article was created with AI assistance and automatically reviewed before publication. Information checked on: 2026-09-22.