Designing a RAG PoC for FDE: Beyond the Search Demo

A RAG proof of concept (PoC) is often easiest to demonstrate as a chat interface that searches internal documents and answers questions. In practice, however, the first question should not be “Can it produce a plausible answer?” but “How far may that answer be used as a basis for a business decision?” If this boundary remains unclear, the result may be a searchable demo without a design robust enough for operational decisions.

First, our position: a RAG (Retrieval-Augmented Generation) PoC should begin as work to design quality controls for business decisions, not as a prototype of a generative AI interface. On day one, FDE (Forward Deployed Engineering) should define the decision boundaries, source documents, evaluation, permissions, and deliverables to retain after the PoC before choosing a model or UI.

For Japanese companies, a practical starting point is work whose decision basis is recorded in documents, such as internal policies, quality manuals, product specifications, and maintenance procedures. At the same time, if documents are inconsistent about the latest version, effective date, applicable department, exceptions, or viewing permissions, putting them directly into AI search will blur the questions the PoC is meant to validate.

On day one, decide what the system may support people in judging—not only what it should answer

Naming the PoC scope “policy search,” “questions about specifications,” or “inquiry support” is not enough. FDE should document the boundaries between decisions an answer may support, decisions it must not support, and decisions that require human review.

For example, with internal policies, “checking the travel expense limit” is different from “deciding whether to approve an exception.” The former mainly requires citing the relevant clause; the latter also involves authority, past practice, and the approval route. In the PoC, we recommend avoiding a design that forces both types of decision into the same chat interface.

If evaluation starts before this boundary is defined, discussion will focus only on how natural the answers sound. The demo may look polished to outsiders, while the user department, audit staff, IT, and procurement still disagree about what the PoC actually tested.

Fix the boundary of source documents before ingesting data

The Azure AI Search overview of RAG explains that rather than passing large volumes of documents directly to a large language model (LLM), the system needs to retrieve relevant chunks. This means the documents included in search, the chunk size, and the search strategy are prerequisites for answer quality. 5

Azure AI Search also identifies chunking during indexing as an element of RAG design. 5

Based on this, FDE should first create an inventory before data ingestion, recording document name, version, effective date, applicable department, and owner. Record separately the operational attributes as well: whether the document may be referenced, why it is excluded, the chunking policy, and who owns updates. This may look like a full-scale document management project, but in RAG these are input conditions for search quality itself.

Avoid ingesting whatever PDFs and web pages happen to be available, then discovering later that old documents were mixed in or that the system relied on documents a user was not allowed to view. For example, if old and new versions of a quality manual are in the same search index, an answer citing the old procedure should be treated as a data-boundary failure, not a UI issue.

Evaluate search quality and answer quality separately

Microsoft’s description of RAG evaluators separates evaluation into process evaluation of retrieval and system evaluation of the final answer. On the retrieval side, Retrieval (retrieval quality) is evaluated; on the answer side, Groundedness and Relevance are among the evaluation targets. Groundedness examines whether an answer is based on the supplied context. 7

This split helps isolate why a PoC failed. Unless we distinguish between search failing to return the right evidence chunk and answer generation omitting or overreaching despite having the evidence, we cannot tell whether to improve search settings, prompts, or document preparation.

Azure AI Search describes chunking during indexing, embeddings for vector search, and hybrid search combining keyword and vector search. The same page also discusses semantic ranking, scoring profiles, and adjusting vector weights and thresholds. 5

The level of detail in FDE’s evaluation table

We recommend designing an evaluation set to include not only questions and expected answers, but also the expected source document and passage, acceptable paraphrases, and conditions under which the system must not answer. Even when using Groundedness, do not stop at the abstract phrase “based on evidence”; specify in the evaluation table which document and which passage count as valid evidence. 7

For example, with a maintenance manual, the question “What is the restart procedure for equipment A?” is too broad. First distinguish whether the expected evidence is “restart after a normal shutdown” or “recovery after an abnormal stop.” Then specify scoring conditions in advance—for instance, omitting the precautions in the latter case is a failure.

For performance evaluation of generative AI workloads, AWS’s Generative AI Lens recommends regularly evaluating with ground-truth data and comparing outputs with known correct answers to track performance. 8

At the PoC stage, we believe placing a broad but lean set of representative cases around each decision boundary makes it easier to check the design intent than simply making the evaluation set larger. Rather than creating an illusion of coverage, prioritize an evaluation table that can explain “this business decision is a candidate for use” and “this exception decision is out of scope.”

Do not treat permissions as a later add-on

In practice, even a PoC should not rely only on login controls in the UI. Decide in advance who can search which document sets, and where that rule will be represented: in the data store, search index, or chunk metadata.

Avoid the compromise of “it’s a PoC, so everyone can try the same index.” Even if logins are separated on screen, shared searchable chunks can allow evidence from documents a user should not see to enter an answer. Even in a PoC, permissions should be treated as part of data-store and search-scope design, not just UI.

Consider putting HR policies and department-specific operating notes on the same RAG platform. If the documents available for reference need to vary by user role, it is natural to put that condition in a pre-search filter or index-separation design, not only in the answer-generation instructions. No matter how carefully the generated answer is controlled, if inappropriate evidence was retrieved at the search stage, it is difficult for generation downstream to compensate for a design flaw.

Express post-PoC deliverables in contract and procurement terms

We recommend that the PoC deliverables not consist only of a working interface. In RAG, document ingestion, chunking, indexing, retrieval, answer generation, and evaluation form a chain. Confirm at the outset which parts will remain as design assets owned by the company and which will be used as service features.

In FDE practice, this question should be addressed at the start, not at the end of the PoC. The index, chunking rules, evaluation set, permission matrix, and records of failure cases created during the PoC may be reused in the next MVP or production procurement. If the parties proceed without documenting what is a deliverable and what is a service feature, it becomes difficult to turn technical validation results into organizational assets.

• Document inventory: list the target documents, versions, owners, scope of application, and reasons for exclusion.
• Chunking and indexing design: record chunking policy, search method, thresholds, and the rationale for the number of top results retrieved.
• Evaluation set: record questions, expected evidence, expected answers, failure conditions, and user roles.
• Permission matrix: map users or roles to the data stores and document sets they may reference.
• Open issues list: record missing documents, deficiencies in version management, exception handling, and decisions that could not be evaluated.

This list is not a vendor selection table. It is the minimum set of design assets needed so that anyone taking over after the PoC can resume validation at a useful level of detail. From an AIX (AI transformation) perspective, a RAG PoC is both an AI introduction initiative and an exercise in connecting documents, audit, contracts, and quality evaluation.

When this approach is not suitable

The design in this article is not suited to cases where the goal is to quickly try a public-facing conversational bot, or where creativity matters more than evidence documents. If the evaluation target is the user experience itself rather than business decisions, an extensive document inventory and permission matrix become lower priorities.

On the other hand, for use cases involving policies, specifications, procedures, contracts, and quality standards—where the basis for an answer may be questioned later—rushing into a search demo can create more rework. FDE needs not only the ability to build a working interface in the first week, but also the ability to agree on boundaries that must be settled before it works.

Practical implications

First, we recommend not opening a PoC kickoff with “collecting the target documents.” Put on one page the decisions the system may support, may not support, and must send for human review. Then work backward to identify the source documents needed.

Second, separate search evaluation from answer evaluation. Search evaluation checks whether the expected evidence chunks are retrieved. Answer evaluation checks whether the response is grounded, whether it omits anything necessary, and whether it refrains from answering under conditions where it must not answer.

Third, translate permissions and deliverables into contract and procurement language on day one. Confirm whether the index, chunking rules, evaluation set, permission matrix, and failure-case records produced in the PoC will be deliverables or prerequisites for using the service.

FDE’s role is not limited to placing AI in the field. It is to articulate decision boundaries, how source documents are handled, evaluation criteria, permissions, and transferable deliverables, then design the conditions under which teams may use AI.

---

References

[1] Digital Society Promotion Standard Guidelines (2026-09-01) https://www.digital.go.jp/resources/standard_guidelines

[5] Retrieval-augmented generation (RAG) in Azure AI Search https://learn.microsoft.com/en-us/azure/search/retrieval-augmented-generation-overview?tabs=docs

[7] Retrieval-Augmented Generation (RAG) Evaluators for Generative AI https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators

[8] GENOPS01-BP01 Periodically evaluate functional performance - Generative AI Lens https://docs.aws.amazon.com/wellarchitected/latest/generative-ai-lens/genops01-bp01.html

This article was created with AI assistance and automatically reviewed before publication. Information checked on: 2026-09-21.