
An enterprise AI service has to work for people with different permissions, across information that changes, while another team keeps the systems running. A convincing demonstration is only the beginning. The service needs a defined job, an operating owner, evidence of quality, and a way to recover when a dependency fails.
The first question is which business outcome the service owns. “Answer questions about everything” is difficult to measure. “Help service-desk staff find the current, approved laptop-replacement procedure for their region” gives you a bounded collection, an audience, and recognizable evidence of success.
On this page
Worked build: a permission-aware operations assistant
The proposed pilot answers questions from approved internal procedures. It returns a concise explanation with source references and routes unresolved questions to an existing service channel. It does not change devices, grant access, or create purchasing commitments.
Use fictional documents and test users for the initial build. The architecture can later connect to an approved document repository, identity provider, search service, model endpoint, and monitoring system.
Prerequisites: an accountable service owner; a document owner for each collection; an identity integration; a supported retrieval platform; a deployment environment; a test corpus; and agreed retention and support requirements. Decide which locations and data classes are permitted before ingestion.
Design the data path before the chat screen
The request passes through five distinct responsibilities: authenticate the caller; retrieve authorized evidence; assemble a bounded model input; validate the proposed answer and citations; return the answer with an observable request record.
| Component | Responsibility | Evidence it should produce |
|---|---|---|
| Identity layer | Establish the caller and relevant groups | Verified identity context |
| Retrieval layer | Enforce document eligibility and freshness | Returned source IDs and versions |
| Model layer | Explain the supplied evidence | Proposed answer and source references |
| Application checks | Reject unsupported references and invalid outputs | Validation outcome |
| Service operations | Monitor, support, and improve the service | Metrics, incidents, and release records |
The model must not receive unauthorized text and then be asked to keep it secret. Enforce access before content enters the model request, and apply the same boundary to titles, snippets, citations, cached responses, and logs.
Create the test collection
[
{"id":"IT-101", "version":4, "status":"approved", "groups":["service-desk"], "text":"Use the replacement-request form for a failed standard laptop."},
{"id":"IT-099", "version":1, "status":"retired", "groups":["service-desk"], "text":"Email the old purchasing mailbox."},
{"id":"FIN-220", "version":2, "status":"approved", "groups":["finance"], "text":"Fictional restricted purchasing terms."}
]
For a test user in service-desk only, “How do I replace a failed standard laptop?” should use IT-101 version 4. IT-099 is retired, and FIN-220 is outside that user’s scope. The assistant should not reveal a restricted document’s title while explaining why it was excluded.
Implement the pilot in eight steps
- Select the collection. Start with a manageable set of approved procedures. Identify owners, versions, effective dates, and retirement behavior.
- Define identity mapping. Map the authenticated user to the repository’s authorization model. Resolve groups in trusted application code; do not accept a group list supplied by the user prompt.
- Ingest content and permissions together. Preserve document identity, source location, version, eligibility, and access metadata on every searchable chunk. Define how deletions and permission changes propagate.
- Build retrieval with access checks. Filter on current authorization and publication status before returning evidence. Test exact keyword questions and paraphrases separately.
- Assemble the model request. Include the question, approved passages, and their source IDs. Treat retrieved prose as evidence. Give the model no tool that can expand its access on its own.
- Validate the result. Require citations to returned source IDs. Check that required fields exist and citations resolve to the allowed result set. A valid citation ID still needs evaluation for whether its passage supports the claim.
- Return an answer or a useful fallback. If the evidence is absent or conflicting, say what remains unresolved and link the user to the existing support route.
- Record and review. Log the request ID, relevant versions, latency, cost, retrieval outcome, and validation result under the agreed retention policy. Limit raw-content logging to what the service genuinely needs.
Use a model contract like this:
Answer the question using only the supplied evidence.
Return answer, citations, unresolved_questions, and status.
Every citation must name a supplied source ID and version.
If evidence is insufficient, set status to needs_review.
If passages conflict, report the conflict with both references.
Do not invent policy, approval, implementation, or completion.
Choose the permission mechanism deliberately
Microsoft’s search documentation describes both application-applied security filters and native permission integrations. Several native ACL, scope, and sensitivity-label paths are marked preview in the reviewed overview. Their availability and suitability must be checked for the intended deployment. Permission metadata also has to stay synchronized with the source. Document-level access overview.
For a filter-based design, the trusted application must construct the identity filter and prevent callers from bypassing it. String matching in an index is not a substitute for authenticating the caller. Microsoft’s security-filter pattern provides the implementation model.
Other stacks may enforce authorization through source queries or another retrieval service. Evaluate the complete data path with actual test identities. Document the expected delay between a permission change and its enforcement, and decide what the service does when synchronization is unhealthy.
Evaluate access and answer quality separately
| Test | Passing behavior |
|---|---|
| Correct procedure exists | Answer cites the approved current version |
| Only retired guidance exists | No authoritative answer from the retired document |
| User lacks access | No restricted text, title, or revealing snippet |
| Group membership changes | New authorization takes effect through the documented mechanism |
| Two current sources conflict | Conflict is shown and routed for resolution |
| Source contains an instruction to ignore controls | Instruction does not alter application permissions |
| Retrieval service fails | Clear unavailable response; no answer fabricated from memory |
| Different user asks the same question | Cache cannot cross the authorization boundary |
These are acceptance criteria for the proposed design. They are not measured results from a deployed enterprise service.
Keep separate measures for retrieval coverage, supported answers, correct refusals or fallbacks, access-boundary violations, latency, and review workload. A good average score cannot excuse an unauthorized disclosure.
Release and operate the service
Begin with a controlled audience and a small source set. Compare answers against an approved reference set, investigate failures, and expand only when the service owner accepts the evidence. Keep model, prompt, retrieval, and document changes traceable. Re-run relevant tests when any of those components changes.
Provide a switch that disables answering while retaining a useful support link. Document rollback, credential rotation, source reindexing, and manual handling during an outage. Define who responds to an incorrect answer and who repairs the underlying document.
Budget for ingestion, search capacity, model input and output, storage, evaluation, observability, support, and engineering maintenance. Track cost per resolved task alongside answer quality; reducing model cost while increasing review work may be a poor trade.
A smaller organization can use the same principles with fewer components and a narrower collection. A multinational deployment may need regional repositories, contractual review, retention controls, and separate service boundaries. Industry requirements add another dimension: compare the healthcare and legal services guides, then use the governance guide to name the responsible people.