Illustrative engineering walkthrough
A permission-aware knowledge assistant: engineering walkthrough
An operations team needs answers from internal procedures. Some documents are shared across the business; others are restricted. This walkthrough shows how access and evidence shape the application.
Ars Codex · Updated 2026-10-28
Scenario and boundaries
Assume a small collection of fictional operating procedures with named owners and document-level permissions. Users need to find a current procedure and verify its source. The assistant is read-only: it cannot change records, approve work or operate equipment.
This is a proposed design with synthetic inputs, not a case study or measured deployment. Source connectors, identity integration and evaluation thresholds would be chosen for the real environment.
Data preparation and retrieval
Ingest only approved sources. Retain document identity, version, owner, access policy and the reference needed to reopen the original. Preserve enough section context for a retrieved passage to make sense on its own.
Resolve user identity and permitted sources before retrieval. Apply the same boundary to snippets, cached answers and displayed references. Reindex or remove content when ownership, permissions or source status changes.
Answer behavior
Retrieve relevant permitted passages, then ask the answer step to use that evidence. Display the supporting references with each answer. When procedures disagree, show the conflict rather than merge contradictory instructions into an invented policy.
If evidence is missing, say so and point to the appropriate source owner. Treat text inside documents as untrusted content; a passage should not be able to override operating instructions or expand tool access.
Acceptance cases
- A current procedure produces an answer supported by the cited section.
- A question outside the collection results in a clear abstention.
- A restricted document is absent for an unauthorized user, including its title and references.
- A superseded procedure is not presented as current.
- Conflicting instructions are exposed with their sources.
- Malicious instructions embedded in a document do not change the assistant’s rules.
Evaluation and handover
Keep development and acceptance questions separate, and record the source collection and configuration for each evaluation run. Measure supported answers, abstention, permission failures, response time and cost per task separately.
Handover should identify who maintains source permissions, refresh jobs, prompts and tests. This design does not claim perfect security, guaranteed accuracy or measured productivity gains.
Read the assistant evaluation guide, or explore AI consulting and AI development.