Practical guide
How to evaluate an internal AI assistant
A convincing demo can hide an unreliable assistant. Evaluate the tasks your users need to complete, the evidence supporting each answer and the boundaries the system must respect.
Ars Codex · Updated 2026-10-28
Build a test set before tuning the system
Collect representative questions from the intended users. Include straightforward lookups, multi-document questions, ambiguous wording, outdated procedures and questions whose answer is absent. Keep expected sources and acceptable behavior alongside each question.
Separate a development set from a held-out acceptance set. If every change is judged on the questions used to tune it, the final score says little about unfamiliar work.
Score more than fluent writing
- Retrieval: was the necessary source available to the answer step?
- Support: do the cited passages actually justify the answer?
- Completeness: are important conditions or exceptions missing?
- Abstention: does it admit when the evidence is insufficient?
- Access: does it avoid retrieving restricted material for an unauthorized user?
- Usability: can a person follow the reference and finish the task?
Keep these dimensions separate. A single average can conceal a critical failure in access control or source accuracy.
Test permissions and misleading source content
Run the same question under accounts with different permissions. A restricted document should not become visible through its title, snippets, citations, cached answers or logs exposed to the wrong user.
Include a source document containing misleading instructions to the assistant. Treat retrieved text as evidence to analyze, not authority to change the system’s operating rules. Check both the answer and any proposed tool action.
Compare changes against a stable baseline
Version the source collection, retrieval settings, prompts and model configuration used for each run. Record the evaluator’s reasons as well as pass/fail judgments. Run the same set after changes to indexing, models or permissions.
Define acceptable response time and per-task cost for the actual workload. Evaluate a useful sample with users before a wider rollout, then use observed failures to expand the test set without discarding the original baseline.
Make acceptance a business decision
Agree which failures prevent release and which require a visible warning, a human check or a narrower scope. A procedure lookup and an assistant allowed to change customer records need different acceptance rules.
See our illustrative knowledge assistant design, AI consulting and custom AI development.