An Internal AI Knowledge Assistant for a Regulated Professional Services Firm

Challenge

Consultants spent significant time locating precedent documents and prior project material spread across several repositories.

Business problem

Precedent material was spread across several repositories with inconsistent naming and no shared search. Senior staff became the de facto search index, which slowed both them and the people asking. In a regulated sector, any solution also had to guarantee that no user could retrieve material they were not cleared to see.

Solution

An internal retrieval assistant over approved internal documentation, with citation of source documents and access controls mirroring existing permissions.

Architecture

• Ingestion pipeline indexing approved repositories with document-level permission metadata
• Chunking and embedding into pgvector, with hybrid keyword and vector retrieval
• Re-ranking stage before context assembly to improve precision
• Permission filter applied at query time, mirroring source repository access
• Answer generation with mandatory citation of every source document used
• Evaluation harness scoring answers against expert-written reference questions
• SSO authentication with full query and access audit logging

Technology stack

Implementation

• Indexed approved repositories with document-level permission mapping
• Built a retrieval pipeline with re-ranking and mandatory citations
• Evaluated answers against a set of expert-written reference questions
• Piloted with one practice group before firm-wide rollout

Results

• Document retrieval time reduced from tens of minutes to seconds in pilot testing
• Every answer traceable to a cited source document
• Permission parity with the underlying repositories maintained

Business impact

Billable time previously lost to internal search was recovered, and newer staff reached useful precedent without senior escalation.

Lessons learned

• Retrieval quality, not model choice, determined whether answers were useful — most of the effort went into chunking and re-ranking.
• Mandatory citations were the feature that earned trust; without them experts would not rely on the output.
• Building the expert evaluation set before development gave an objective bar to ship against and caught two regressions late in the build.