Skip to content
All case studies

Case study · 2024 to present · enterprise client

Crystal: an enterprise RAG and copilot platform

Grounded, permission-aware answers inside Teams, CRM and Office for 2K+ active users.

active users
2K+
front ends: Teams, CRM, Office, web
4
shared retrieval platform behind all of them
1

My role: Technical lead and architect. I led the team that designed, built and runs the platform.

01

The problem

Knowledge workers were answering the same questions by digging through documents spread across several systems. They needed answers in the tools they already worked in, with a source they could check.

Two constraints shaped everything. Answers had to respect who was asking, so nobody could see content they couldn't already open. And retrieval had to be fast enough to feel like part of Teams or CRM, not a separate search tool.

What I owned

02

How it works

  1. 01IngestContent is pulled from the source systems with its metadata and access control lists, then normalized so every chunk keeps a link back to the original document.
  2. 02Enrich and chunkDocuments are cleaned, split and embedded. Chunking quality turned out to decide retrieval quality more than any later step.
  3. 03IndexAzure AI Search stores the text, vectors and permission fields together, so a single query can filter by identity.
  4. 04RetrieveHybrid retrieval runs vector and BM25 keyword search together, trimmed to what the caller is allowed to see.
  5. 05RerankSemantic ranking and a cross-encoder reorder candidates, so the model gets fewer and better passages.
  6. 06GenerateAzure OpenAI answers only from the retrieved passages, returns structured output and cites each source.
  7. 07LearnFeedback and telemetry from every front end flow back into one view of answer quality.

03

Key decisions

Hybrid retrieval (vector plus BM25), not vectors alone

Enterprise questions are full of product names, IDs and acronyms that embeddings blur together. Keyword search catches the exact terms and vectors catch the meaning.

Tradeoff · Two ranking signals to tune, and a fusion step to keep stable as content changes.

Rerank before generation

Semantic and cross-encoder reranking let the model read a short list of strong passages instead of a long list of possible ones. That improves grounding and keeps prompts small.

Tradeoff · An extra step on every query, so reranking depth is capped to stay inside the latency budget.

Enforce permissions at the index, not in the prompt

Security trimming is applied as a filter inside the search query, so content a user can't open never reaches the model at all. A prompt is never the access control.

Tradeoff · ACLs have to be kept in sync with the source systems, and permission changes need a re-index path.

One retrieval platform behind several front ends

Teams, CRM, Office and the web app all call the same API, so a retrieval fix improves every surface at once and quality can be measured in one place.

Tradeoff · The shared API needs careful versioning, because every client depends on it.

Cache by identity and freshness

Caching cuts repeat cost and latency, but a cached answer must never cross a permission boundary or outlive the content it came from.

Tradeoff · Lower hit rates than a naive global cache, which is the right price for correctness.

04

What broke, and what we changed

  1. 01Permission-aware retrieval was the hardest part to get right. Answers have to match what each person can open in the source systems, and those permissions keep changing.

    Permissions became indexed fields filtered inside every query, with a re-index path for ACL changes. Access is decided before generation, never by the model.

  2. 02Prompt drift. As prompts, models and content changed, answer quality could slip without anyone noticing.

    Feedback and telemetry are watched continuously, so a regression shows up as data rather than as a complaint.

  3. 03Citation quality. A citation that points to roughly the right document isn't enough for users to trust an answer.

    Every chunk keeps a link to its source, and the model has to answer from the retrieved passages and cite them in structured output.

  4. 04Feedback was scattered across four front ends and several data sources, so nobody had a full picture of quality.

    Feedback and telemetry from every surface go into one view, which made analytics part of the product loop instead of an afterthought.

05

Results and lessons

Where it stands

  • In production serving 2K+ active users across Teams, CRM, Office and a React web app
  • Answers are grounded with citations and trimmed to each user's permissions
  • One shared retrieval platform, so improvements reach every surface at once

What I’d tell the next team

  • Retrieval quality beats prompt cleverness. Most bad answers trace back to the wrong passages, not the wrong wording.
  • Caching has to be scoped by identity and freshness, or it becomes a data leak.
  • Analytics belong inside the copilot product loop, not in a report someone reads later.