Skip to content
All case studies

Case study · 2025 to present · enterprise client

Agent Assist: a multi-agent document intelligence pipeline

OCR, classification, translation, extraction and validation for 70K+ documents a month, with people reviewing only what needs them.

documents processed per month
70K+
pipeline stages, from intake to CRM hand-off
7
VNet-restricted networking with Managed Identity
Private

My role: Architect and technical lead. I designed the system and led the team that built and deploys it.

01

The problem

Support agents were receiving a high volume of documents in many formats and languages, and they handled each one by hand: read it, work out what it is, pull out the details, then key them into CRM.

The goal was to do that work automatically and reliably, then send people only the documents that genuinely needed a human, without letting OCR and LLM costs grow in step with volume.

What I owned

02

How it works

  1. 01IntakeDocuments arrive from the CRM workflow and are stored with their case context. Logic Apps and Functions start the pipeline.
  2. 02OCR and parseAzure Document Intelligence, Content Understanding and a dedicated parser service turn scans and native files into structured text.
  3. 03ClassifyA lightweight classifier identifies document type, language and priority, and decides the route before any expensive model runs.
  4. 04TranslateNon-English documents are translated so the downstream agents work from one language.
  5. 05Extract and summarizeExtraction agents pull the fields each document type needs, and a summary is written for the agent handling the case.
  6. 06ValidateOutputs are checked against a validation contract. Anything below the confidence threshold goes to a human review queue instead of straight through.
  7. 07Hand off and indexResults go back to Dynamics CRM, and embeddings are indexed in Azure AI Search so the content can be retrieved later.

03

Key decisions

Route first, then spend

Cheap classification and confidence thresholds decide what happens before expensive OCR or LLM calls. Easy documents take a short path, and only hard ones pay for the full pipeline.

Tradeoff · Thresholds need tuning per document type, and a wrong route costs a review.

Durable Functions for the long-running flow

Processing one document is a multi-step job that can take a while and fail partway. Durable orchestration keeps state between steps, so retries resume rather than starting over.

Tradeoff · Orchestrator code has to be deterministic, which takes some discipline from the team.

Explicit validation contracts between agents

Each agent hands off structured output with a defined shape. Validation checks that contract, which makes failures visible and gives human review a clear reason for each escalation.

Tradeoff · More up-front schema work for every new document type.

Move to the Responses API with the Files API

Documents can be passed to the model as files instead of being stuffed into prompts as text, and the Responses API is where new Azure OpenAI capabilities are landing.

Tradeoff · A migration across working code paths, done while the system stayed in production.

Canary releases with Container Apps revisions

Revision-based traffic splitting sends a small share of documents to a new version first, so a bad release is caught on a fraction of traffic and rolled back by shifting weight.

Tradeoff · Two versions run side by side during a rollout, so both have to handle the same inputs.

Private networking and Managed Identity throughout

These are customer documents. Services talk over a VNet-restricted network and authenticate with Managed Identity, so there are no keys in config to leak.

Tradeoff · Harder local debugging and more networking setup for every new service.

04

What broke, and what we changed

  1. 01Python Function App deployments to pre-production failed during the Oryx remote build and ZipDeploy steps, even though the same code ran fine locally.

    We worked through the build and packaging path until deploys were repeatable, instead of patching individual releases by hand.

  2. 02When Azure Content Understanding went GA, calls that worked against the preview API started returning errors.

    The integration was reworked against the GA contract, and the provider was kept behind its own stage so a change like this stays contained.

  3. 03The document parser on App Service had to handle large, slow documents alongside quick ones without one blocking the other.

    The parser runs under Gunicorn with Uvicorn workers, so several async worker processes share the load.

  4. 04Latency and cost both grow with every step in a multi-step pipeline.

    Routing and confidence thresholds keep expensive models off documents that don't need them, and asynchronous processing keeps large files off the fast path.

05

Results and lessons

Where it stands

  • Processing 70K+ documents a month in production
  • Agents get structured fields and a summary in CRM instead of reading every document by hand
  • Low-confidence documents are routed to people with the reason attached, and the rest go straight through
  • Releases go out as canaries and can be rolled back by shifting traffic

What I’d tell the next team

  • Treat evaluation as product infrastructure, not a one-time test before launch.
  • Use routing and confidence thresholds before invoking expensive models.
  • Design agent workflows around clear validation contracts, so every failure has a name.