A reference walkthrough: document backlog under data-residency constraints
This is a reference pattern, not an account of a specific client. It describes a shape of problem we are asked about often enough to be worth writing down, and the architecture decisions that follow from it. Where a decision could reasonably go the other way, that is said rather than smoothed over.
The shape of the problem
A public-sector organization holds a large backlog of submitted documents — applications, correspondence, supporting evidence — arriving faster than they can be read. Three constraints shape everything that follows. The data cannot leave the jurisdiction. Decisions influenced by the system are appealable, so their basis must be reconstructable months later. And the organization has a small platform team with no machine-learning specialists.
That third constraint disqualifies more architectures than the first two. A design that assumes someone will tune a model, curate an evaluation set, and watch a dashboard is not a design for this organization.
What the system is allowed to do
The first decision is scope, and it is the one that determines whether the project survives. The system does not decide. It extracts structured fields, flags apparent inconsistencies, and drafts a summary for a caseworker. Every output is a proposal attached to a document, and a person accepts, edits, or rejects it.
This is not timidity. An appealable decision needs a human who can explain their reasoning, and "the model suggested it and I agreed" is not an explanation that survives scrutiny. Constraining the system to proposals keeps the accountability where it already legally sits, and has the useful side effect of generating a labelled dataset — every acceptance and edit is a judgment about output quality.
Every correction a caseworker makes is a labelled example. A system that discards those is throwing away the only evaluation data it will ever get for free.
The stack
Residency settles the model question: open weights, self-hosted, in-jurisdiction. A mid-sized instruction-tuned open model is sufficient for extraction and summarisation over a known document taxonomy — this is not frontier work. Serving through vLLM on a modest GPU allocation, sized for steady throughput on a backlog rather than for interactive latency, because nobody is waiting on a chat response.
Retrieval over Postgres with pgvector rather than a dedicated vector database. The corpus is large but not enormous, the team already operates Postgres, and adding a datastore they cannot troubleshoot at 2am is a real cost against a marginal benefit. Authorization is evaluated at retrieval time against the requesting caseworker, so the assistant cannot surface a document the person could not have opened themselves.
Every inference is traced — input, retrieved context, output, model version, and the caseworker’s eventual disposition. This is the appeal record. It is also the evaluation dataset, and treating it as one artefact rather than two is what makes ongoing quality measurement affordable for a team with no capacity to build a separate one.
What goes wrong
Three failure modes are worth naming because they recur.
Automation creep. The system proves useful, the backlog stays large, and someone proposes auto-accepting high-confidence extractions. Confidence scores from a language model are not calibrated probabilities, and the appeal exposure of a wrong auto-accepted decision is categorically different from a wrong suggestion a person approved. If this threshold is ever crossed, it should be a governance decision with a named owner, not a configuration change.
Silent degradation. Document formats change, an upstream form is redesigned, and extraction quality falls without any error being raised. Nothing breaks; the outputs are simply worse. Only sustained measurement against caseworker corrections catches this, which is the argument for building that loop before it is needed rather than after.
Retrieval leakage. The index is built with a service account that can read everything, while the people querying it cannot. A permission error that previously required knowing a file path now requires only knowing how to ask a question. This is the failure most likely to become an incident, and it is an architecture decision rather than a model one.
Why this shape holds up
The architecture is unremarkable, which is the point. It uses components the team can already operate, keeps data inside the boundary the law draws, puts a person on every consequential decision, and produces its own audit trail as a by-product of normal use. There is no step in it that requires a specialist the organization does not have.
Ambitious AI programs in constrained environments do not usually fail on model quality. They fail because the design assumed capacity that was not there, or because accountability was never located in a person, or because nobody could say afterwards why the system did what it did. Those are architecture and governance problems, and they are solvable before a single model is chosen.
