Architecture

Building on open weights: a reference architecture

Sep 2026 · 13 min read

For two years the honest answer to "should we use an open model?" was usually no. The capability gap was wide enough that the question answered itself. That is no longer true for most enterprise workloads, and the organizations still asking it are asking the wrong question. The interesting decision was never raw capability. It is who controls the roadmap you are building on, where your data goes when you use it, and what happens to your architecture when a price list or a licence changes.

What actually changed

Three things moved at once. Open-weight families — Llama, Mistral, Qwen, Gemma, DeepSeek among them — became genuinely competent at the tasks enterprises actually run: extraction, classification, summarisation, retrieval-grounded question answering, structured output. Serving stacks matured, so running a model efficiently stopped being a research project. And the tooling around models — evaluation, tracing, policy enforcement — caught up enough that you can now operate one responsibly rather than merely deploy one.

The frontier is still ahead on the hardest reasoning tasks, and for some work that gap matters. But most production AI in an enterprise is not frontier work. It is a well-scoped task, run at volume, against sensitive data, where predictability and cost per unit matter more than the last few points of benchmark performance.

Most production AI in an enterprise is not frontier work. It is a well-scoped task, run at volume, against sensitive data.

The layers

A workable open stack has six layers, and the mistake we see most often is treating it as one. Teams adopt a model, wire it directly into an application, and discover eighteen months later that they have no way to evaluate a change, no record of what the system did, and no way to swap the model without a rewrite.

Weights. The model itself, with a licence you have actually read. Licences in this space vary more than people assume: some are genuinely permissive, some carry usage thresholds or field-of-use restrictions that matter at scale. This is a legal review, not a technical one, and it belongs at the start rather than after a pilot succeeds.

Serving. vLLM and SGLang for throughput on GPUs; llama.cpp and Ollama where the constraint is a single machine or an air-gapped environment. This layer decides your cost per token and your latency profile, and it is where most of the operational surprises live — batching behaviour, memory pressure, and what happens under concurrent load.

Retrieval and data. Postgres with pgvector where the volume is moderate and you would rather not run another datastore; Qdrant, Weaviate, or OpenSearch where scale or hybrid search justifies one. The important decision here is not which vector store. It is whether authorization is evaluated at retrieval time against the requesting user, which is the difference between a search tool and a data breach with a natural-language interface.

Orchestration. LangGraph, LlamaIndex, Haystack, or nothing at all. Framework choice matters far less than whether the control flow is legible — whether someone can read the code and say what the system will do with a given input. A great deal of production AI would be improved by deleting a framework rather than adding one.

Evaluation and observability. Langfuse and Phoenix for tracing and scoring; OpenTelemetry’s generative-AI conventions where you want this to land in the observability stack you already run. This layer is skipped more often than any other, and its absence is the single strongest predictor that a system will silently degrade without anyone noticing.

Policy. Open Policy Agent for authorization decisions expressed as code rather than scattered through application logic. Once several services need to agree on who may ask what of which data, having that in one place stops being tidiness and starts being the only way to answer an auditor.

What open source does not solve

Adopting an open stack removes a vendor dependency. It does not remove the hard parts, and it is worth being blunt about which problems remain exactly as difficult.

Evaluation is still unsolved for most teams. Knowing whether a change made the system better requires a dataset that reflects your actual work, and building that dataset is unglamorous, expensive, and cannot be bought. Data quality is unchanged: a model grounded on a corpus nobody owns will produce confident answers from documents nobody has checked. And accountability does not move — if a model-driven decision affects someone, a named person is answerable for it whether the weights were open or rented.

Open weights remove a vendor dependency. They do not remove the obligation to know whether the thing works.

When open weights are the wrong call

Three cases, honestly. When the task genuinely needs frontier reasoning and the gap is decisive. When the team has no operational capacity to run inference and no realistic path to acquiring it — a self-hosted model nobody can keep running is worse than an API. And when the volume is low enough that per-token cost never becomes the dominant term, in which case self-hosting is a fixed cost bought to avoid a variable one that was never large.

The strongest architectures we see are not purist. They route by task: open weights for the high-volume, sensitive, well-scoped work, an API for the long tail where capability matters more than control, and a serving interface that makes the boundary between the two an implementation detail rather than a rewrite.

Where this connects to governance

An open stack makes governance more achievable and more necessary at the same time. More achievable, because you can inspect what the system did, log it into infrastructure you control, and enforce policy at the boundary rather than trusting a provider’s terms. More necessary, because nobody else is doing it for you: the guardrails a hosted provider applies by default become your responsibility the moment you take the weights in-house.