← Back to site
A layer-by-layer look at how modern language models are built and trained, what degrades them, how they are tested, and how enterprise data systems and agents connect to them. Eleven interactive modules. Open and closed models compared on the same axis.
Every model here is a decoder-only autoregressive transformer. That includes Claude, GPT, Gemini, Grok, DeepSeek, Qwen, Llama and Gemma. Click any component for its mechanism, its formula, and where the design choices diverge.
This section applies to closed models as much as open ones. The components below are common to all of them. What is not public for Claude, GPT, Gemini and Grok is the configuration: how many layers, how wide the residual stream is, how many heads, dense or sparse, and how many parameters. Wherever a specific number appears in this report it comes from a published technical report, which is why the named examples are open-weights models.
Reading several years of technical reports back to back, the core block is unchanged: pre-norm, attention, gated feed-forward, residual. What changed is the efficiency work around it: the KV cache, the sparsity pattern, the numeric precision and the training recipe. The intelligence gains of the last two years came overwhelmingly from data quality and post-training, not architecture. Worth establishing before any business case rests on architecture selection being the decisive variable.
Two things happen in the attention sublayer that matter in production. Specific heads perform identifiable, testable functions. And the key-value cache they populate is the largest serving cost at any realistic concurrency.
The head taxonomy below is general. It was established on models researchers could open up, including GPT-2, Claude 3 Sonnet and Claude 3.5 Haiku through Anthropic's own interpretability work, and it holds across architectures. The efficiency ladder names open models in its last column because those are the ones that publish their attention configuration. All closed frontier models almost certainly sit somewhere on the same ladder, since full multi-head attention at 1M context would price far above what they charge.
| Variant | Mechanism | Cache reduction | Quality cost | Used by |
|---|---|---|---|---|
| MHA | Every query head has its own K and V head | 1× | Baseline | GPT-2/3 era, now rare |
| MQA | All query heads share a single K/V head | n_heads× | Measurable, plus instability | Largely superseded |
| GQA | Query heads grouped; each group shares one K/V head | 4–8× | Negligible | Qwen3, gpt-oss, Llama, Mistral, Gemma |
| MLA | K/V compressed to a low-rank latent, reconstructed on the fly. Needs decoupled RoPE because rotation is position-dependent and compression is not | ~10× | Reported none | DeepSeek V3/V4, Kimi K2 |
| Sliding window | Each token attends to the previous W tokens; receptive field grows as L×W across layers. Needs attention sinks or the model destabilises when the first token falls out of window | seq/W× | Small, task-dependent | gpt-oss (1:1 interleave, W=128), Gemma |
| NoPE interleave | Layers with no positional encoding at all, alternating with RoPE layers, plus inference-time attention temperature scaling | n/a | Length generalisation gain | Llama 4 (iRoPE, ~1 in 4 blocks) |
| Linear / hybrid | Fixed-size recurrent state that does not grow with sequence length, interleaved with a minority of full-attention layers | O(1) in length | Still being established | Qwen3-Next (Gated DeltaNet), Kimi K3 (KDA, 3:1) |
Llama 4 Scout advertises 10M tokens but was pre-trained and post-trained at 256K. Liu et al. documented the lost-in-the-middle effect: performance is highest when relevant information sits at the beginning or end and degrades significantly in the middle, even in models explicitly built for long context. Retrieval heads explain why. Long-context recall depends on a small, specific set of heads rather than being a general property of the model. Anthropic's own long-context retrieval score moving from 18.5% on Opus 4.5 to roughly 76% on 4.6 makes the point: the window did not change, the retrieval behaviour did. Test on your own documents with synthesis across separated passages, not needle-in-a-haystack.
Mixture-of-experts splits the feed-forward layer into many independent experts and routes each token to a handful. Total parameters set your memory bill. Active parameters set your compute bill and latency. Those numbers have now diverged by up to 32×.
The frontier models are in the selector below, but with no numbers. Anthropic, OpenAI, Google and xAI do not publish parameter counts, expert counts or routing for their production models. Select one to see what can be inferred and what cannot.
Each square is one expert in one MoE layer. Filled squares compute. Blue is an always-on shared expert.
Shared experts. DeepSeek and Kimi keep one expert always active on the theory that general capability should not be routed. Qwen3 dropped shared experts, unlike Qwen2.5-MoE, to force specialisation. When JetBrains scaled a Qwen3-style architecture down for Mellum2 they found a shared expert gave no measurable quality gain and consistently hurt inference performance from the always-on FFN compute. The answer is scale-dependent.
Load balancing. Routers collapse if left alone. A few experts get slightly better, attract more tokens, improve faster, and the rest stop learning. The classic fix is an auxiliary loss, but it competes with the language modelling objective. DeepSeek-V3 pioneered an auxiliary-loss-free strategy using a learnable per-expert bias adjusted dynamically to equalise load, with no gradient conflict. Qwen3 took the other path with a global-batch balancing loss.
All experts must be resident in memory even though only a fraction compute. DeepSeek-V3 at BF16 needs well over a terabyte of weight memory. You pay the memory of a 671B model and get the compute cost of a 37B model.
Excellent at high utilisation on a multi-node cluster. Poor for a single node at low request volume, which describes most enterprise deployments. This is the most common sizing error in on-prem LLM projects.
Note the counter-examples: Mistral Medium 3.5 is dense at 128B and currently the only frontier-credible non-MoE model, and Gemma 4 31B is dense and leads small-footprint deployments. For a fixed single-node budget, dense often wins.
A model is grown rather than built, and the recipe matters more than the architecture. Click any stage. The post-training stages are where the last two years of capability gains were manufactured.
| Method | Mechanism | Cost | When to use it |
|---|---|---|---|
| RLHF / PPO | Human comparisons train a reward model; policy optimised against it with PPO, KL-constrained against the SFT model to prevent collapse | Highest. Reward model plus critic plus rollouts | Frontier labs with dedicated RL infrastructure. Rarely correct for an enterprise |
| DPO | Closed-form loss directly on preference pairs. No separate reward model, no RL loop | Low. Looks like supervised training | The default for enterprise preference alignment. Slightly lower ceiling, vastly simpler |
| GRPO | Sample multiple completions per prompt; the group mean reward is the baseline, replacing a learned value network entirely | Medium. No critic, so much lower memory | The method behind the reasoning-model wave. Needs a reward signal |
| RLVR | Where correctness is checkable, the verifier is the reward. No reward model, no human labellers | Low per-sample, high engineering | The one with real differentiation potential for a data practice. See callout below |
| Rubric / self-critique | Extends verifiable rewards to unverifiable tasks via a model-generated rubric score. Kimi K2 combines both in one framework | Medium | Subjective quality dimensions where no verifier exists |
| Distillation / MOPD | Train specialised experts then merge. Kimi K3 trained nine task-specific RL models at different reasoning efforts and merged via Multi-Teacher On-Policy Distillation | High to produce, cheap to serve | Compressing a large capable model into a servable one |
Where a task has a checkable answer, you do not need a reward model or human labellers. The verifier is the reward signal. This is why reasoning models improved so fast: it removes the human bottleneck so the loop runs at machine scale. For data engineering workloads, verification is often mechanically available. Does the generated SQL execute? Does it return the expected row count? Does the extracted field match the source system? Does the lineage graph resolve? Does the migrated report produce identical numbers to the original? If you can define correctness, you can improve a model on your own workloads without labelling anything. That is worth more than fine-tuning on a pile of documents.
The least visible stage, the most decisive one, and the last real trade secret in the field. Scale: Qwen3 used ~36T tokens across 119 languages; DeepSeek-V3 14.8T; Kimi K2 15.5T; Kimi K3 with a progressive context-length curriculum; Llama 4 ~30T.
Pipeline: bulk crawl, language ID, dedup at document and near-duplicate level, quality classification, toxicity and PII filtering, then deliberate up-weighting of code and mathematics far beyond natural web frequency because both give dense verifiable reasoning signal.
Your domain is underrepresented. Insurance policy language, clinical coding, trade settlement messaging, utility asset hierarchies are not in web crawl. That is the root of most "the model does not understand our business" complaints. It is a data problem, not an architecture problem.
AdamW is standard. DeepSeek-V3 used β₁ 0.9, β₂ 0.95, weight decay 0.1. Two optimizer states per parameter, so 8 bytes per parameter at FP32 on top of weights and gradients.
The interesting exception is Muon, used by Kimi K2 for better token efficiency. At scale it hit a wall: attention logits rapidly exceeded 1000 mid-run, risking divergence. The fix was MuonClip, adding QK-Clip to rescale query and key projections. With it they pre-trained a trillion-parameter model on 15.5T tokens with zero loss spike.
A clean illustration that architecture, optimizer and normalisation are not separable concerns. QK-Norm solves the same problem structurally.
Five strategies compose. Data parallel splits the batch (gradient all-reduce per step). Tensor parallel splits individual matrices (all-reduce inside every layer, so very high traffic, keep it intra-node). Pipeline parallel splits layers (activations at stage boundaries, creates bubble idle time). Expert parallel distributes MoE experts (all-to-all token routing). Sequence/context parallel splits sequence length.
DeepSeek built DualPipe specifically to overlap computation with communication and hide the pipeline bubble, and hand-wrote cross-node all-to-all kernels because the off-the-shelf ones bottlenecked.
Their reported 2.788M H800 GPU hours (~$5–6M) is the marginal cost of the final run, not of the capability. It excludes failed runs, data pipeline construction, staff, and the ablations that found the recipe.
Quantization is how a model that needs 240GB fits on one 80GB card. It is also where quality is lost in ways that benchmarks often miss. This section covers what breaks, why, in what order, and how to get the loss back.
Dettmers et al. established that activation outliers emerge suddenly around 6.7B parameters, affecting all transformer layers and roughly 75% of sequence dimensions. They are about 0.1% of features, yet zeroing them collapses top-1 attention softmax mass by over 20% and degrades validation perplexity by 600–1000%. Formally: dimensions with magnitude ≥6.0 in ≥25% of layers and ≥6% of sequence positions. Mechanically they stretch the quantization range so every other value compresses into a handful of buckets. This single phenomenon explains why weights quantize easily and activations do not, and why W4A16 is near-lossless while W4A4 remains hard.
| Method | Mechanism | Handles outliers by | Practical note |
|---|---|---|---|
| LLM.int8() | Mixed-precision decomposition: the 0.1% outlier dimensions stay in 16-bit, the other 99.9% multiply in 8-bit | Isolating them | ~1.96× memory reduction on BLOOM-176B with essentially no loss. The reference for why 8-bit is free |
| SmoothQuant | Migrates quantization difficulty from activations to weights via per-channel scaling before quantizing | Moving the problem | Enables W8A8, which is what makes 8-bit activations viable at all |
| GPTQ | Layer-wise weight updates using approximate second-order (Hessian) information to minimise reconstruction error | Compensating for them | Strong at 4-bit, extends to 3-bit. Sensitive to calibration data |
| AWQ | Identifies the ~1% salient weight channels by activation magnitude and protects them with per-channel scaling | Protecting what they touch | Activation-aware, so calibration data quality matters directly |
| MXFP4 | Two fp4 (e2m1) values per byte with a shared power-of-two scale per 32-element group. 4.25 bits per parameter | Group-wise scaling | Used natively by gpt-oss and Kimi K3, applied during post-training rather than after, which is why it holds quality |
| NF4 (QLoRA) | Each value assigned to one of 16 quantile bins of a normal distribution; double quantization compresses the scales to 8 bits | Distribution-matched bins | Built for fine-tuning on quantized frozen weights, not primarily for serving |
| AQLM / 2-bit | Additive quantization for extreme compression | Poorly, alone | Accuracy degrades substantially below 3 bits. Needs a recovery mechanism to be usable |
Post-training quantization is training-free and cheap, and it leaves quality on the table. There are four established ways to recover it, in ascending cost.
The cheapest lever and the most neglected. GPTQ and AWQ both depend on a calibration set to estimate activation distributions. Generic calibration data on a domain-specific workload produces biased quantization parameters. Calibrate on your own representative traffic. Recent work on family-aware calibration regenerates calibration samples using a larger model from the same family. The samples carry chain-of-thought and match the expected activation distribution. This reported up to 28.5% less accuracy loss than baseline calibration data on Qwen3-8B.
Fine-tune the quantized model with the same loss function, quantizing weights and activations in the forward pass while gradients stay high-precision. PyTorch reported QAT recovering 62% of the normalized accuracy degradation on hellaswag and 58% / 57% of word and byte perplexity degradation on wikitext, versus PTQ, for Llama 8da4w. Cost is a retraining run, which is why parameter-efficient variants exist.
Train a low-rank adapter on the quantized model, distilling from the full-precision original on synthetic data. This is the practical recovery mechanism for aggressive quantization. Recent work shows selective 2-bit quantization of gate and up projections delivering 7.5 to 23.3% throughput gains over uniform W4 across 4B to 20B models, with Recover-LoRA reclaiming the accuracy that would otherwise be lost. Far cheaper than full QAT.
For models whose capability came from RL post-training, plain QAT is awkward, because quantization-aware RL is barely explored. QAD distils from the full-precision model instead, which sidesteps the problem. This matters increasingly because reasoning capability is RL-derived, and it is exactly the capability most damaged by aggressive quantization.
Safety degrades separately from capability. Research on quantized model safety found that quantization can degrade alignment even where utility benchmarks hold, and that calibration data with harmful content makes it worse. Quantization-aware safety patching exists specifically to re-align quantized models. If you quantize a model and only measure task accuracy, you have not measured what changed.
Degradation is not uniform across tasks. A model that holds its benchmark scores at 4-bit can still fall apart on your specific structured extraction task, on long-chain arithmetic, or on precise code generation. Per-channel quantization vulnerability is even model-specific. The Llama3-70B series showed a distinct susceptibility traced to weight outliers in its early layers. Llama2, the 8B variants and the 405B did not share it. Test on your workload. Never on someone else's leaderboard.
Public benchmarks are useful for rough ranking and close to useless for choosing a model for your own work. Without your own evaluation set you cannot tell whether any change you made helped. With a new release roughly every week, model swaps are now routine.
Evaluation without instrumentation is guesswork. Standardise on OpenTelemetry GenAI semantic conventions for spans and attributes. Track token accounting including hidden reasoning tokens. They bill as output and do not appear in the response body, so a cost model built on visible output length will be wrong by an order of magnitude on reasoning models. Decompose latency into TTFT (prefill) and TPOT / inter-token latency (decode), The two have different fixes. TTFT regressions point at oversized context. Inter-token latency regressions point at memory bandwidth and batch composition.
Open and closed on one axis. Open-weights rows are verifiable from published technical reports; closed rows carry no architecture figures because none are published. Anything circulating that claims them is speculation. Click any column header to sort.
| Model | Access | Total | Active | Sparsity | Attention | Context | $ In | $ Out | Licence |
|---|
Sparsity = total ÷ active; dense shows 1×. Prices are per million tokens as of late Aug 2026 and move roughly monthly. OpenAI repriced twice in one month. Open-weights prices are representative hosted rates, not your self-hosted cost. Verify the licence on the exact checkpoint: terms have changed within families, and Qwen variants above the 27B tier now carry commercial restrictions.
Reasoning tokens bill as output on both platforms. You pay for the full internal chain though the API returns only a summary. Anthropic hides raw thinking behind opaque signatures to prevent distillation. A hard task can emit 5–50× the visible output. Claude 4.7+ changed tokenizer and produces roughly 30% more tokens for the same text, so nominal rate comparisons against older models understate cost. And OpenAI applies a 2× input / 1.5× output surcharge on the entire request once input exceeds 272K tokens. That applies to the whole request, not only the part above the limit.
Not public: parameter counts, layer counts, attention configuration, expert counts and routing, training data composition, tokenizer details, training compute.
Reasonable inference: all are decoder-only with pre-norm, RMSNorm or equivalent, gated FFN, rotary-derived position encoding. All flagships are almost certainly sparse MoE, because the pricing and latency profiles do not fit dense models at that capability level. All use grouped or latent attention; serving cost at long context makes full MHA implausible. All offer variable inference-time compute, implying reasoning-mode training.
The window into each: for Anthropic it is the interpretability work. They trained sparse autoencoders with up to 34 million features on the middle-layer residual stream of a stated production model. Attribution graphs then showed two-hop reasoning with a manipulable intermediate step, and forward planning in poetry. For OpenAI it is gpt-oss, which reveals the design conventions: attention sinks, sliding/full interleave, MXFP4, Harmony, o200k.
Anthropic RSP v3.0 (Feb 2026) defines AI Safety Levels with Deployment and Security standards. It is outcome-focused rather than prescriptive, and adds Frontier Safety Roadmaps and recurring Risk Reports. ASL-3 active since May 2025 for CBRN.
OpenAI Preparedness v2 (Apr 2025) tracks three categories: Biological/Chemical, Cybersecurity, and AI Self-improvement. Two thresholds, High and Critical. GPT-5.6 Sol assessed High, not Critical, for cyber.
Data residency. Anthropic inference_geo: "us" applies a 1.1× multiplier on Claude 4.6+. Bedrock and Vertex have independent regional endpoints at roughly +10%. ZDR available on both.
The precedent worth modelling. Claude Fable 5 was suspended on 12 June 2026 under a US export-control directive and restored on 1 July. First frontier model switched off and back on by regulatory action. A capability you depend on can vanish for reasons outside your vendor relationship, your SLA and your control.
Vendor-neutral reference architecture. From a document landing in a bucket to a token arriving in a response. Every stage, what it mechanically does, and where it fails.
Contextual retrieval prepends a 50–100 token LLM-generated blurb situating each chunk in its document, before both embedding and BM25 indexing. Measured effect on top-20 retrieval failure rate:
One-time contextualisation costs roughly $1.02 per million document tokens with prompt caching enabled. Hybrid search fuses dense embeddings with BM25 at roughly 0.8 / 0.2 weighting, then a cross-encoder reranker rescores the top-20 down to the 3–5 chunks actually injected. BM25 catches the exact matches that dense retrieval reliably misses: part numbers, policy codes, ISINs and dates. Those are the tokens enterprise queries turn on.
| Layer | What it does | Failure mode it prevents |
|---|---|---|
| Constrained decoding | A grammar compiled from the schema masks non-conforming tokens at sampling time. Implementations include JSON mode, strict schema, XGrammar, Outlines, Guidance and llama.cpp grammars. Output is valid by construction, not validated afterwards | Malformed output breaking parsers. Note: structure guaranteed, semantics not |
| Semantic layer | Model queries governed metric and dimension definitions rather than raw schemas; emitted query is parsed, dry-run or EXPLAINed, then executed with row and cost guards | Hallucinated joins and invented metric definitions. The most effective anti-hallucination control for analytics agents |
| Input guardrails | PII detection and redaction, prompt-injection and jailbreak classifiers, before the model sees the request | Injection reaching the model; regulated data leaving the estate |
| Output guardrails | PII, toxicity, policy and groundedness or citation checks after generation | Ungrounded assertions reaching a user. Never rely on model alignment alone |
| Semantic cache | Embed the query, serve a cached response for semantically similar prior queries | Repeat cost on near-duplicate questions. Watch staleness and false hits |
| Identity propagation | OAuth token exchange so the agent acts with the user's scoped permissions; identity passed to the data layer so row-level security applies at query time | The confused deputy problem. See section 09 |
| Gateway / router | One abstraction across providers, centralised auth and key management, model routing, fallback chains, canary rollout | Provider lock-in and single-provider availability risk |
Three things separate a working agent from a demo: how it remembers across sessions, how it is instructed, and how it talks to other agents and to your systems.
Collapsing all memory types into a single retrieval problem. Working memory is a context-budget problem, not a retrieval problem. There is nothing external to retrieve from, so manage it by compression and prioritisation. Running semantic similarity search across episodic logs is the specific mistake. Episodic memory is indexed by time and cause, not by meaning. Treating "what did I try last Tuesday" as a nearest-neighbour question returns results that look right and are not. Different memory types require different retrieval logic. Most production systems implement two layers well and handle the transitions between layers with crude heuristics.
Instruction reaches a model through four distinct channels with different persistence, cost and cache behaviour. Confusing them is why agents behave inconsistently across sessions.
| Channel | Persistence | Cache behaviour | What belongs here |
|---|---|---|---|
| System prompt | Every request, whole session | Stable prefix. Cache at 0.1× | Role, constraints, output conventions, refusal boundaries. Keep it stable; every edit invalidates the cache for everything after it |
| Tool schemas | Every request while tools are available | Stable prefix, but changing mid-conversation historically invalidated the cache. Opus 5 now permits tool changes without invalidating it | Capability surface. Note the overhead: 286–406 tokens of injected system prompt on Opus 5 just for having tools, plus ~4,500 for the computer-use toolset and ~6,600 for browser use |
| Skills | Loaded conditionally, on relevance | Loading one mid-session breaks the cache prefix unless skills sit in a stable block | Procedural memory made explicit. Reusable task recipes, house style, review checklists, domain workflows. The right home for "how we do this here" that would otherwise be re-explained in every prompt |
| Retrieved context | Per request | Never cacheable. Always fresh | Facts. Not behaviour, not procedure. Retrieval solves knowledge; it does not solve how the agent works |
The architectural point. Procedural memory is undergoing a transition from explicit non-parametric templates toward implicit parametric policies. What lives in a skill file today may live in a fine-tuned executor model tomorrow. In practice today it manifests as skill libraries and workflow templates, and the deciding question is whether a behaviour needs to be inspectable and editable by a human (keep it a skill) or fast and consistent at volume (distil it into weights).
Four protocols address four different layers. They are complementary, not competing, and the sensible enterprise path is phased adoption rather than picking one.
| Protocol | Layer | Mechanism | Enterprise position |
|---|---|---|---|
| MCP | Agent ↔ tool | JSON-RPC 2.0. Hosts run clients connecting to servers exposing tools, resources and prompts, plus sampling. Two official transports: stdio for local subprocesses, Streamable HTTP for remote (POST+GET, optional SSE, Mcp-Session-Id, OAuth). The older HTTP+SSE transport is deprecated | Stage 1, adopt now. Servers must validate the Origin header against DNS rebinding. Solves connectivity, not governance |
| A2A | Agent ↔ agent | Google, contributed to the Linux Foundation 2026. Agents discover each other via Agent Cards, which are JSON metadata files at a public URL describing capabilities, skills, endpoint and auth. Core objects: Agent Card, Task, Message, Artifact. Sync, streaming and async push. Delegation-centric: answers "which agent can handle this task?" | Stage 3, for cross-team and cross-vendor work. Preserves opacity, so agents collaborate without exposing internal memory, logic or tools. v1.0.1 added an extension mechanism |
| ACP | Agent ↔ agent, negotiated | IBM Research, drawing on FIPA-ACL heritage. Formalises multi-agent communication with typed performatives (propose, accept, reject, counter) and multi-turn negotiation semantics. Merged under the A2A umbrella in 2025 | Stage 2. Useful where agents have to negotiate rather than simply hand off work |
| ANP | Open agent markets | Decentralised discovery, DID-based identity verification, peer-to-peer communication over JSON-LD graphs | Stage 4. Not yet an enterprise concern; watch it |
Recent analysis of governance gaps found that MCP, A2A and ACP all lack the vocabulary to express policy constraints, data handling obligations or accountability. A2A's four official extensions do not close this: Secure Passport, Timestamp, Traceability and Agent Gateway. Gartner projects that 60% of agentic analytics projects relying solely on MCP will fail by 2028 without semantic foundations, and that by 2030 half of enterprise agent deployment failures will trace to insufficient governance runtime enforcement rather than capability gaps. The protocol layer is necessary and nowhere near sufficient.
Durable execution matters more than topology. Long-running agents need checkpointing and resume. A four-hour run that dies at minute 200 with no saved state is worse than no agent at all.
The hardest unsolved problem in enterprise agent architecture. A privileged agent is tricked into using its authority on the attacker's behalf, usually via prompt injection embedded in a document or an MCP tool result. The agent is not compromised. It is doing exactly what it was told to do, by the wrong party.
Service accounts make this worse. An agent with broad credentials can read everything, so any successful injection reaches everything. A2A widens the exposure again, because a remote agent's output is now untrusted input to yours.
Layered defence. Delegated auth via OAuth token exchange so the agent carries the user's scoped permissions. Identity propagated to the data layer so row-level security applies at query time. Least-privilege scopes per tool. Human-in-the-loop for high-impact actions. And an explicit invariant: content arriving from retrieval, tools or other agents can never change the agent's permission set. Design for this before the first agent reaches production.
There is no single answer to open versus closed. There is an answer per workload, and a short list of patterns that hold up.
Structured fields from documents at millions of pages a month. Repetitive, verifiable, latency-tolerant.
Multi-step tool use, code generation, long-horizon planning across systems. Low volume, high value per task.
Reasoning over data that cannot leave the estate. BFSI, healthcare, public sector, sovereignty-constrained.
Internal assistant over enterprise content. Bursty traffic, wide query variety, moderate stakes.
Translating between platforms. Verifiable output, project-shaped rather than continuous volume.
Constrained hardware, intermittent connectivity, or per-device deployment.
Ascending cost. Finish each rung before climbing. Most teams jump straight to fine-tuning because it feels like real engineering, and most of those projects could have been solved at rung one or two.
| # | Factor | Typical failure |
|---|---|---|
| 1 | Right data at the right moment | Blamed on the model; it is a retrieval problem |
| 2 | Task decomposition | One impressive leap instead of reliable steps |
| 3 | Verification of outputs | Generated artefacts trusted directly |
| 4 | Evaluation on real workloads | Public benchmarks used as a proxy |
| 5 | Memory and instruction architecture | All memory types collapsed into one retrieval problem |
| 6 | Cost and latency fit | Sized for weights, not KV cache |
| 7 | Raw model capability | Over-weighted in almost every proposal |
Architecture sets your cost structure and your deployment limits. It does not decide whether the system works.
| Technique | What it does | Where it pays |
|---|---|---|
| Continuous batching | Iteration-level scheduling; requests join and retire mid-flight rather than waiting for a batch boundary | Universal. The default primitive in vLLM and SGLang |
| PagedAttention | Manages KV cache as virtual memory pages, eliminating fragmentation and enabling large flexible batches | High-throughput multi-user serving |
| RadixAttention | Radix tree caches and shares KV state across requests with matching prefixes, LRU eviction | Long shared system prompts; reported 30–50% savings on 2K+ prefixes |
| Chunked prefill | Mixes prefill chunks with decode tokens to relieve prefill/decode interference (Sarathi-Serve) | Mixed short and long request workloads |
| P/D disaggregation | Separates prompt encoding from generation across node pools to co-optimise TTFT and TPOT independently | Cluster scale, tight latency SLOs. Moonshot's Mooncake reports ~90% cache hit on coding workloads |
| Speculative decoding | Draft k tokens, verify in one parallel pass via rejection sampling, which is lossless. EAGLE-3 autoregresses at feature level with dynamic draft trees; DeepSeek's MTP head bakes it into the model | 2–3× throughput on the same hardware, no quality loss. The least used optimisation in enterprise deployments |
The architectural gap between a frontier closed model and a strong open model is small and narrowing. The gap in data curation, RL infrastructure and post-training scale is large. But neither one decides whether an enterprise deployment succeeds. That is decided by data access, decomposition, verification, evaluation, and the memory and instruction architecture around the model. That is the work a data practice is already good at, and the market currently assumes the opposite.