This article assumes internal DNS, private IP ranges, switches, routing, firewalls, and datacenter networks. I covered those from first principles in From a Home Router to a Hospital Network — if "the AI zone has no outbound route" does not yet mean something concrete, start there.
The requirement
A hospital wants an internal assistant: doctors ask questions in natural language — clinical policy, drug interactions, internal protocols — and get answers grounded in the hospital's own documents, with citations.
One constraint: patient and clinical data must never leave infrastructure the hospital controls.
That sentence disqualifies almost every obvious implementation.
Why the obvious answer is already gone
OpenAI GPT-4 API ❌
Claude API ❌
Gemini API ❌
Groq API ❌
Not because they are insecure, but because using them means:
Hospital data → external inference provider
The moment a doctor's question — clinical context, often patient specifics — crosses the perimeter firewall to a third party, the requirement is violated. Zero-retention agreements change the legal posture, not the network path. The data left.
So inference has to run on hardware the hospital owns: a self-hosted open-weight model from the Qwen, Llama, or Mistral families, chosen on evaluation against the hospital's own questions and on license terms. That selection is a project of its own; here the only thing that matters is that the weights sit on a disk in the hospital's datacenter.
And "install GPT-4 on our GPUs" is not a deployment task someone forgot to schedule — those weights are not distributed. Say that in the first meeting, not three weeks in.
Define "air-gapped" before designing anything
The word covers two materially different architectures, and the cost difference is enormous.
Definition 1 — internet-isolated AI network. The AI environment sits on the hospital network, reachable by staff, but has no route to the internet.
Doctors
↓
Hospital LAN
↓
AI Datacenter
✗
Internet
Definition 2 — true physical air gap. No physical connection to the general hospital network at all.
Normal Hospital Network
✗
✗ no physical link
✗
Dedicated AI Network
The second is real — defense, some critical infrastructure — but doctors then cannot reach the system from their normal laptops; they need dedicated terminals on the isolated network. That is a different product, not the same one with a stricter firewall rule.
Nearly every hospital asking for "air-gapped AI" means the first, but "nearly every" is not a basis for a build: access model, update process, layout, and cost all branch on the answer. This article assumes Definition 1.
The architecture
Doctor
↓
Internal DNS
↓
Segmentation Firewall
↓
Internal Load Balancer
↓
API Gateway
↓
Frontend + IAM
↓
Backend
↓
RAG Orchestration
├──────────── Retrieval
│ ↓
│ Embedding Model
│ ↓
│ Vector Database
│ ↓
│ Relevant Context
│
└──────────── Generation
↓
Local LLM
↓
Answer + Citations
Every box runs on hospital hardware, addressed with private IPs, on a segment with no outbound route.
Why RAG needs two models
RAG is not "the LLM searches the documents." The LLM never searches anything. Two different models do two different jobs.
Model 1 — the embedding model. Turns text into a vector, so similar meanings land near each other in vector space.
"When should warfarin be stopped before surgery?"
↓
embedding model (BGE / E5 / similar)
↓
[0.19, -0.42, 0.73, ... ] 768–1536 numbers
That vector is the search key. Comparing it against every stored chunk's vector is what "finding relevant information" means.
Model 2 — the generative LLM. Reads the question plus whatever retrieval found, and writes the answer.
Question + retrieved context
↓
local LLM (Qwen 72B, or whatever wins the eval)
↓
Answer, grounded in the context, with citations
Optionally, model 3 — a reranker. Vector search is fast and approximate; a cross-encoder reranker is slow and accurate. Run the fast one wide, the accurate one narrow:
Top 30 results from vector search
↓
reranker
↓
best 5 chunks
The sentence that makes this stick:
The embedding model finds information. The LLM explains information.
Both have to run locally. A system that self-hosts the LLM but calls a hosted embedding API has not solved the problem — every question still leaves the building, through a different door.
Ingestion and retrieval are two separate pipelines
They run at different times, at different rates, with different latency budgets.
Ingestion — when documents arrive or change:
Hospital document (PDF, scan, policy doc)
↓
Parser / OCR
↓
Chunking
↓
Embedding model
↓
Vector database
Batch work: minutes per document is fine, it runs off a queue, nobody watching a spinner.
Retrieval — on every question:
Doctor's question
↓
Embedding model
↓
Vector search
↓
Relevant chunks
↓
Local LLM
↓
Answer
Interactive, with a doctor waiting.
The embedding model appears in both, and it must be the same model at the same version. Embeddings from different models are not comparable — vectors written by one and searched with another return noise. Changing it means re-embedding the entire corpus.
Authorization belongs inside retrieval, not after it
This is the most important design decision in the system, and the easiest to get wrong.
The naive retrieval query is:
SELECT chunk_text, document_id
FROM document_chunks
ORDER BY embedding <=> :query_embedding
LIMIT 10;It returns the ten chunks most similar to the question, and has no idea who asked. The nearest chunk might come from a document this doctor has no right to read — another department's records, HR material, a restricted protocol.
Once those chunks are in the context window, the damage is done. Filtering the output afterwards does not help: the answer is already synthesized from data the user was never authorized to see, and the citation list is not the only leak — the prose is.
Authorization has to be part of the search:
SELECT chunk_text, document_id
FROM document_chunks
WHERE department_id = ANY(:authorized_departments)
AND classification_level <= :user_clearance
ORDER BY embedding <=> :query_embedding
LIMIT 10;The ordering is not negotiable:
Authentication → who is this?
↓
Authorization → what may they see?
↓
Retrieval → search only within that
Never:
Retrieval
↓
"hope the user was allowed to see that"
Two consequences. Every chunk needs access-control metadata attached at ingestion, because you cannot filter on what you did not store. And the index has to support filtered search efficiently — a filter applied after an approximate-nearest-neighbor scan can silently return fewer results than requested, or none.
Every box, briefly
Internal DNS. Resolves hospital-ai.internal to the internal load balancer at 10.50.0.10. Private zone, no public record.
Segmentation firewall. Enforces the zone policy:
Doctors → AI application (HTTPS) ✅
Doctors → vector database ❌
Doctors → GPU inference nodes ❌
AI zone → Internet ❌
The AI zone is physically cabled to the core switch. Its isolation is a firewall rule, not an absence of wire — which is why that rule needs auditing.
Internal load balancer. Spreads requests across backends, drops a failed one from rotation.
Backend 1
/
Request → LB — Backend 2
\
Backend 3
API gateway. Routing, token validation, rate limits, TLS policy, request size limits, versioning — one place for cross-cutting rules instead of thirteen.
IAM. Identity and authorization context: department, role, clearance. Its output feeds the retrieval filter, which is why it sits upstream of everything.
Cache. Redis for sessions and permission metadata, possibly retrieval results. Cache retrieval carefully: a key that omits the user's permission set will serve one doctor's results to another.
Message queue. Drives asynchronous ingestion:
Upload → Queue → Worker → OCR → Chunk → Embed → Vector DB
Model storage. Weights, tokenizers, and configs for the generative model, embedding model, and reranker — what inference servers load at startup, not Hugging Face.
Vector database. Chunks, embeddings, and the access-control metadata that makes filtered retrieval possible. pgvector, Qdrant, Milvus — the choice matters far less than the metadata design.
Audit log. Who asked what, which documents were retrieved, what was answered, when. In healthcare that is a compliance requirement: append-only, outside the reach of the application writing to it.
How the model actually runs
A common misconception is that the model is a Python object inside the backend — load it in FastAPI at startup, call a method. That works on a laptop and fails in production: a 72B model does not fit in one process, GPU memory cannot be shared across web workers, and batching is the entire reason inference is affordable.
In production it looks like this:
GPU cluster
↓
vLLM (or TensorRT-LLM / TGI)
↓
Model weights loaded from internal model storage
↓
Internal inference API
http://llm.internal/v1/chat/completions
The inference server owns the GPUs, batches concurrent requests, and exposes an HTTP API — very often an OpenAI-compatible one. The embedding model runs the same way on its own endpoint:
http://embeddings.internal/embed
Which produces a pleasant result: the backend code looks almost exactly like code that calls OpenAI. Same request shape, same response shape — the only difference is a base URL that resolves to a private IP in a datacenter the hospital owns.
So the application layer is not where the air gap lives. It lives in DNS, in the firewall rules, and in the absence of a default route.
Getting updates into an isolated environment
The AI zone cannot reach the internet, so none of this works from production:
docker pull ...
pip install ...
git pull
huggingface-cli download ...
Every one is an outbound connection to a public registry. Updates arrive through a controlled path instead:
Connected build environment
↓
Download artifact / model weights
↓
Verify signature and checksum
↓
Malware and vulnerability scanning
↓
Approval / change control
↓
Transfer into internal registry / model store
↓
Production pulls from internal source
Production resolves only internal names:
registry.hospital.internal container images
models.hospital.internal model weights
packages.hospital.internal language package mirrors
Teams underestimate this part. It is not one-time setup but a permanent process with owners, an SLA, and a change-control queue. "How do we patch a CVE in a base image" needs an answer before launch — an environment that cannot be patched quickly is not more secure than a connected one, just differently insecure.
One complete request
A doctor asks: "What is our anticoagulation policy before surgery?"
1. Browser requests hospital-ai.internal
↓
2. Internal DNS resolves → 10.50.0.10
↓
3. Segmentation firewall allows HTTPS from clinical VLAN
↓
4. Internal load balancer selects a backend
↓
5. API gateway validates the token, applies rate limits
↓
6. IAM resolves identity → Dr. A, Cardiology, clearance level 2
↓
7. Backend receives question + authorization context
↓
8. Local embedding model turns the question into a vector
↓
9. Vector search, filtered to Cardiology + clearance ≤ 2
↓
10. Returns candidate policy chunks
↓
11. Local reranker narrows 30 candidates to the best 5
↓
12. Question + context → local LLM on the GPU cluster
↓
13. Answer generated, with citations to source documents
↓
14. Audit record written: who, what, which documents, when
↓
15. Response returned to the doctor
Fifteen steps. Now count what happened outside the hospital:
OpenAI 0 requests
Anthropic 0 requests
Google AI 0 requests
Public DNS 0 lookups
Cloud logs 0 records
Every step ran on hardware the hospital owns, over addresses that do not exist on the public internet, behind a firewall with no outbound rule. That is what the requirement asked for — it just is not achievable by calling an API.
What I should be able to explain after reading this
- Why GPT-4 cannot simply be installed on hospital GPUs, and what replaces it.
- Why air-gapped RAG requires both a local embedding model and a local generative model.
- The difference between the ingestion and retrieval pipelines, and why the embedding model version must be pinned across both.
- Why authorization has to happen inside the retrieval query rather than after it.
- What the load balancer, API gateway, and IAM each contribute.
- What the cache, message queue, model storage, vector database, and audit log are for.
- The difference between a network-isolated and a physically air-gapped deployment, and which one a hospital is actually asking for.
- How model weights and application updates get into an offline environment safely.
If I can explain all eight from memory, I understand the architecture. If I can only draw the diagram, I have memorized a picture.
I write about the systems behind shipping AI features — RAG pipelines, evals, and the gap between demo and production — in my newsletter, AI Shipped. New issue every week.