Your Client's Data Can't Leave Germany. Here's the AI Stack That Actually Works.

The sentence that ends most AI conversations in a German office is short: "Das darf hier nicht raus."

That data cannot leave. Not the building, sometimes not the country. And the person saying it is usually right, which is the part people underestimate. I have sat in rooms with a Steuerkanzlei, with a medical practice, with an insurance intermediary, and the conversation always arrives at the same wall. They want the productivity. They have seen the demos. They also have a legal duty that a US API endpoint quietly breaks.

So this post is the other half of the conversation. The 5 Questions to Ask an "EU-Compliant" AI Vendor Before You Sign is about vetting somebody else's product. This one is about what you build when the answer to those questions is no. A gdpr compliant local ai setup is not one thing, it is four tiers with very different price tags, and most people pick the wrong one because they only looked at the legal side or only at the hardware side.

I Am Not a Lawyer, and This Post Is Not Legal Advice

Let me put that down once, clearly, and then stop hedging every sentence.

I build systems. I read the regulations because I have to scope projects around them, and I have spent a lot of hours with clients' actual lawyers. But I do not give legal advice, and nothing below replaces a conversation with a Fachanwalt für IT-Recht or your Kammer. What I can tell you is which technical decisions the lawyers keep flagging, because I have watched the same three flags come up over and over.

The useful thing to understand is that you are not dealing with one rulebook. You are dealing with three, and they have different failure modes. GDPR governs personal data and processor relationships. The EU AI Act governs the system itself. And German professional secrecy law, § 203 StGB, governs whether a specific person is even allowed to see the content, personal data or not. They are not the same test, and passing one does not pass the others.

The Blocker Usually Is Not GDPR. It Is § 203 StGB.

Here is the part that surprises people. For a Kanzlei, a Steuerberater, a doctor or a Notar, GDPR is often the easier hurdle.

Under GDPR you sign an Auftragsverarbeitungsvertrag, an Article 28 processor agreement, and if the vendor's terms are honest you are a long way toward defensible. § 203 StGB is criminal law, and it works differently. Disclosing a client secret to an unauthorised third party is a criminal offence for the professional, not a fine for the company. The 2017 reform of § 203 created room for "sonstige mitwirkende Personen", other contributing persons, which is what lets a Kanzlei use an external IT provider at all. But that room has conditions attached: bringing the provider in has to be genuinely necessary for proper practice of the profession, the provider has to be obliged to secrecy in text form and instructed about the criminal consequences of breaching it, and the professional has to select and supervise that provider carefully.

Read that list again and think about a self-service API signup with a click-through terms page. That is the mismatch. An AVV alone does not satisfy § 203. This is why a Kanzlei's compliance officer can be perfectly happy with a GDPR analysis and still say no. And it is why the technical architecture matters so much: the fewer parties who can theoretically read the content, the shorter that whole conversation gets.

An EU Region Is Not the Same Thing as EU Data

In July 2025, Microsoft France's director of public and legal affairs told a French Senate inquiry, under oath, that Microsoft could not guarantee that French public sector data held on its infrastructure would be shielded from US authorities without the customer's consent. Not "we would fight it". Cannot guarantee.

That is the cleanest statement of the problem I have seen, and it came from the vendor. The US CLOUD Act reaches data held by US-controlled providers regardless of where the servers physically sit. Picking the Frankfurt region does not change who controls the company. As of late 2026, no law has repealed that extraterritorial reach.

The EU-US Data Privacy Framework is a separate question and it is currently standing. The General Court dismissed Philippe Latombe's challenge in September 2025 and confirmed the adequacy decision, but only on the facts as they stood in 2023, expressly declining to look at anything after that. The appeal is pending before the Court of Justice as Case C-703/25 P. So transfers under the DPF are legal today. If your client's risk horizon is longer than the next court date, that is worth saying out loud rather than pretending the question is settled. I wrote about a related German ruling in Google's AI Overview Can Now Lie About Your Business, and the pattern is the same: German courts are willing to put the liability somewhere the vendor did not expect.

For a Berufsgeheimnisträger, none of this is really the point anyway. Even a perfectly legal transfer is still a disclosure to another party. The § 203 question does not care that the transfer mechanism is valid.

The Four Tiers, and What Each One Actually Costs You

Tier one: fully local, on the client's own hardware. This is what most people mean when they say gdpr compliant local ai. Nothing leaves the building. This is the only tier where the § 203 conversation basically disappears, because there is no third party to disclose to. It is also the tier where you inherit every operational problem: backups, updates, someone to call at 7am. Cost is capital plus your time forever.

Tier two: EU-hosted open-weight inference. A European company runs open models in European data centres and gives you an API. The market here got real in 2026. Scaleway serves open weights from French and other European data centres. IONOS runs a German-hosted model hub. OVHcloud's AI Endpoints offer a zero-retention API across a few dozen open models. Nebius, Mistral and a long tail of smaller EU providers sit in the same space. This tier is genuinely useful and much cheaper to operate than tier one. But you are back to a third party, which means an AVV, a § 203 secrecy undertaking in text form, and a hard look at retention and training terms.

Tier three: the EU region of a US hyperscaler. This is what most clients already have, and it is the tier that fails the test it looks like it passes. It is fine for a lot of internal business data. For Berufsgeheimnis content it is the argument you do not want to be having.

Tier four: hybrid, where only the sensitive fields stay local. In practice this is often the right answer and almost nobody proposes it, because it is more work. You run a small local model that identifies and strips or tokenises the names, Aktenzeichen, Steuernummern and diagnoses, then send the de-identified remainder somewhere better and faster, then re-insert locally. It is a real engineering job, and the de-identification step becomes the thing you have to be right about. But it lets a small office get frontier-quality drafting on content whose identifying parts never left.

What the AI Act Actually Asks of a Small Deployer in Late 2026

Do not take the AI Act timeline from anyone's memory, including mine when I wrote the first draft of this. It moved.

The Digital Omnibus on AI was published in the Official Journal on 24 July 2026 and entered into force on 27 July 2026, six days before the AI Act's original high-risk deadline. It pushed the high-risk obligations for standalone Annex III systems out to 2 December 2027, and for AI embedded in regulated products under Annex I to 2 August 2028. So the thing everyone was bracing for in August 2026 largely did not land in August 2026.

What did not move matters more for a small office. The Article 50 transparency obligations applied from 2 August 2026. The Article 4 AI literacy duty has applied since February 2025; the Omnibus softened its wording from ensuring a sufficient level to taking measures to support its development, but it still binds every deployer, with national supervision from 3 August 2026. Article 5's prohibited practices have been in force since February 2025.

And here is the framing that saves clients money: a Kanzlei running an open model on its own hardware to summarise its own files is almost always a deployer, not a provider. Deployer obligations are far lighter. The moment you fine-tune a model and put it in front of other people's clients as a product, that changes. Most of the panic I get asked about is people reading provider obligations and assuming they apply.

What a Practice Can Realistically Run

The rough sizing rule that has held up for me: at 4-bit quantisation, budget somewhere around 0.5 to 0.6 GB of memory per billion parameters, then add another 20 to 30 percent for context and overhead. That is the arithmetic, not a benchmark, and it tells you most of what you need before you buy anything.

Single machine on a desk: a workstation with a 24 to 32 GB GPU, or a Mac with 32 to 64 GB of unified memory, runs a mid-size open model comfortably. Qwen3's smaller and mixture-of-experts variants and Mistral's small models sit in this bracket, as does OpenAI's gpt-oss-20b, which fits in roughly 16 GB. Licensing is permissive across most of that set, which matters for commercial deployment. That machine serves one or two heavy users, or a handful of light ones.

Server: gpt-oss-120b needs something in the 80 GB class. That is not a desk machine, that is a box in a rack with a maintenance contract. For a five-person Kanzlei, this is usually where the project stops making financial sense compared with tier two.

I did the actual break-even arithmetic in When Does Buying Hardware for Local LLMs Actually Pay Off?, and for capability specifics Local Models Caught Up covers what runs on a desk today. The short version for this context: the model is rarely the constraint any more. Concurrency is.

The Tasks That Work, and the Ones That Do Not

Five things work well enough to be worth building, and they are all unglamorous.

Document classification, sorting incoming post into the right file and the right type. Extraction from scanned correspondence, pulling dates, amounts, reference numbers into a structured record. Drafting standard letters from a template plus a case file, where a human signs. Summarising a long file before a meeting. And internal search over the document archive, which is the one that generates the most genuine delight, because everybody has lost a document.

What does not work: anything where the output goes out unreviewed, anything that requires current law rather than the contents of the file, and anything with a number in it that nobody checks. A local model is a fast, tireless, moderately careless junior. Staff it accordingly. If you are still working out whether you need outside help for any of this at all, Do You Actually Need an AI Consultant? is the honest version.

OCR and Retrieval Are Where These Projects Actually Die

Nobody has ever called me because the language model was disappointing. They call because the scans are bad.

German office archives are full of documents that OCR hates: faxes, crooked scans, stamps over text, handwritten margin notes, older material in Fraktur. On clean printed text, a classic engine like Tesseract is still competitive and enormously cheaper. On handwriting, bad photographs and layout-heavy pages, vision models win by a wide margin. The design that actually works is neither one alone: run the cheap engine, keep its confidence scores, and route only the pages it is unsure about to the vision model. Cheaper than a human reviewing everything, better than the cheap engine alone.

Then there is the file share. Twelve years of Schreiben_final_v3_NEU.docx in folders named after people who left. Retrieval over that is not a vector database problem, it is a metadata problem, and the honest scoping answer is that the cleanup is a real line item. If you skip it, the system confidently retrieves the 2019 version of the letter.

The Question Nobody Asks Until Month Fourteen

Who patches this when I am not here?

A local stack is infrastructure. It needs OS updates, model updates, backups that get tested, and a person who understands why the retrieval index is stale. In a five-person office, that person is usually the one partner who is good with computers, and when they leave, the thing rots quietly until it starts giving wrong answers that look right.

So I ask that question in the first meeting now, before hardware. A gdpr compliant local ai deployment is a maintenance commitment, not a purchase. If there is no honest answer, the recommendation changes: tier two with a managed EU provider and a proper contract beats a beautiful local box with nobody behind it. A slightly less private system that is maintained is safer than a maximally private system that is six months out of date.

That is genuinely the whole decision. Not which model. Not which GPU. Who owns it on a Tuesday in eighteen months.

If you are working through this for a German or EU client and want somebody to sanity-check the architecture before the hardware invoice, that is the kind of scoping I do. Bring your lawyer to the same call. It goes faster when everyone is in the room.