eRacks Systems Tech Blog
Open Source Experts Since 1999
A Federal Court Just Changed the Rules for AI Confidentiality: US v. Heppner and the On-Premise Answer
On February 10, 2026, in United States v. Heppner (No. 25-cr-00503-JSR, Southern District of New York), Judge Jed Rakoff held that roughly thirty-one documents a defendant prepared using the consumer version of a generative AI platform were not covered by attorney-client privilege (the confidentiality protection on lawyer-client communications) or the work-product doctrine (the protection on material prepared for litigation). The defendant had used a public AI chatbot to work on his own defense. The court’s reasoning reaches far beyond one criminal docket: putting information into a public AI platform is disclosure to a third party, and disclosure to a third party is how confidentiality dies.
It was not an isolated signal. In a separate matter, a federal court ordered roughly 20 million ChatGPT conversation logs produced to the plaintiffs in the consolidated copyright litigation against OpenAI, over the platform’s objections that production would invade its users’ privacy. Two different courtrooms, one consistent message: every prompt your team types into a rented AI service is a business record on someone else’s server, kept under someone else’s retention policy, reachable by someone else’s litigation.
What the court actually held
The privilege analysis in Heppner turns on a doctrine every first-year law student learns: confidentiality protections survive only as long as the communication stays inside the protected circle. Hand a draft to an outside party with no duty of confidentiality and the protection is waived. The court treated the consumer AI platform as exactly that kind of outside party: its privacy policy gave the user no reasonable expectation of confidentiality, so material routed through it was shared with a stranger to the privilege. The work-product claim failed separately, because the documents were not prepared by or at the direction of counsel.
The holding was expressly tied to those facts: a public, non-enterprise platform, used without counsel’s direction. That is the door left open, and legal commentators analyzing the ruling have walked straight through it: tools that contractually or architecturally guarantee confidentiality can support a different analysis. On-premise AI is the strongest form of that guarantee, because the data never leaves the organization’s control and no third party ever holds it. Confidentiality by the system’s design, not by a vendor’s promise.
This is not only a law firm problem
Privilege is the sharpest version of the issue, but the underlying logic applies to any confidential information: client lists, financials, personnel matters, unfiled patents, M&A discussions, source code. If it is confidential, and your team pastes it into a public AI prompt box, you have shared it with a third party whose logs are discoverable and whose retention policy you do not control. The 20-million-logs production order makes that concrete: the logs existed, so they were produced.
The architecture answer
A private AI server dissolves the third-party problem instead of papering over it. The model runs inside your walls. Prompts, drafts, and outputs never cross the internet. The only logs are on your hardware, under your retention policy, inside your discovery perimeter, exactly like the rest of your files.
What that looks like in practice on an eRacks system:
- Air-gapped or egress-controlled networking: the machine physically cannot send your data out.
- The full open-source AI stack pre-installed free: Ollama (the model runner), Open WebUI (the familiar chat interface), and vLLM (high-throughput serving), tested before the system ships.
- Current open models, yours to run: DeepSeek, Llama, Qwen, Mistral, selected and sized for your workload.
- Real configured prices: the private AI line starts with the 2U AILSA at $7,695, and every price on the site is an orderable number, not a starting point for a sales call.
- Turnkey deployment if you want it: AI Provisioning and Setup at $1,795 flat takes the system from powered-on to production-ready, with your models tuned on your hardware and 30 days of follow-up included.
One necessary caveat: we build architecture, not legal opinions. Whether and how the Heppner analysis applies to your practice is a question for your counsel. What we can say is that the technical side of the answer is now the easy part.
The full breakdown for legal practices, including the ruling timeline and an architecture comparison, is at eracks.com/law-firm-ai-server. For everyone else wondering what it would take to bring confidential AI work inside the building: tell us what your team runs through AI today, and we will tell you straight which box does it, or whether you need one at all.
joe July 28th, 2026
Posted In: AI Servers, News
Tags: AI confidentiality, AI provisioning, AILSA, air-gapped AI, attorney-client privilege, ChatGPT logs, law firm AI, legal AI server, on-premise AI, private AI, US v Heppner, work product
Private AI, Sized and Priced: How to Spec a Server to Run LLMs On-Premise (from $7,695)
Running large language models (LLMs, the AI models behind chat assistants) on your own hardware, often called “private” or “on-premise” AI, keeps your data inside your building, replaces per-user cloud fees with a one-time purchase, and removes vendor lock-in. The catch is sizing it right. We just published a full, vendor-neutral guide to doing exactly that, and here is the short version.
Why run AI on your own hardware?
Three reasons come up again and again. Privacy and compliance: protected health information (HIPAA), attorney-client material, controlled government data, and source code often cannot legally or contractually leave your control. Predictable cost: a one-time purchase instead of per-seat or per-token billing that grows with every user and every query. Control: your models, your uptime, no rate limits, and no vendor quietly deprecating the model your workflow depends on. For light or occasional use a cloud API is cheaper and simpler; private AI wins when you have data you cannot send out, or when usage is steady and everyday.
The first number: GPU memory (VRAM)
A model has to fit in GPU memory (VRAM, the fast memory on the graphics card) to run at full speed. How much you need is set by the model’s parameter count and its quantization (compressing the weights to fewer bits each: Q4 is about 4 bits per weight and near-lossless for most tasks, Q8 is about 8 bits, fp16 is full precision).
| Model size | Q4 (4-bit) | Q8 (8-bit) | Good for |
|---|---|---|---|
| 7 to 8B (Llama 3.1 8B, Mistral) | ~6 GB | ~10 GB | chat, RAG, coding assist |
| 32 to 34B (Qwen 2.5 32B) | ~22 GB | ~38 GB | strong reasoning, agents |
| 70B (Llama 3.3 70B) | ~42 GB | ~80 GB | frontier-class open models |
| 120B+ or several at once | 70 GB+ | 140 GB+ | heavy or multi-tenant |
A quick rule: VRAM in GB is roughly the parameter count in billions times 0.6 for Q4, or times 1.1 for Q8, with context headroom included. (RAG, or retrieval-augmented generation, feeds the model your own documents at query time.)
It is not only VRAM: system RAM and CPU matter too
System RAM stages models into the GPUs, runs the model server and your data pipeline, and spills over when a model is slightly too big for VRAM. Size it at roughly 1.5 to 2 times your total VRAM. CPU and PCIe lanes: the processor feeds the GPUs through PCIe lanes, so a multi-GPU server needs enough lanes to drive every card at full bandwidth. That is why we build on server-class AMD EPYC and Intel Xeon processors rather than desktop chips: far more PCIe lanes, and support for ECC (error-correcting) memory.
When self-hosting beats the cloud
The arithmetic is direct. A cloud subscription such as ChatGPT Team runs about $30 per user per month. For a 30-person team that is roughly $10,800 a year, every year, with your prompts on someone else’s servers. An on-premise eRacks AILSA at $7,695 covers the same everyday inference on hardware you own, and pays for itself in under a year. In practice, self-hosting wins at roughly 5 to 10 or more regular users, or any privacy mandate.
The GPUs: VRAM without the NVIDIA tax
You do not need flagship NVIDIA silicon to run these models. You need VRAM.
- Intel Arc Pro B50 16GB (low-profile, about $349 to $399): the value pick. Four give 64 GB for well under $8,000 of GPU.
- Intel Arc Pro B70 32GB (about $949): roughly half the price per gigabyte of comparable NVIDIA professional cards. Four give 128 GB.
- NVIDIA RTX PRO 4000 Blackwell SFF 24GB: when you need the CUDA ecosystem and ECC memory in a small, 70-watt card.
The eRacks AI lineup
| Server | GPU memory | Comfortably runs | From |
|---|---|---|---|
| AILSA (2U) | up to 96 GB | Llama 3.3 70B (Q4), Qwen 2.5 32B | $7,695 |
| AIDAN (2U) | 32 GB | 32 to 34B models, 8B at full precision | $13,895 |
| AINSLEY (4U) | 128 GB | 70B with room for long context | $21,995 |
| AISHA (4U) | up to 256 GB | 70B at Q8, or several models, multi-tenant | $30,995 |
Every eRacks AI server ships with Ubuntu LTS (long-term-support Linux) and a complete open-source AI stack (Ollama, Open WebUI, vLLM, llama.cpp, PyTorch) pre-installed and tested. Staff reach the AI from a browser on day one. No per-seat or per-token fees, you own the hardware, and your data never leaves the building.
Bottom line
Start from the model, not the GPU: decide the largest model you will run and at what quantization, size the VRAM (about params times 0.6 for Q4), then add system RAM at 1.5 to 2 times that, and choose a server CPU with the lanes for your GPU count. If privacy is the driver, on-premise is the answer and the only question is which size. And the entry is lower than people expect: a 70B-class model, private, from $7,695.
Configure an AI server → or read the full sizing guide.
Want us to size one to your exact models and user count, at no charge? Reply to this post, a real engineer will help.
joe July 27th, 2026
Posted In: AI Servers, News
Tags: AI inference server, AILSA, GPU VRAM, HIPAA AI, Intel Arc Pro B50, Intel Arc Pro B70, Llama 70B, local LLM, on-premise AI, open source AI server, private AI, Qwen 32B, self-hosted LLM, system RAM