self-hosted LLM – eRacks Systems Tech Blog

eRacks AILSA 2U private AI server — A private AI server runs the models in your building, on hardware you own.

Running large language models (LLMs, the AI models behind chat assistants) on your own hardware, often called “private” or “on-premise” AI, keeps your data inside your building, replaces per-user cloud fees with a one-time purchase, and removes vendor lock-in. The catch is sizing it right. We just published a full, vendor-neutral guide to doing exactly that, and here is the short version.

Read the full sizing guide →

Why run AI on your own hardware?

Three reasons come up again and again. Privacy and compliance: protected health information (HIPAA), attorney-client material, controlled government data, and source code often cannot legally or contractually leave your control. Predictable cost: a one-time purchase instead of per-seat or per-token billing that grows with every user and every query. Control: your models, your uptime, no rate limits, and no vendor quietly deprecating the model your workflow depends on. For light or occasional use a cloud API is cheaper and simpler; private AI wins when you have data you cannot send out, or when usage is steady and everyday.

The first number: GPU memory (VRAM)

A model has to fit in GPU memory (VRAM, the fast memory on the graphics card) to run at full speed. How much you need is set by the model’s parameter count and its quantization (compressing the weights to fewer bits each: Q4 is about 4 bits per weight and near-lossless for most tasks, Q8 is about 8 bits, fp16 is full precision).

Model size	Q4 (4-bit)	Q8 (8-bit)	Good for
7 to 8B (Llama 3.1 8B, Mistral)	~6 GB	~10 GB	chat, RAG, coding assist
32 to 34B (Qwen 2.5 32B)	~22 GB	~38 GB	strong reasoning, agents
70B (Llama 3.3 70B)	~42 GB	~80 GB	frontier-class open models
120B+ or several at once	70 GB+	140 GB+	heavy or multi-tenant

A quick rule: VRAM in GB is roughly the parameter count in billions times 0.6 for Q4, or times 1.1 for Q8, with context headroom included. (RAG, or retrieval-augmented generation, feeds the model your own documents at query time.)

It is not only VRAM: system RAM and CPU matter too

System RAM stages models into the GPUs, runs the model server and your data pipeline, and spills over when a model is slightly too big for VRAM. Size it at roughly 1.5 to 2 times your total VRAM. CPU and PCIe lanes: the processor feeds the GPUs through PCIe lanes, so a multi-GPU server needs enough lanes to drive every card at full bandwidth. That is why we build on server-class AMD EPYC and Intel Xeon processors rather than desktop chips: far more PCIe lanes, and support for ECC (error-correcting) memory.

When self-hosting beats the cloud

The arithmetic is direct. A cloud subscription such as ChatGPT Team runs about $30 per user per month. For a 30-person team that is roughly $10,800 a year, every year, with your prompts on someone else’s servers. An on-premise eRacks AILSA at $7,695 covers the same everyday inference on hardware you own, and pays for itself in under a year. In practice, self-hosting wins at roughly 5 to 10 or more regular users, or any privacy mandate.

The GPUs: VRAM without the NVIDIA tax

You do not need flagship NVIDIA silicon to run these models. You need VRAM.

Intel Arc Pro B50 16GB (low-profile, about $349 to $399): the value pick. Four give 64 GB for well under $8,000 of GPU.
Intel Arc Pro B70 32GB (about $949): roughly half the price per gigabyte of comparable NVIDIA professional cards. Four give 128 GB.
NVIDIA RTX PRO 4000 Blackwell SFF 24GB: when you need the CUDA ecosystem and ECC memory in a small, 70-watt card.

The eRacks AI lineup

Server	GPU memory	Comfortably runs	From
AILSA (2U)	up to 96 GB	Llama 3.3 70B (Q4), Qwen 2.5 32B	$7,695
AIDAN (2U)	32 GB	32 to 34B models, 8B at full precision	$13,895
AINSLEY (4U)	128 GB	70B with room for long context	$21,995
AISHA (4U)	up to 256 GB	70B at Q8, or several models, multi-tenant	$30,995

Every eRacks AI server ships with Ubuntu LTS (long-term-support Linux) and a complete open-source AI stack (Ollama, Open WebUI, vLLM, llama.cpp, PyTorch) pre-installed and tested. Staff reach the AI from a browser on day one. No per-seat or per-token fees, you own the hardware, and your data never leaves the building.

Bottom line

Start from the model, not the GPU: decide the largest model you will run and at what quantization, size the VRAM (about params times 0.6 for Q4), then add system RAM at 1.5 to 2 times that, and choose a server CPU with the lanes for your GPU count. If privacy is the driver, on-premise is the answer and the only question is which size. And the entry is lower than people expect: a 70B-class model, private, from $7,695.

Configure an AI server → or read the full sizing guide.

Want us to size one to your exact models and user count, at no charge? Reply to this post, a real engineer will help.

joe July 27th, 2026

Posted In: AI Servers, News

Tags: AI inference server, AILSA, GPU VRAM, HIPAA AI, Intel Arc Pro B50, Intel Arc Pro B70, Llama 70B, local LLM, on-premise AI, open source AI server, private AI, Qwen 32B, self-hosted LLM, system RAM

Good: AILSA 2U, from $5,995

The AILSA is the entry point. It is a 2U system (a U is 1.75 inches of rack height) assembled, burned in, and certified by eRacks, with Intel Arc GPUs (graphics processing units, the chips that run AI models). The base build carries two Arc B50 low-profile cards for 32GB of VRAM (the GPU’s onboard memory, which holds the model), with larger Arc Pro options available in the configurator. It handles 70B-class models (roughly 70 billion parameters), which covers most private chat, coding, and RAG workloads (retrieval-augmented generation, where the model answers from your own documents).

Better: AISLING 4U, from $16,995

The new AISLING is the workhorse of the line. It pairs a 24-core AMD Threadripper 9960X with 128GB of ECC memory (error-correcting code memory, which detects and fixes memory errors) and a single 1600W power supply. The 4U chassis takes up to three dual-slot GPUs, which means 96GB of total VRAM with Intel Arc Pro B70 cards. That is enough headroom to run a 70B-class model at higher precision, serve more simultaneous users, or hold longer context windows.

Best: AILEEN 4U, from $21,995

The new AILEEN steps up to server-class silicon: a 32-core AMD EPYC 9355 with 12 memory channels and 192GB of ECC memory. The extra memory bandwidth feeds the GPUs and speeds up CPU-side work like document indexing. It takes up to four GPUs for 128GB of VRAM, and it has redundant 1+1 power (two supplies, either one can run the system alone). AILEEN also ships in custom colors: blue, black, white, or red. The blue unit is the one pictured here.

Model	Form factor	GPU memory (VRAM)	From
AILSA	2U	32GB base, larger Arc Pro options	$5,995
AISLING	4U	up to 96GB (3 x Arc Pro B70)	$16,995
AILEEN	4U	up to 128GB (4 GPUs)	$21,995

The enterprise tier

Some buyers need validated OEM server systems rather than our eRacks-Certified workhorse builds, usually because their ops teams require out-of-band management (a dedicated channel for remote hardware control, such as IPMI, that works even when the operating system is down). For them, the AIDAN 2U EPYC starts at $13,895, and the AISHA 4U starts at $30,995 with support for up to 10 GPUs.

What every system includes

All five models ship with Ubuntu LTS and the open-source AI stack pre-installed: Ollama, Open WebUI, vLLM, llama.cpp, and PyTorch. Each unit is burned in (run under sustained load before shipping) and tested. You get browser access to your own models on day one: unbox, rack, log in.

Not sure which tier fits? Start with our private AI sizing guide, which walks through how much GPU memory and system RAM a given model actually needs. Then configure the system that matches.

Browse the AI server line

joe July 5th, 2026