local LLM – eRacks Systems Tech Blog

eRacks/AINSLEY dual Intel Arc Pro B70 AI server, top off — eRacks/AINSLEY: two Intel Arc Pro B70s, 64GB of GPU memory, benchmarked before it ships.

This week we ran a dual Intel Arc Pro B70 server through our full AI provisioning pass: burn-in, GPU bring-up, model deployment, and benchmarks. Not a spec-sheet estimate, not a vendor slide: a production machine on our bench, running the models our customers actually ask for. Here are the numbers, and the three undocumented problems we had to solve to get them.

The numbers

Serving stack: llama.cpp’s official Intel build (SYCL, Intel’s open GPU compute layer), running rootless under Podman (containers with no root daemon), exposing the standard OpenAI-compatible API on localhost. Models in GGUF format, 4-bit quantization, 16K context window.

Qwen3-14B: 54 tokens per second. A token is roughly three-quarters of a word, so this is around 40 words per second: the fast daily driver for chat, summarization, and coding assistance.
Qwen3.6-27B: 26 tokens per second on a single B70. The larger flagship, held steady across back-to-back 300-token responses. For reference, that is faster than most people read.

Both models run entirely in GPU memory. One card serves the model; the second card is free to carry a second model, an embedding model for document search, or headroom for more users. No API fees, no per-token metering, no data leaving the building.

Why the Arc Pro B70 matters

The B70 gives you 32GB of VRAM (the GPU’s onboard memory, the hard limit on what models fit) per card. Two cards put 64GB of GPU memory in a server at a price that undercuts a single big-name datacenter GPU by a wide margin. In 2026’s GPU market, with datacenter cards on allocation and prices climbing, that ratio of memory to dollars is the story. If your workload is private AI inference (running models on your own hardware, on your own data), the B70 class is the value play right now.

And the density curve is still bending: board partners are already building single-slot 32GB B70 variants (Sparkle’s 160W blower design, shown at Computex). Eight single-slot cards put 256GB of GPU memory in one chassis, enough to serve 200B-parameter-class models, and exactly the direction our 8-GPU platforms are built around.

The gotchas nobody documents

Getting those numbers took more than racking cards. Three real problems, none of them in any manual:

Arc cards go to sleep and do not come back. Linux power management idles a headless Arc card (one with no monitor attached) at three separate levels: runtime power management, PCIe port power, and the display engine. On a multi-card system the second card can lock up until the next full power-off. The fix is a three-layer configuration: a udev rule pinning the cards awake at driver bind, a boot-time service covering the PCIe ports, and two kernel boot parameters. A warm reboot will not recover a card that has gone down this hole; only a cold power-off will.
Rootless container port-forwarding that resets every connection. On current Ubuntu, the rootless port forwarder reset every request to the AI server while the server itself was perfectly healthy: a failure that looks exactly like an application crash and can eat hours or days of debugging. We now bind these stacks to host networking by design.
Container image format quirks that silently drop build instructions, and upstream configuration files with inline comments that become literal arguments and crash the server on startup.

We solved all three on the bench, wrote them into our provisioning playbook, and bake the fixes into every AI build we ship. That is bench time you do not spend, and downtime your team never sees.

What this costs to own

Every machine below is configured online, priced live, and ships benchmarked:

eRacks/AIDAN: one Arc Pro B70, 32GB VRAM, 2U EPYC platform, from $13,895.
eRacks/AINSLEY: two Arc Pro B70s, 64GB VRAM, Threadripper platform, from $21,395. This is the configuration class benchmarked above.
eRacks/HIGHLANDER: the 8-GPU flagship for serious multi-user inference and fine-tuning, from $154,995.
The full AI server line starts at $7,695.

And the work described in this post is a product: eRacks AI Provisioning & Setup. We install the serving stack, deploy and benchmark your chosen models on your actual hardware, apply every fix above, and hand you the numbers and the rebuild notes: $1,495, or $2,495 with a private RAG stack (retrieval-augmented generation: a chat interface plus a vector database that lets the models answer from your own documents, entirely offline). Included at no charge on flagship orders.

Renting this class of GPU in the cloud runs thousands of dollars a month, forever, with your data on someone else’s disks. Run your own numbers in our TCO calculator: for daily AI workloads, ownership typically pays for itself inside a year.

Configure a machine online, or request a quote and tell us what models you want to run: we will spec the memory, the cards, and the stack to match. Questions first? Call us at 408-455-0010.

joe August 26th, 2026

Posted In: AI Servers, News

Tags: AI server benchmarks, GPU server, Intel Arc Pro B70, llama.cpp, local LLM, open source AI, Podman, private AI, Qwen3, rootless containers

Why run AI on your own hardware?

Three reasons come up again and again. Privacy and compliance: protected health information (HIPAA), attorney-client material, controlled government data, and source code often cannot legally or contractually leave your control. Predictable cost: a one-time purchase instead of per-seat or per-token billing that grows with every user and every query. Control: your models, your uptime, no rate limits, and no vendor quietly deprecating the model your workflow depends on. For light or occasional use a cloud API is cheaper and simpler; private AI wins when you have data you cannot send out, or when usage is steady and everyday.

The first number: GPU memory (VRAM)

A model has to fit in GPU memory (VRAM, the fast memory on the graphics card) to run at full speed. How much you need is set by the model’s parameter count and its quantization (compressing the weights to fewer bits each: Q4 is about 4 bits per weight and near-lossless for most tasks, Q8 is about 8 bits, fp16 is full precision).

Model size	Q4 (4-bit)	Q8 (8-bit)	Good for
7 to 8B (Llama 3.1 8B, Mistral)	~6 GB	~10 GB	chat, RAG, coding assist
32 to 34B (Qwen 2.5 32B)	~22 GB	~38 GB	strong reasoning, agents
70B (Llama 3.3 70B)	~42 GB	~80 GB	frontier-class open models
120B+ or several at once	70 GB+	140 GB+	heavy or multi-tenant

A quick rule: VRAM in GB is roughly the parameter count in billions times 0.6 for Q4, or times 1.1 for Q8, with context headroom included. (RAG, or retrieval-augmented generation, feeds the model your own documents at query time.)

It is not only VRAM: system RAM and CPU matter too

System RAM stages models into the GPUs, runs the model server and your data pipeline, and spills over when a model is slightly too big for VRAM. Size it at roughly 1.5 to 2 times your total VRAM. CPU and PCIe lanes: the processor feeds the GPUs through PCIe lanes, so a multi-GPU server needs enough lanes to drive every card at full bandwidth. That is why we build on server-class AMD EPYC and Intel Xeon processors rather than desktop chips: far more PCIe lanes, and support for ECC (error-correcting) memory.

When self-hosting beats the cloud

The arithmetic is direct. A cloud subscription such as ChatGPT Team runs about $30 per user per month. For a 30-person team that is roughly $10,800 a year, every year, with your prompts on someone else’s servers. An on-premise eRacks AILSA at $7,695 covers the same everyday inference on hardware you own, and pays for itself in under a year. In practice, self-hosting wins at roughly 5 to 10 or more regular users, or any privacy mandate.

The GPUs: VRAM without the NVIDIA tax

You do not need flagship NVIDIA silicon to run these models. You need VRAM.

Intel Arc Pro B50 16GB (low-profile, about $349 to $399): the value pick. Four give 64 GB for well under $8,000 of GPU.
Intel Arc Pro B70 32GB (about $949): roughly half the price per gigabyte of comparable NVIDIA professional cards. Four give 128 GB.
NVIDIA RTX PRO 4000 Blackwell SFF 24GB: when you need the CUDA ecosystem and ECC memory in a small, 70-watt card.

The eRacks AI lineup

Server	GPU memory	Comfortably runs	From
AILSA (2U)	up to 96 GB	Llama 3.3 70B (Q4), Qwen 2.5 32B	$7,695
AIDAN (2U)	32 GB	32 to 34B models, 8B at full precision	$13,895
AINSLEY (4U)	128 GB	70B with room for long context	$21,995
AISHA (4U)	up to 256 GB	70B at Q8, or several models, multi-tenant	$30,995

Every eRacks AI server ships with Ubuntu LTS (long-term-support Linux) and a complete open-source AI stack (Ollama, Open WebUI, vLLM, llama.cpp, PyTorch) pre-installed and tested. Staff reach the AI from a browser on day one. No per-seat or per-token fees, you own the hardware, and your data never leaves the building.

Bottom line

Start from the model, not the GPU: decide the largest model you will run and at what quantization, size the VRAM (about params times 0.6 for Q4), then add system RAM at 1.5 to 2 times that, and choose a server CPU with the lanes for your GPU count. If privacy is the driver, on-premise is the answer and the only question is which size. And the entry is lower than people expect: a 70B-class model, private, from $7,695.

Configure an AI server → or read the full sizing guide.

Want us to size one to your exact models and user count, at no charge? Reply to this post, a real engineer will help.

joe July 27th, 2026