Homelab

Documentation on my journey to a little homelab. It currently consist of 3 routers, 1 managed switch, 3 always running servers and one LLM server that can be activated on demand:

• LLM server llm.server.home with 30 GB GDDR5 on demand
• Raspberry Pi 4 pi4.home for Home Assistant, DNS Resolver and filter
• Raspberry Pi 3 pi3.home
• History

This machine runs only when started manually, and is switched off most of the time. It's more a playground to see what's possible with local LLMs due to its 3 GPUs and 22 GB of fast VRAM. Other specifications:

• Intel i3-6100 2C/4T 3.7 GHz Skylake
• EVGA Z170 Classified 4-way motherboard with Quad-SLI, 6 PCIe slots
• RAM: 32 GB DDR4 2400 (early 2026 reduced to 16 GB)
• GPU: all NVIDIA
  • P104-100 8GB GDDR5X 314 GB/s 5005 MHz PCIe 1.0 x4
  • GTX 1070 8GB GDDR5 220 GB/s 3802 MHz PCIe 3.0 x16 (x8 on the mainmoard)
  • P106-100 6GB GDDR5 176 GB/s 4006 MHz PCIe 1.0 x16 (x8 on the mainmoard)
  • P104-100 8GB GDDR5X 314 GB/s 5005 MHz PCIe 1.0 x4
• NVMe: 256 Toshiba PCIe 4.0 x4
• SSD: 240 Kingston SATA3 for the models of LLM used by Ollama

This image shows the 47 layers of glm-4.7-flash:q4_K_M using 19 of the 22 GB VRAM. Each GPU has about 1 GB headroom for the K-V-pairs related to their layers when doing inference over longer context. An update of the tested maximum context length follows. 2026-02-05

One week later with 53 layers from nemotron-3-nano and 31.6B parameter the power of the CPU spikes to more than 400 Watt:

As stable foundation until April 2029, and with Ubuntu Pro (ESM) until April 2034 this should be enough time to just keep this system running. Almost all other modules run in a docker container. With a fixed IP 10.10.10.40 the services get their own subdomains of server.home. It is routed with a Raspberry Pi 1B in the same network, using pihole.

• traefik to handle the domain names and access from the web
• wordpress under wp.server.home as a test wordpress installation
• ollama as the backbone for the llms being accessed over Open WebUI and n8n
• Open WebUI is accessed via llm.server.home
• Grafana creates beautiful visuals of the use of the server
• n8n tries to automate some workflows and use open APIs and LLMs in the process
• Further wordpress installations:
  • hofkoh.server.home to have an image of the old hofkoh.de
  • saihtorg.server.home to have an image of the old saiht.org

Over SSH I can see some details with btop and nvtop about the systems stage. And OpenWebUI renders a beautiful LLM interface that allows to upload documents and rendered LaTeX output, structured text and tables

I was surprizes how simple the setup with docker is. Inside your home folder you create a folder /docker and for each container a subfolder, for example ~/docker/wordpress. Then you put a YAML docker-compose.yml file in there that describes services (db, wordpress), volumes and networks. For a first start you need

docker compose up -d
docker compose down
docker restart traefik

You can also forward instructions into the docker container, or open a console in it:

docker exec -it ollama ollama run mistral --verbose
docker exec -it ollama bash

Don't forget to create your network:

docker network create my_network
docker network ls

Now I have a root certificate for server.home. Once it is imported, all the other sites as subdomains are trusted with the wildcard *.server.home certificate. All network traffic even inside the local network is therefore encrypted.

Link to certificate:

Technically it is an i5-8500T processor inside the HP EliteDesk mini 800 G4 with 2 NVMe slots inside. The motivation was not just size, but the power consumption of just 4 Watt when idle. And homeservers are idle most of the time, why waste a lot of electricity. It has a separate network and is actually connected to the internet to n8n.io.vn or c103.io.vn

• i5-8500T
• 32 GB DDR4 2400 MT/s with
• NVMe PCIe 4.0 x4 512 GB
• USB Network 2.5 GB

There was a lot to learn about the several layers of abstraction. Here is a little table as visual vertical block:

With the UHD 630 we can even use the GPU to support ollama: You will need to "Pass-through" the GPU device from Proxmox (Level 2) $\rightarrow$ Ubuntu (Level 3) $\rightarrow$ Docker (Level 4) so that Ollama (Level 5) can use it.

I would like to have a few unique domain names for different services, maybe subdomains. That's not supported by my Asus router RT-AX55. I can only get a pi4.home assigned to a predetermined IP, further parts could possibly be done with traefik and subfolders. But better to have a dedicated DNS server. Here is where the Raspberry Pi 4 steps in.

Traefik is forwarding incoming requests to the right container, but getting a DNS entry is another question. With Pihole I also get a great AD blocker. Its surprizing that about 25% of all DNS requests have to be blocked!

It's nice to have a graphical interface. It just took a little longer to get the service running on 10.10.10.4/admin over http to be accepted locally on https://pi4.hv.io.vn/admin with a valid certificate. But it works now!

It is just an attempt to use this tool. Maybe some extra accounts will help to not have this spread out.

This is where the fontend of Open WebUI responds to my requests, in the backbone it connects to the ollama server. It might first have to power it up, but after less than 60 seconds the machine is ready.

This would be reckless, but I'm curious nonetheless. The old Clawdbot made some waves end 2025 and early 2026. What can we actually build?

This is in my own network, all Wifi sensors in my home run on the other network. But I can connect to some Bluetooth temperature and humidity sensor and show it on the info panel. And toner status of the laser printer.

The network for gadgets and devices I don't know a lot about.

I started to build my own LLM server, and I knew already that I need a lot of RAM and processing power. So I got a used Xeon E5-2696v3 18C/36T and 128 GB ECC RAM. It fit's larger models like the llama3.1:70b but comes to a crawl of only 0.2 token/s since I only have 8.27 GB/s memory bandwidth. The i5-8500 would probably get 34 GB/s and speed up to 1 token/s.

In 2026 I realized that there must be a glitch. The model stated above was the llama3.1:70b with 4 bit quantization. The download is 39 GB, and it used 42 GB RAM with ollama. The smaller llama3.1:70b-instruct-q3_K_M is only 34 GB in download and uses 38 GB RAM. Yet here I get a much faster 0.83 token/s speed. Assuming linear dependency on model size and memory speed this aligns with all other measurements (30GB, 19GB, 12GB, 8GB) of the E5-2696 v3. The comparison below should therefore assume 0.74 as reference, 4.6x faster and relating to 38 GB/s memory speed. Being quad-channel of up to 68 GB/s this is more in line with the expectation. The P104-100 would still be 10x faster!

GPUs do not only posess a lot of parallel compute power (important for the initial PP - prompt processing for an LLM task) but usually also have a fast memory. My P104-100 for example has 314 GB/s memory bandwidth for the 8 GB of GDDR5X RAM, that's 38x faster than the DDR3 ECC RAM. Which would result in ca. 7.6 token/s for the llama3.1:70b model in 4bit quantization, requiring 43 GB (that I don't have). And since the GPUs require a lot of power, they also get hot. My Z170 would support 4 graphics cards, but using only 2 slots per GPU is actually rather narrow. It is better to have one slot empty space for airflow between them. But then you can only fit 3 GPUs on the motherboard:

The GTX 1060 has a broken HDMI port, but I still can use the 3 DP to connect a display. So I moved it to my E3-1226 v3 machine. I lost 6 GB VRAM of the combined 26 GB. Then one P106-100 broke, so I replaced it with a P104-100. More memory, and much faster! Now 2 GB more and with 320 GB/s bandwidth. That is the machine that is now running. The theoretical power consupmtion of the three GPUs is 180W + 150W + 120W = 450W but in reality during inference and prompt processing they only use 150W.

With more time in training models the available modesl also get significantly better. Now I can use a 19GB glm-4.7-flash that DOES fit into the 22 GB VRAM, distributed over my 3 GPUs (all 47 layers). The speed should now be 17.2 token/s if we use a linear approach (or 12 resp. 10 for the slower 1070 and P106). Effectively I get 23 token/s in Open WebUI, probably due to being a MoE model to effectively double the speed again (see my insight on speculative execution from 2024-11-27)

Now I finally have a usable system. I get 23 token/s instead of theoretically only 7.6 token on llama3.1:70b (3x as fast) while also getting a similar or better result:

There are a few models that fit into the 22 GB:

• 19 GB glm-4.7-flash with MoE and reasoning
• 17 GB translategemma:27b for 55 languages, maybe timeline project?
• 20 GB qwen3-vl:30b vision-language model in the Qwen family

A short chat wit Gemini revealed that to fully use the context window of 198K one need 40.5 GB VRAM, even though the model weights are only 15 GB. That's because the KV Cache needs 0.12 GB per 1k tokens in the MLA architecture. Maybe I should investigate that with a few longer text documents, since my available VRAM has been reduced by 4 GB in 2026, while also reducing the power consumption from 560W to 450W. (This will change in February, another P104-100 is added). The last GPU was only connected PCIe 1.0 x1 anyway (0.25 GB/s vs. 16 GB/s for 3.0 x16).

This repository was renamed from Homeserver to Homelab, since more than just the server play a role here. Within a month the network expanded, and the server got a fourth GPU and 8GB of GDDR5X RAM. Now even larger models run, and all without a riser card. Unfortunately it also lost 16 GB of RAM. But maybe in 2027 it can get 16 GB DDR4 2667 back. And a CPU upgrade to a 7600K and an attempted 5 GHz frequency. Let's see. Now we can use more models in ollama with 30 GB VRAM:

• Qwen2.5:32b with 32.8B parameter and 20.4 GB memory required.
• nemotron-3-nano with and 24 GB required.

I got the H61 and H81 mainboards repaired for 100k VND each. Some filter capacitors were broken, they run fine again. And the P106-100 could not be fixed. But placed inside the G2030 it started working again and performed the OpenCL Benchmark without problems! Microfracture on the mainboard? Replaced thermal paste? Anyway, it works again.

In January 2025 I tried Qwen2.5:32b with 32.76B parameters and its 65 layers of 20 GB to fit into my 26 GB VRAM (8/6/6/6) machine. With 32 GB DDR4 I could run about 0.92 t/s from the CPU, limited by the memory bandwidth of about 32 GB/s. But I could not get the layers split and load into VRAM successfully. See ollama_multi_GPU.csv. With 3 GPUs (8/6/6) and 20 GB VRAM I could offload 80% to the GPU and got 21/14/15 layers there. The speed increased to 2.34 token/s. With a fourth GPU (8/6/6/6) I could get 98% of layers to the GPU: 19/15/15/15 and increased the inference to 5.11 token/s. Why not 100%? Just one more layer, you got already 21 into the 8GB GPU earlier! Well, I even commented on ollama Github about similar problems (#7509 of ollama and I think in the time since then it has been fixed.) With the parameter num_gpu=65 I got all layers offloaded, but also had an unstable system and 6.37 t/s. Retest in 2026 with 30 GB of VRAM (8/8/8/6) and the layers are easily offloaded 18/18/18/11 and the inference is up to 8.42 token/s. About 10x as fast as the CPU, with memory up to 320 GB/s on GDDR5X.

A comparable model in size in 2026 is now available with nemotron-3-nano with 31.6B parameters, 53 layers and 24 GB model size. The context window is no longer just 32K but 1M! Now more MoE models are available, and the general speed has further increased for the same hardware, while the quality of the models also improved. Even though the memory footprint is 4GB larger the model is significantly faster! I get 38 t/s instead of just 8, almost 5x the speed because of MoE. And the answer is also much more sophisticated. Here a few more details of the comparison:

Surprisingly the largest model in this 30B class is also the fastest: nemotron-3-nano. With its MoE architecture it rivals much smaller 20B and 4B models! All that on 10 year old hardware.