I run an open-weight model on my own server: Gemma 4 12B, 4-bit quantized, served by llama.cpp from a headless Ubuntu VM under Proxmox, on a single RTX 3060 with 12 GB of VRAM. It isn’t the fastest way to use a language model, and renting one is cheaper and easier. I did it to see every layer between the hardware and the text that comes out, instead of treating them as an API.
One constraint runs through everything
The 3060’s 12 GB of VRAM is the number that every other decision has to respect. A 12B-parameter model at 4 bits per weight needs roughly 7 GB just for the weights. What’s left has to hold the contextThe KV cache, which grows with every token in the conversation., the CUDA runtime, and some headroom.
Right now it runs at around 36 tokens per second with a 100k-token context, which is roughly where the budget lands on this card.
So model size, quantization level and context length are not three separate settings. They are one budget, and spending more on one means less for the other two. Seeing that trade-off on real hardware teaches you more than any table of model sizes does.
The layers
Running it yourself means you own each layer, and each one fails differently.
- Hardware and hypervisor. The GPU sits in a Proxmox host, and the model runs in a VM, with the card passed through to it. The first question is how the VM gets access to the GPU at all.
- Operating system. A headless Ubuntu VM, with no desktop, just SSH. It’s small and boring, which is the point: everything that happens on it is something you set up.
- Driver and CUDA stack. The NVIDIA driver, the CUDA version and the inference engine’s build all have to agree. This is where “it should work” most often becomes an afternoon of reading version tables.
- Inference engine. llama.cpp loads the model, offloads layers to the GPU, manages the KV cache and serves the result. Its flagsHow many layers go on the GPU, and how long the context is. are where the VRAM budget becomes concrete.
- The model. A quantized file. Quantization is a trade of precision for memory, and it’s what makes a 12B model fit on this card in the first place.
Why bother
Hosted APIs hide all of this, and for most uses they should. But when you’ve had to make the layers fit together yourself, the abstractions stop being mysterious. You know why a longer context costs memory, why a quantized model is smaller, why a driver mismatch gives a confusing error. That intuition carries over to any deployment, local or in the cloud.
It also makes the practical question honest: how much can you actually do on modest hardware? A 12 GB card is enough for a capable model, with a context window that you have to budget rather than assume.
The setup itself is listed in the Lab.