How we ran a 744-billion-parameter AI model on our own GPUs and made it 36× faster

From a crashing pod to 188 tokens per second: a hands-on story of self-hosting GLM-5.2 on vLLM, with the kind of problems you won’t find in the tutorials. Short and to the point.

More and more companies (ours included) want to run large language models on their own infrastructure – for data confidentiality, for cost at high volume, or simply for control. The internet is full of “just run vllm serve and you’re done” guides. This story is about what happens when the model has 744 billion parameters, an architecture that’s three months old, and nothing works on the first try.

The starting point

• Model: GLM-5.2, an open 744B-parameter model (MoE: ~40B active per token), with a native 1M-token context and a DSA (dynamic sparse attention) architecture.
• Hardware: a single node with 4× NVIDIA H200 (564 GB VRAM in total) in an OpenShift/OKD cluster.
• Goal: an OpenAI-compatible API, everything deployed via Helm, reproducible from the repo.

What we found: a vLLM pod in CrashLoopBackOff – and that was only the beginning.

Wall #1: GGUF got pulled out of vLLM

The 2-bit quantized model (GGUF, 239 GB) refused to come up: Load format ‘gguf’ is not supported. It turned out that GGUF support had been split out of the main repository into a separate plugin. Lesson one: image: latest in AI infrastructure is asking for trouble. We pinned the version and installed the plugin.

Wall #2: the plugin doesn’t know our architecture

Unknown gguf model_type: glm_moe_dsa. The plugin maps the tensor names in the GGUF file to the names vLLM expects for the model – and it simply didn’t know the GLM-5.2 architecture. We built our own adapter: a map of 1,431 tensors, based on a “meta-model” (transformers can build a model’s structure without loading the weights – just the parameter names). We validated the whole thing locally, on a laptop, before touching the cluster, because every failed attempt on the cluster meant 10+ minutes of loading weights.

Walls #3–5: tensor surgery

It only got more interesting from there. Three cases where the file format and the engine’s implementation speak different languages:

1. vLLM keeps certain attention projections fused and unquantized, whereas in the GGUF file they arrive separately and quantized. Solution: dequantization on the fly.
2. One layer type in this version of vLLM couldn’t load quantized weights at all – we had to force the float path.
3. The best one: the GGUF file doesn’t contain the kv_b_proj tensor that vLLM expects. During conversion, llama.cpp splits it into two others (one of them transposed) for its own optimizations.
4. We had to read the converter’s code and reverse its math: reshape → transpose → concatenate → [28672×512]. One mistake in the dimensions and the model would have generated noise.

Result: the model came up and responded. Speed: 5 tokens per second. On hardware worth hundreds of thousands of złoty. Well… not quite what we’d hoped for.

Why so slow?

Here’s the intuition worth taking away from this piece even if nothing else interests you. During decoding, most of the model’s active weights have to be re-read from GPU memory for each successive token. In our case, that meant roughly 20 GB read per token. Decoding isn’t compute-bound – it’s memory-bandwidth-bound. The experimental GGUF kernels in vLLM simply weren’t squeezing out that bandwidth. We switched the quantization to AWQ-INT4 (410 GB, native Marlin kernels in vLLM): 89 tok/s. Seventeen times faster – with no change in hardware. Phew…

The cherry on top of our LLM cake: speculation with 85% accuracy

Since reading the weights per token is expensive, while verifying several tokens at once costs almost as much as verifying one (again: memory-bound), speculative decoding comes into play. GLM-5.2 has a built-in MTP (multi-token prediction) layer, trained together with the model, that guesses the next tokens. The engine verifies the guesses in a single pass – preserving the quality of the base model’s output. The catch: our AWQ checkpoint didn’t include the MTP layer, and the official one exists only in FP8. So we performed a “transplant”: we stitched 1,569 FP8 tensors onto the INT4 checkpoint (the new directory is hardlinks – zero copying of 400 GB) and taught vLLM mixed quantization with a small patch injected via a Python import hook – no custom image build, no root, and fully reversible from Helm.

Speculation accuracy: 80–87% of tokens guessed. The end result:

36× faster than the first working version. Same hardware, same model quality.

With several concurrent requests, we peak at as much as 295 tok/s.

…but that’s a story for another time.

Stack: OpenShift/OKD, vLLM 0.24, GLM-5.2 (754B MoE), 4× NVIDIA H200, Open WebUI. All figures from the test environment – single sequence, real API.

This is only the start of our experiments with self-hosting large models. AI is developing at a dizzying pace, which is why we treat this project as just another stage of learning.