Qwen3.8 27B at 256K: 50 TPS on a 24 GB GPU
AI Economics Tools piszczek.pl
Listen to this summary
I gave Qwen3.8's MTP drafter another 69.2 MiB of precision. Throughput fell from 50.44 to 37.02 tokens per second. That result sums up the whole experiment: the best local inference setup is rarely made from the individually "best" parts.
I wanted a dense 27B model, its full 262,144-token context, multimodal input, maximum useful quality, and speculative decoding on an NVIDIA RTX PRO 4000 Blackwell SFF with 24 GB of VRAM. The server also had to survive real agent work after printing model loaded. The experiment followed a hunch I had written about earlier: careful operation may matter as much as moving to a larger model.
More ways to use this
Open on another device
Scan to open this page. Your saved notes and account access are not included.