Speculative Decoding Should be Tuned On-Line
| Written: | 2026-11-19 |
| Last Modified: | 2026-11-19 |
| Tags: | #technical #LLM |
I've been experimenting with hosting my own models at home (article coming soon), and I noticed that serving backends (vLLM/llama.cpp/etc.) treat speculative decoding depth (MTP) as a parameter that you set once to a number like 5 and that's it.
In my experiments, I, of course, tried to pinpoint the "ideal" config number here. But it's incredibly hard to find, as measurements are quite noisy and dependent on the workload. I found that anything upwards of ~3 was not yielding measurable improvements, and I did not see any drawbacks to pushing this number up to 6. That's simply because I am utterly memory-bound with my simple RTX 3090 setup. However, when switching to multiple parallel generations, I noticed that suddenly, higher MTP numbers had an adverse effect on performance. In hindsight, this makes sense too! Having four parallel generations, generating 7 tokens each (1 "true" + 6 speculative), pushed the batches (with the kernels llama.cpp was using) more towards the compute-bound regime. That got me thinking that this ought to be a free per-batch tuning parameter in the hands of the serving engine, rather than a fixed knob for the user to pick semi-randomly.
A competent serving engine ought to know the performance of its kernels on the hardware, and how it varies with the parameters used. This should yield a rather simple optimisation problem in which the speculative decoding depth is a free parameter the engine can choose to maximise whichever metric it so desires: picking a higher value for batches that are on the memory-bound side, and dialling it back once batches become compute-bound. It would be interesting to plot these regimes with actual data; I might do that one day and update this post. Until then, treat all this as unproven conjecture.