Measuring Adaptive Speculative Decoding in SGLang with EAGLE3 on Triton

I do not know if this was my poor luck or me being late, but while trying to understand and measure speculative decoding myself, I realized that SGLang had already merged adaptive speculative decoding (f*ck ;)

And when you first look at the idea, it actually sounds pretty beautiful. The controller takes the accepted draft count from every request in the batch, averages them into one number, smooths that using an EMA, and then uses that number to decide K.

That's mainly the whole idea.

But let me say the same thing in slightly slower words, because this is where I started having questions.

Imagine 32 requests are running together. A few of them are generating very predictable text, so maybe they are happily accepting four or five draft tokens every round. Some other requests are more open-ended and are accepting almost nothing.

Now SGLang takes all of these acceptance counts, adds them together, divides by 32, and gets one number.

And that one number decides the draft depth for everybody in the batch. So, if every request has its own acceptance behaviour, why are we throwing all of that away and keeping only one mean?

Maybe the mean really is enough.

Maybe these curves are stable enough that one number summarizes everything we care about.

That sounds nice when you say it like that. But I wanted to know whether it is actually true.

The questions in my mind were basically:

• If the curve belongs to the request, why does one number work at all?
• What if the curve itself moves when you change K?
• And how would you even know if you never measured it?

These were some of the questions I had while reading adaptive_spec_params.py.

I did not really want to argue about them theoretically, so I built a harness and rented one A6000 to go find out.

What I actually measured

Let me first explain what the harness does.

I run the same workload at a fixed K, across a few concurrency levels.

For K:

1, 3, 5, 7

And for concurrency:

1, 8, 32

So that gives me 12 static runs.

I also run one baseline with speculative decoding completely switched off, because without that I do not really know whether speculation is helping in the first place.

There are two controls here that I think are worth mentioning, because without them the whole grid can become junk very easily.

First, every server gets warmed up before I start measuring anything.

The first few requests into a fresh server are slower for reasons that have nothing to do with my experiment.

Triton may still be compiling kernels.

The allocator may still be growing.

Caches are still cold.

And when the differences you are chasing are only a few percent, all of these things can quietly become your "result."

The second thing is that every server runs with:

--disable-radix-cache

The reason is that I reuse the same seeded prompt pool across different concurrency runs.

Without disabling the radix cache, later runs could become cheaper simply because parts of those prompts are already cached from earlier runs.

I am not studying prefix caching here, so I definitely do not want it quietly moving my batch-size axis.

Then came the first real problem

SGLang already gives us something pretty useful:

spec_correct_drafts_histogram

For every request, it tells us how many rounds accepted 0 drafts, how many accepted 1, then 2, and so on.

From that, we can compute something like:

P(position k was accepted)

Which is exactly the acceptance curve I wanted to look at.

But there is a problem.

That number only means something if K was held still.

Let me explain why.

Suppose one request is running under adaptive K.

And for most of its life, the controller gives it depth 2.

So position 1 gets proposed.

Position 2 gets proposed.

But positions 3, 4, 5, 6 and 7 never even existed.

Nobody proposed those draft tokens.

Nobody asked the model whether it would accept them.

Now imagine those later positions are sitting there as zeros in the histogram.

If I average those zeros into my curve, I am treating "this token was never proposed" as "this token was rejected."

Those are two completely different things.

And the annoying part is that you cannot repair this afterwards.

That information simply is not in the file.

It was never written.

So I patched my fork.

I added an iteration-level trace.

Every decode iteration now writes the active K and the request IDs that were inside that batch.

Now I can actually bound K per request from the trace instead of assuming it.

And if I do not trust a request's curve, I can simply throw it away.

Across the three adaptive runs, this produced something like 31,000 trace lines.

Not very pretty, but at least now the adaptive capture is something I can actually reason about.

Then there was the oracle

I built four levels of it.

All of them are scored on the deepest static capture, and all of them have a train/test split.

That split may sound like a small implementation detail, but it is actually very important here.

Imagine I create enough fine-grained bins.

Eventually I can put almost every batch into its own bin.

At that point the policy can simply memorize everything.

Then I can proudly report an oracle gap of exactly zero.

But I have not learned anything.

I have just memorized my measurement.

So some batches have to stay unseen.

But before looking at any oracle gap at all, I added one more thing.

A gate.

And honestly, this ended up becoming the part of the experiment I am happiest with.

There is an assumption hiding underneath all of this

Every level of the oracle eventually has to answer a question like:

What would this request have done at a depth it never actually ran?

But we do not know that.

We only observed the request at the K it actually experienced.

So whenever I estimate its behaviour at another depth, I am making an assumption.

The assumption is:

the acceptance curve does not depend on K.

And this assumption is kind of quietly sitting underneath a lot of reasoning around speculative decoding.

So before using it, I decided to check whether it was actually true.

The rule was simple.

Run the same workload.

Same seed.

Same batching.

At:

K = 1, 3, 5, 7

Then compare:

P(A at k)

across those different depths.

If the curve moves when I move K, then I will not report the oracle gap.

Because at that point the counterfactual I am using is not something I can properly defend.

And one thing I really wanted to do was write this rule down before seeing any GPU numbers.

Because after you see the result, it becomes very easy to invent a reason for why your result should still be acceptable.

So I wrote the gate first.

Then I ran it.

It failed.

At position 1:

K=1 → 0.440

K=3 → 0.376

K=5 → 0.360

K=7 → 0.362

And the same kind of split shows up at every concurrency level I tested.

So this does not look like one noisy cell.

What makes it even more interesting is that K=3, K=5 and K=7 are actually pretty close to each other.

The weird point is K=1.

Which means the wobble does not really look random.

The acceptance behaviour itself seems to move when the draft depth changes.

At that point I had two choices.

I could still calculate the oracle gap, show the number, and then explain why maybe the assumption is "close enough."

Or I could follow the rule I wrote before seeing the result.

So I stopped there.

The gap is not reportable.

And I am not going to show a number that I cannot properly defend.

But the funny thing is that the same GPU session answered two other questions, and honestly those became more interesting to me than the oracle gap itself.

Speculation actually lost to no speculation

This one surprised me first.

At concurrency 32, the run with speculation completely switched off did:

1049 tokens/sec

The best speculative cell did:

753 tokens/sec

So in this setup, simply turning speculation off made the server around 39% faster.

Initially this looked pretty weird.

Then I looked at the arithmetic and it started making much more sense.

The number that matters is basically:

How many useful output tokens am I getting for every position I push through the target model?

At K=0, that number is:

1.000

One position goes through and one token comes out.

At K=7, it becomes:

0.239

So I am pushing 8 positions through the verification pass to collect only around 1.9 tokens at the end.

And at this batch size the GPU is already busy.

Those extra draft positions are not free.

Every extra position is costing almost a full pass here, and the acceptance rate is simply not high enough to pay for that extra work.

So I do not think the draft model is "broken."

The arithmetic just does not work out for this workload.

But the heterogeneity I was looking for is real

This was actually the thing that made me start the experiment in the first place.

Same batch.

Same concurrency.

Same setup.

The repetitive prompts reached:

1.66 accepted drafts per round at K=7

The open-ended prompts were around:

0.50

That is roughly a 3.3× spread inside the same batch.

And this is exactly the thing that made me suspicious of using one mean for the whole batch.

Different requests really are behaving very differently.

So the original premise survived.

The thing that did not survive was my tool for measuring how much that heterogeneity is actually worth.

And honestly, I think that is a better result than getting some pretty oracle-gap number.

One correctness check before trusting any of this

Before trusting the throughput numbers, I wanted to make sure the speculative pipeline itself was not silently changing the output.

At temperature 0, speculative decoding should produce the same tokens as plain decoding.

That is basically the deal.

So I compared the completion length for every prompt across every K.

At concurrency 32, I had 768 prompts.

Every single one matched exactly.

So the pipeline is lossless here.

Which means the throughput difference is a real cost outcome and not some symptom of the generation pipeline being broken underneath.

Two bugs that ate way too much time

There were also two bugs during this project that wasted a ridiculous amount of time.

But I think both of them taught me something useful.

The first one was almost funny afterwards.

Every measured request came back as:

HTTP 500

Not some requests.

Every single request.

But warmup was passing every time.

Same server.

Same prompts.

Same model.

Same endpoint.

So I stared at this for a while and kept asking:

What is actually different between warmup and the real benchmark?

Same temperature.

Same max_tokens.

Same everything.

Eventually I found one difference.

My warmup request did not send a seed.

My measured request did.

I was sending:

seed

But in this SGLang version the field is actually:

sampling_seed

The server constructs SamplingParams(**kwargs) from whatever I send inside sampling_params, and it does not silently filter out unknown fields.

So my seed field caused a TypeError on the server.

The client just showed me an empty HTTP 500 with no useful body.

The real error had been sitting inside the server log the entire time.

I found it by reading the log instead of guessing.

Which somehow is a lesson I keep having to learn again.

The second bug looked like an OOM.

The launcher died with:

-9

And if you have spent enough time debugging things on rented GPU boxes, your brain immediately says:

OOM killer.

So naturally I started looking at memory.

But it was not memory at all.

SGLang had hit a ValueError in the draft model's context-length check.

Then during cleanup it killed its own process tree.

The launcher dying with -9 was just the final side effect.

The actual error was sitting three lines above the kill message in the server log.

So both bugs had basically the same shape:

The thing that killed me was not the thing that was reported.

And maybe that is one sentence I should keep somewhere near my terminal.

The local tests also fooled me

This part actually bothered me more than the bugs.

More than 400 tests were passing on my laptop.

Meanwhile, the real SGLang server was rejecting every single measured request.

How does that happen?

My mock server accepted whatever payload I handed it.

So all of my local tests were checking whether my code behaved correctly against my own fake server.

They were not checking whether I was actually speaking the same API language as SGLang.

That test does not work like that anymore.

If this project left one useful engineering improvement behind, it is probably that.

Where I stopped

I stopped the experiment here.

The gate says that the method cannot honestly answer the question I built it to answer.

And I would rather write that down than publish a clean-looking number that I cannot stand behind.

There is also one very important boundary around the performance numbers above.

I ran using the Triton attention backend.

SGLang's own EAGLE3 tests use FlashInfer.

I used Triton mostly because it kept FlashInfer off the attention path and saved me from pulling another wheel.

At the time it felt like a harmless convenience.

Now I do not think I should treat it that way.

The verification pass attends over K+1 positions at once.

And that is exactly the kind of shape where one backend may behave very differently from another.

So the honest interpretation of my result is not:

EAGLE3 is slower.

It is:

On this workload, with this setup, EAGLE3 using the Triton attention backend was slower.

That boundary matters.

If I run this experiment again, changing the attention backend is probably the first thing I would do.

And I think that is actually the artifact for me.

Not:

"Adaptive K does not work."

I have not shown that.

What I originally wanted to understand was what gets lost when we compress the behaviour of an entire batch into one scalar.

And somewhere along the way I ran into an even more basic problem.

The acceptance curve that the scalar is supposed to summarize does not seem to stay fixed when K changes.

So before asking whether one number is enough to summarize the curve, I first need to understand:

what exactly is the curve I am summarizing?