Artifact 18
Building a safe async policy for RL rollouts in vLLM
Actually, I've been spending a lot of time around LLM inference lately. And one thing that I find particularly interesting is how much of RL training actually depends on inference infrastructure.
Think about it. We usually explain RL training as: generate responses, calculate rewards, update the model, and repeat. Which makes sense. But the model generating those responses is often running inside a separate inference engine. And the trainer keeps changing the weights that the inference engine needs to use.
So here's the question I started with: what happens if the trainer wants to update the weights while vLLM is already generating a response?
Maybe we can just transfer the new weights using NCCL and continue, right? vLLM already supports weight transfer. Why build anything else?
Well, I went through vLLM's implementation, and the more interesting problem wasn't how to move the weights. It was how to know that a policy transition was actually safe. An ongoing rollout could belong to one policy at the beginning and another at the end. And even if every request finishes, the KV cache might still contain tensors produced by the previous policy.
So I built RolloutCore: a RL rollout control plane on top of vLLM. It coordinates live weight updates, binds rollouts to a model generation, clears stale caches, and refuses to publish an update it cannot verify. It does all of this without patching vLLM or restarting the inference process for every update.
But rather than just showing you the architecture, I want to go through why I built each part, what I actually tested on GPUs, and the problems I ran into along the way. Some of the most useful findings were the ones where the system didn't behave as I initially expected.
First, where do rollouts come into the picture?
Let's take a simple example. Suppose we're training a reasoning model to solve math questions using RL.
We give it a prompt, it generates a response, and we use some reward signal to evaluate that response. The generated response is a rollout. If we sample eight responses for the same prompt, we have eight rollouts.
And a basic training loop looks something like this:
prompts
|
v
+------------------------+
| vLLM: policy version N |
| generate rollouts |
+------------------------+
|
v
rewards / advantages
|
v
+------------------------+
| trainer updates model |
| N -> N+1 |
+------------------------+
|
v
new weights for vLLM
|
+----> next rollouts
The trainer's job is to optimize the model. The inference engine's job is to generate rollouts efficiently. We want the inference engine to pick up the trainer's new weights and start generating from the updated policy.
And in a real system, those two things don't necessarily finish at exactly the same time.
Now suppose a rollout is meant to generate 256 tokens. It has already generated 100, and the trainer announces that the next checkpoint is ready.
What if we update the weights immediately?
One rollout (256 tokens)
[ tokens 1 ........ 100 | tokens 101 ................ 256 ]
[ policy N | policy N+1 ]
^
weights changed
Which policy generated this rollout?
It's not completely policy N, and it's not completely policy N+1. In ordinary inference you might not care about naming the exact policy behind every token, but for RL training that information matters. We need to know which behavior policy produced the samples we're using for optimization. This becomes especially important when working with policy logprobs and importance ratios.
There are asynchronous RL designs that allow more complicated version behavior. But they need the training side to account for it. For this project I wanted a stricter, easy-to-reason-about rule:
A rollout admitted under version N must finish before the model's weights can mutate to N+1.
In other words, one rollout, one committed policy generation.
But fixing this alone isn't enough. There's another problem hiding in the cache.
The second problem: your KV cache doesn't automatically know the policy changed
We all use KV caching to avoid recomputing the old token representations during autoregressive generation. And with prefix caching, a future request can reuse KV blocks for a prompt prefix that was already processed.
Now remember that these K and V tensors were produced using the model's weights.
So imagine this sequence:
Request A, with policy N
|
+--> computes prefix KV for prompt P
+--> cache stores that KV
Trainer installs policy N+1
|
+--> Request B arrives with the SAME prompt P
+--> prefix cache finds P
+--> can it reuse the old KV?
The tokens in the prompt are the same. But the model weights changed. And the K/V representations depend on those weights. So using KV computed under N while running the rest of the model under N+1 is generally incorrect.
This is a very different problem from a rollout spanning a weight update. Even if we drain every request perfectly, stale prefix KV may still be sitting there waiting for the next request.
That left me with two correctness questions:
- Can I guarantee that an admitted rollout does not cross a weight mutation?
- Can I guarantee that the new policy never uses cached state from the old one?
And then a practical question: can I solve both without restarting vLLM every time the trainer updates?
So what was actually missing from vLLM?
Before implementing anything, I went through vLLM's RL serving and weight-transfer code at commit 00b7847c8…, from September 24, 2026.
I wasn't trying to argue that vLLM had no RL support. Quite the opposite. It already provided the important low-level mechanisms: pausing and resuming the engine, an opaque weight_version label, and NCCL-based trainer-to-inference weight transfer.
But the mechanisms didn't automatically give me the contract I wanted. The audit surfaced two gaps.
First: weight_version wasn't bound to the request.
The engine could report a version, but the audited Request and RequestOutput didn't carry a per-request binding to that generation. This was not necessarily a bug. vLLM's asynchronous flow can intentionally support requests that span versions, including with pause(mode="keep"). But for strict one-policy-per-rollout RL, simply reading the current version after generation isn't proof of which weights produced it.
Second: finishing a weight update wasn't a cache-coherence boundary.
The audited finish_weight_update path did not reset the prefix, encoder, or multimodal caches. And the prefix-cache block hash didn't include a model-weight generation by default. So a caller performing a weight update also needed to establish an explicit cache-invalidation boundary.
Notice what's interesting here. Neither problem is solved just by broadcasting the new parameters successfully.
That's where I decided RolloutCore should sit: not another inference engine, not another NCCL backend, but a correctness layer around the existing vLLM lifecycle.
The contract I wanted before writing the integration
I started with a few rules rather than with GPU code.
The most important one is that we only accept new rollouts when the engine is READY. Once someone asks to update the policy, new admissions stop. Existing rollouts can finish, but the weights cannot change until the engine is drained.
The next rule is that the new version isn't considered committed just because the NCCL transfer returns. We still need to clear the caches and check what the engine reports. And I wanted those checks to happen while the engine is paused, not after we've let users generate on potentially wrong weights.
Finally, failures should be explicit. If a transfer fails halfway through, we can't casually say, "It's probably still the old model." We might have partially updated weights in GPU memory. That's the kind of uncertainty I didn't want the controller to hide.
So the lifecycle became:
UNINITIALIZED
|
bootstrap
v
READY <------------------------------+
| |
update requested |
v |
DRAINING |
no new admissions |
| |
no active work |
v |
QUIESCED |
engine paused |
| |
v |
UPDATING |
NCCL weight install |
| |
v |
INVALIDATING |
reset old caches |
| |
v |
VALIDATING |
still paused; verify |
| |
v |
RESUMING -------------------------------+
confirm resumed
ambiguous failure ---> TAINTED
(terminal in V1)
That's nine states, counting UNINITIALIZED and TAINTED. The key point isn't the number. It's that every state tells us what we're allowed to assume about the engine.
And I kept the main controller deliberately boring: pure Python, no vLLM import, no CUDA calls, no network access. It just knows the legal transitions, tracks active rollout bindings, and checks the evidence returned after an external operation.
The code lives in lifecycle.py. A separate runner.py executes the operations in order, while adapters are responsible for talking to the actual engine.
LifecycleController
(policy + invariants)
|
v
LifecycleRunner
(what operation next?)
|
v
LifecycleAdapter
(effects + observations)
|
+-----+-----+
| |
HTTP control NCCL trainer
plane weight transfer
| |
+-----+-----+
v
real vLLM
Why split these things? Because I wanted to test the correctness logic without needing to rent a GPU every time I changed a transition. Then the real-engine tests could focus on whether the adapter's observations matched what vLLM actually did.
And one detail I like about this design: an adapter doesn't get to advance the controller just by returning "okay". It returns typed evidence, and the controller checks the postconditions before advancing. A successful HTTP status and a successful engine operation are not always the same thing.
For example, vLLM's prefix-cache reset API can return HTTP 200 with {"success": false} if blocks are still in use. So the adapter checks the response body. Otherwise, a technically successful request could make us believe old KV had been cleared when it hadn't.
First I proved the lifecycle without moving a single weight
I didn't immediately try the full NCCL path. It would have made debugging much harder: if a run failed, was it the lifecycle, the HTTP API, the NCCL group, a CUDA issue, or the checkpoint itself?
So the first real GPU experiment was intentionally limited. I used facebook/opt-125m with a running vllm serve on an RTX A6000 and tested the control plane only.
That meant bootstrapping a fresh engine, checking its label, sending a 256-token request, asking it to pause with mode=wait, waiting for the request to finish, resetting caches, validating, and resuming. No tensors moved, so I didn't pretend I'd proven weight updates yet.
The important observation was the drain. The 256-token request took about 2.020 seconds. The pause completed about 2.131 seconds after it started. So it didn't silently abort the active request or return immediately while generation was still going on.
The control-plane operations excluding that wait took about 46 ms in the recorded run.
And there was a useful sanity check: with no weight update, greedy generation before and after the drain/reset/resume sequence produced identical token IDs. That tells us the lifecycle itself didn't arbitrarily change what the model generated.
At this point I knew that the HTTP lifecycle I wanted could work against a real engine. But the biggest part was still missing: actually replacing its weights.
Then came the real NCCL weight transfer
This was more interesting than I initially expected.
vLLM already has a trainer-side weight-transfer stack, including WeightTransferTrainerFactory and send_weights(). I wanted to reuse those. Building a second NCCL implementation would just create new problems that the project wasn't meant to solve.
But there was a subtle integration issue: the weight-transfer engine itself coordinates the /start_weight_update, /update_weights, and /finish_weight_update calls while the NCCL broadcast is happening. So I couldn't independently call all those HTTP endpoints from my lifecycle runner and then ask NCCL to send the tensors. Some of those calls block waiting for the corresponding collective. Driving them twice or in the wrong order could deadlock the transfer.
So I split the responsibilities:
- RolloutCore decides when an update is allowed, what target version we're installing, and what to validate afterwards.
- vLLM's trainer-side
send_weights()owns the actual NCCL transfer and the interleaved update calls.
There was another detail in the audited vLLM path. Its trainer engine called finish_weight_update() without a target version argument, even though the HTTP sync client supported one. The weights could transfer successfully while the engine still reported the old label.
For RolloutCore that would be a real problem. Its validation would ask for rc-1, see rc-0, and correctly refuse to resume. But the transfer itself might have been perfectly fine.
I fixed that with a small target-aware wrapper around the existing weight-sync client. Before send_weights() starts, the wrapper receives the pending target. When vLLM finalizes the update without a version argument, it supplies the target label in that finalize call.
# Simplified from adapters/nccl.py
def set_target(self, target):
self._pending_label = target.label
def finish_weight_update(self, weight_version=None):
label = weight_version or self._pending_label
if label is None:
raise WeightTransferNotConfiguredError(
"finish_weight_update", "no target was set on the sync client"
)
self._inner.finish_weight_update(weight_version=label)
Why not just write rc-1 through /update_weight_version afterwards? Because that would be changing the label to match what we wanted, instead of getting it from the transfer's finalize path. It would make an inconsistent system look correct.
I also made the driver compare the declared target identity against the trainer's actual source metadata before sending anything. If they disagree, it should fail before mutation, not after we've started changing the engine.
To keep the setup honest, I first proved that an adapted version of vLLM's upstream NCCL example worked on the same GPUs. Only after that did I test RolloutCore's integration. Otherwise it would be too easy to blame my controller for a broken NCCL environment.
My first complete cycle: dummy weights to real weights
For this experiment, I used two RTX 3090 GPUs on one node: one running vLLM (TP=1), and one holding the BF16 trainer model.
The inference server started with --load-format dummy, so its initial weights were not the real checkpoint. That made the before/after comparison very easy to see.
READY (rc-0, dummy weights)
|
| generation: <s><s><s><s>... [16 repeated tokens]
|
+--> drain
+--> real NCCL broadcast of OPT-125M weights
+--> invalidate caches
+--> validate rc-1 while paused
+--> resume
|
READY (rc-1, real weights)
|
| generation: " the capital of the French Republic..."
The first full cycle completed in 0.673 seconds. The server process didn't restart. It was the same vLLM process before and after the update, and the post-update generation matched the reference generation from the earlier real-checkpoint run.
That was the first point where this stopped being just a lifecycle abstraction for me. The same running inference engine had actually accepted new tensors and resumed with a newly committed policy version.
But the test had no in-flight rollout. Which means it hadn't yet tested the main rule that made me build the project.
What happens when the update arrives during a live rollout?
Now let's make the test a little more realistic.
I started a 256-token rollout under rc-0 and waited until it was genuinely mid-generation. Then, while it was still running, I requested an update to rc-1.
Here's the expected behavior:
admit R-long under rc-0
|
+----> generation is still running
|
| update requested
v
DRAINING
|
| mutation is NOT allowed
|
R-long finishes
|
finish_rollout(R-long)
|
confirm_drained
|
QUIESCED
|
begin_update
|
NCCL installs rc-1
|
invalidate + validate
|
READY (rc-1)
And that's what the experiment showed. The active rollout finished with the old dummy-weight behavior. The journal recorded finish_rollout before begin_update, and the engine resumed under the new version only afterwards.
The recorded numbers were:
| Measurement | Value |
|---|---|
| Full cycle with a live rollout | 2.149 s |
| Time the drain held for that rollout | 1.375 s |
| Comparable cycle without active work | 0.673 s |
That 1.375-second delay is not controller overhead or NCCL time. It's how long the update had to wait for an existing rollout. In this design, waiting is the correct behavior.
And this test exposed a concurrency detail I didn't want to leave to chance. The request thread releases its rollout binding when its request returns, while the lifecycle thread is waiting for the engine to drain. Those two threads update and inspect shared state. I used a lock around the controller's state transitions and active-rollout bookkeeping instead of depending on Python's GIL to make a sequence of operations atomic.
There's also a small race at the boundary: the engine can become idle slightly before the request thread has released the local binding. If the engine reports a completed drain but RolloutCore still sees an active rollout, the controller refuses to assume things are safe. The experiment gave that release enough time to land; a more robust confirmation/reconciliation protocol would be useful beyond V1.
Now we had a working live update and a rollout that stayed on one policy. But had we really solved the cache problem?
The cache experiment: I deliberately made the bad behavior happen
This is probably my favorite experiment from the project.
It's easy to write a test that says, "After the update, there were zero stale cache hits." But what if the cache wasn't being used in the first place? Then the test proves almost nothing.
So I wanted two things: prove that prefix caching was actually working, and then show what happens when we intentionally skip the cache boundary.
I used a repeated prompt with prefix caching enabled and read vLLM's own prefix-cache metrics. Sending the same prompt again before an update produced 32 prefix-cache hits. Good. Reuse was real.
Then I did a safe update through RolloutCore. The first post-update request for that same prompt saw zero old-version hits. A second request after the update saw cache hits again, showing that caching wasn't simply disabled.
But I didn't stop there.
For the negative control, I performed a weight update outside RolloutCore with clear_cache=false and intentionally skipped explicit cache invalidation. The updated checkpoint was made visibly different by negating the first decoder layer's weights. This wasn't supposed to be a useful model; it was supposed to make any incorrect mixing observable.
And here's what happened:
After the weight update, same prompt, same target weights:
With stale cache left in place:
prefix-cache hits: 32
generated text: " Madrid Madrid Madrid Madrid ..."
After explicitly clearing the cache:
prefix-cache hits: 0
generated text: " a few few,,,,, and the the first..."
That's the result I wanted to know existed: old-policy KV was not merely kept in memory; it was actually reused under different model weights, and the generated output changed.
The zero-hit safe run alone wasn't enough to attribute safety specifically to the post-update reset, because that earlier harness also cleared cache during pause. I corrected the implementation so normal draining uses clear_cache=false and the INVALIDATING state is the explicit post-mutation boundary. The negative control, which skipped both clears, is what directly demonstrated the stale-reuse hazard.
One more small detail: a version-specific cache_salt helps isolate prefix KV hash entries, but it doesn't automatically version every other cache that vLLM may have. That's why RolloutCore explicitly requests prefix, encoder, and multimodal cache resets instead of assuming one salt solves everything. My actual GPU cache experiments were text-only, so the encoder and multimodal lanes haven't been tested with real multimodal data.
This was the moment the cache invalidation step went from "seems like the right thing to do" to something I had directly reproduced a failure without.
A version number is not the identity of the weights
While building this, I ran into another problem that is easy to miss.
Say I have two checkpoints of the same model architecture, one at training step 100 and another at step 500. They probably have the same parameter names, dtypes, and tensor shapes. If I hash only that parameter manifest, the digest will be identical even though the actual weights are different.
I saw exactly this with OPT-125M. Dummy-initialized weights, real checkpoint weights, and even the intentionally modified checkpoint could share the same manifest digest.
So I separated two ideas in RolloutCore:
WeightVersion answers, "Which committed generation is the inference engine serving?" For example, rc-0, rc-1, or rc-2.
WeightIdentity answers, "Which declared source was this generation meant to contain?" It can include a parameter-manifest digest plus checkpoint, trainer run, and training-step provenance.
When a rollout is admitted, RolloutCore creates a binding to the current committed version and identity. And the trajectory record can store that information alongside the prompt, generated tokens, and other available observations.
Trajectory R-long
policy version: rc-0
training step: 0
source identity: declared checkpoint A
generated tokens: 256
Trajectory R2
policy version: rc-1
training step: 1
source identity: declared checkpoint B
generated tokens: ...
In the recorded provenance experiment, both steps had the same manifest digest, but their provenance-qualified identities were different. That's what I wanted: the record should not accidentally claim the wrong checkpoint just because the architecture didn't change.
Of course, I don't want to overstate it. Declared provenance isn't a byte-level hash of GPU memory. It tells us which source the trainer declared and what the update process reported, not a cryptographic proof of every tensor value. RolloutCore also includes a token/logprob replay validator, but the committed replay example exercises token comparison with synthetic replay observations; real-engine logprob replay remains future work.
And then I tried to break the update itself
What if a weight update fails halfway through? Can we just retry and continue serving?
Not safely, at least not without more information.
Imagine NCCL has already updated some tensors, and then the trainer loses its connection to the inference process. We now have three possibilities: nothing landed, everything landed, or only part of the model changed. If we don't know which one happened, we cannot truthfully say the engine is serving either the old generation or the new one.
That's why RolloutCore has TAINTED.
For errors that prove no mutation happened, it can refuse the operation without tainting. For incomplete drain observations, it can keep waiting or allow an appropriate recovery. But if a mutating operation has an ambiguous outcome, it refuses to publish the target or admit more rollouts. In V1, TAINTED is terminal, because blindly rolling back isn't safe without a way to prove the old tensors were restored.
There was an especially interesting difference between the small and large model tests.
With OPT-125M, killing the engine around the update caused the transfer call to fail promptly. The controller could classify the error and enter TAINTED in about 0.177 seconds, without committing the target.
But with Qwen3-8B, I killed the inference engine roughly halfway through the NCCL broadcast. This time the trainer-side collective didn't return within the experiment's 120-second budget. The engine was dead, the target wasn't published, and the controller was still sitting in UPDATING, waiting for a call that never completed.
So there are two different properties here:
- Safety: never publish a new generation if the update hasn't been verified.
- Liveness: recover or return an error in bounded time when something goes wrong.
The 8B experiment preserved the first property by never publishing the target. It did not provide the second. Without a return from the blocked NCCL collective, the synchronous runner couldn't classify the failure and move itself to TAINTED.
That's a limitation I would rather show than hide. It points toward proper collective timeouts, supervision, recovery/ownership protocols, and possibly upstream changes for a production system.
A few implementation problems I didn't expect
Getting a design right is one thing. Getting it to work in a real GPU environment is another. Some details cost more debugging time than the state machine itself.
1. NCCL plus CUDA device context can be surprisingly easy to get wrong. PyTorch's current CUDA device is thread-local. The vLLM packed-tensor producer uses CUDA streams associated with the current device, so starting a transfer from a different thread without setting that device can point the transfer at the wrong GPU. In my integration notes this surfaced as NCCL's rather unhelpful Cuda failure 400: invalid resource handle. The driver now records the initialization device and refuses a transfer from a thread whose current CUDA device doesn't match. A caller must select the right device in that thread.
2. A successful HTTP request isn't always a successful operation. As mentioned earlier, reset_prefix_cache can answer HTTP 200 and still report success: false. That's why typed evidence and explicit postcondition checks are not just extra ceremony in this project.
3. The receive buffers need GPU memory too. My first 8B attempt ran into a server-side CUDA OOM: the transfer path wanted roughly 1.16 GiB while only around 531 MiB was free. The KV cache had already occupied too much of the GPU memory budget, leaving insufficient room for the weight-transfer receive buffers. Reducing the server's --gpu-memory-utilization to 0.75 and using --packed-num-buffers 1 let the measured 8B update complete. That's an important practical lesson: a model that fits for serving doesn't necessarily have enough headroom for live weight replacement.
4. Request completion and local bookkeeping aren't the same instant. vLLM can finish a request before the application thread records its completion in RolloutCore. A strict drain must account for both observations instead of assuming one implies the other. That's why the controller checks the engine's drain completion against its own active-rollout count, and why the lock and completion timing mattered in the concurrency experiment.
None of these details is particularly glamorous. But they are exactly the things that distinguish a working GPU experiment from an architecture diagram.
Okay, but is it actually faster than restarting vLLM?
Correctness was the primary goal, but I also wanted to know whether the approach was operationally useful. So I compared a hot update with terminating vLLM and starting it again on the target checkpoint.
The final Qwen3-8B measurement was on one node with 2× RTX 3090 24 GB GPUs: TP=1 inference on one card and a BF16 trainer model on the other. The path was PCIe without usable peer-to-peer access or NVLink in that setup, so NCCL's effective transport was host-mediated.
Here are the numbers from the committed GPU artifacts:
| Model | RolloutCore hot transition | Full restart → ready | Restart / hot ratio |
|---|---|---|---|
| OPT-125M | 0.230 s | 30.032 s | 130.6× |
| Qwen3-8B | 4.254 s | 37.045 s | 8.71× |
The comparison is deliberately about model-transition time. It's not a claim of 8.71× more tokens per second or 8.71× higher RL training throughput. The restart baseline also assumed the target checkpoint was already available on local disk.
And the difference between the 125M and 8B ratios is important. Restarting pays a large startup cost. Hot updating avoids that startup cost, but still has to move the model weights. So as the model gets larger, the transfer becomes a much bigger fraction of the hot-update time.
For Qwen3-8B, the measured update phase took about 4.129 seconds out of the 4.254-second total cycle, roughly 97%. That's around 16 GB of BF16 parameters moved at about 3.9 GB/s effective in that configuration.
Qwen3-8B hot transition: 4.254 s total
drain/poll + control + checks ~0.125 s
actual weight-update phase ~4.129 s
97% of the cycle
spent in UPDATING
So the controller itself wasn't the main cost in this run. The data transfer was. With a different interconnect, a different tensor-parallel layout, or different serving settings, those measurements would change. This is why I don't think reporting a big speedup without the hardware and baseline is particularly useful.
For this workload and hardware, though, it answers the practical question: yes, a safe in-process transition can be much cheaper than restarting the engine.
What RolloutCore does, and what it deliberately doesn't
I think it's important to draw the boundary here.
RolloutCore gives me a strict policy-generation contract when the application actually routes rollout admission, completion, and updates through it. It owns the ordering, version binding, cache-reset boundary, and conditions for publishing a new generation. It doesn't patch vLLM, and it doesn't intercept arbitrary clients sending requests directly to the underlying server.
It also doesn't make RL training automatically on-policy. Even if every trajectory is attributed to exactly one model generation, the training system still has to manage sample freshness, rewards, behavior-policy probabilities, and whatever off-policy corrections its algorithm requires.
And V1 is intentionally narrow: a single vLLM engine, single-node NCCL setup, strict blocking updates, and text-only cache experiments, with the committed GPU weight-update tests at TP=1. It doesn't yet demonstrate multi-node fan-out, asynchronous overlapping rollout policies, tensor-parallel updates beyond TP=1, multimodal cache correctness, or bounded recovery from a stuck collective.
Why make those limitations explicit? Because they tell us which problems were actually solved and which ones are interesting next.
For example, you could imagine keeping two inference replicas on different policy generations so one continues generating while the other updates. Or an async training system that permits staleness but records behavior policy information accurately. Both could improve utilization, but both also introduce a more complicated coordination problem. I'd rather establish a strict, tested baseline before making that trade-off.
What I took away from building it
The most interesting thing about this project wasn't the final state machine. It was realizing that moving the weights is only one small part of updating a serving policy.
NCCL can transfer tensors. vLLM can pause and resume. But between those operations we still have to answer some very practical questions: Is an old rollout still running? Which policy generated these tokens? Can the next request accidentally reuse old KV? Did the new version really pass the checks before we made it available? And if a collective fails halfway through, what exactly are we allowed to claim?
I started with those questions, built the smallest control plane that could enforce a strict answer, and then tested it against real vLLM processes. The experiments gave me a working hot-update cycle, a measured drain boundary, a direct stale-cache counterexample, and a useful negative result from a collective that never returned.
There's a lot more to explore here, especially around asynchronous RL and multi-replica serving. But I think this is a good foundation: first make the version transition correct and observable, and then work on making it more concurrent.
If you're working on RL rollout infrastructure or vLLM weight updates, I'd love to know how you're handling these boundaries. Especially the trade-off between strict policy consistency and keeping the inference GPUs busy.