Building a quantized HiCache for SGLang

KV caching is probably one of the most impactful optimizations in LLM inference. If u think about it: without KV caching, every time we generate a new token, we're unnecessarily recomputing the previous tokens. With KV caching, we can simply reuse their previously computed keys and values.

For standard attention, this takes us from recomputing the entire sequence with O(S²) attention work to O(S) per decode step, where S is the current context length. And when you're serving LLMs at scale, avoiding that redundant computation makes an enormous difference.

But there's still a problem.

As context lengths grow and more requests arrive, the KV cache itself starts growing like crazy. And GPU memory (HBM) is limited. We cannot keep every reusable prefix in GPU memory forever.

So what do we do with the KV that no longer fits?

I think this is where hierarchical KV caching comes into the picture. So, instead of completely discarding that KV, we can store it in CPU memory and restore it whenever a future request needs the same prefix.

SGLng already does this through HiCache. Basically, it has two relevant tiers: L1, which is HBM, and L2, which is CPU memory (DRAM). So when we need to free up GPU KV-cache space, we can retain a copy in CPU memory rather than losing that reusable computation entirely.

In the normal HiCache BF16 path, the CPU copy is also stored in BF16. I was going through this implementation, and initially, that seemed perfectly reasonable. After all, we're just moving KV from one memory tier to another.

So, it you go through the docs, you'll find it very normal. But there is still one thing which seems a natural question for me.

If the CPU copy is primarily meant for storage, why do we need to keep it in the same precision as the GPU copy?

I mean, attention still needs BF16 KV on the GPU. That's fine. But what if we keep the active GPU KV in BF16 and store the CPU copy in a smaller INT8 representation? Or perhaps even INT4/FP4 with a different codec?

Whenever that KV is needed again, we could simply restore it to BF16 before attention consumes it.

And of course, there's a trade-off: quantization introduces some numerical error, and encoding/decoding adds computation. But if we can fit significantly more KV into the same CPU memory, perhaps the additional cache hits could outweigh those costs.

But hat question eventually led me to build HiQCache, which is basically an experimental quantized L2 KV cache for SGLang.

And the core idea is quite simple.: can I make SGLang's existing CPU KV cache more memory-efficient without changing its attention kernels, caching policy, or scheduler?

but the results are interesting

HiQCache reduced host KV memory consumption by 43.75% per token, allowing 1.78× more KV tokens to fit into the same CPU memory budget. In a shared-prefix workload, this also increased the cache-hit rate from roughly 57% to 73%.

And implementing it wasn't as simple as quantizing a tensor and copying it to CPU memory.

The real challenges were figuring out how to fit the compressed representation into SGLang's existing memory layout, how to reuse its CUDA transfer kernels without modifying them, and how to ensure that asynchronous KV restoration remained correct.

And i think that's where most of the good-level engineering happened.

So, let's dive into how I built it. :)

First, how does hierarchical KV caching work?

Before getting into compression, let's first understand what is actually being cached.

Suppose multiple requests share a long system prompt. When we process that prefix for the first time, the model computes the corresponding keys and values across its transformer layers.

When another request starts with the same prefix, we can reuse that KV instead of running the same prefix through the model again. That's the mainly basic idea of prefix caching.

Now suppose we have thousands of reusable prefixes. Keeping all of them in GPU memory can become expensive, especially when we're also using that memory for model weights, activations, and active requests.

So instead of relying on GPU memory alone, SGLang's HiCache introduces multiple storage tiers:

And one imp thing to understand: L1 and L2 here are logical KV-cache tiers, not the physical L1 and L2 caches inside the GPU. HiCache's L1 lives in GPU VRAM, while its L2 lives in CPU DRAM.

Ok, so when reusable KV is backed up to L2, HiCache can retain it even after the corresponding GPU cache entries are evicted. On a later prefix-cache hit, that KV can be restored to the GPU instead of recomputing the prefix.

And of course, restoring KV from CPU memory has a transfer cost. But for sufficiently expensive prefixes, transferring already-computed KV can be cheaper than recomputing it.

But there is still one limitation: the CPU cache also has a finite capacity. And that is mainly what i wanted to optimize.

Can we make the CPU KV cache smaller?

In the normal BF16 HiCache configuration, the CPU representation mirrors the GPU KV precision. That means we generally have the tokens or blocks in the same configuration of quantization in both the CPU and GPU.

Let's take Qwen3-8B (which i used for experiments). It has:

• 36 transformer layers
• 8 KV heads
• 128 dimensions per KV head
• BF16 KV values, which occupy 2 bytes each

For one token, across all layers, the total KV storage is:

KV bytes per token = 2 × layers × KV heads × head dimension × bytes/value = 2 × 36 × 8 × 128 × 2 = 147,456 bytes

The first 2 represents keys and values. So one cached token consumes 144 KiB across the model's layers. A long prefix containing thousands of tokens can occupy hundreds of megabytes.

With an 8 GB CPU cache budget, the tested BF16 configuration accommodates approximately 54,254 tokens.

*Now what if we store the CPU representation in INT8 instead? *

INT8 uses one byte per value instead of two. We cannot simply cast BF16 values to integers, because we'd lose the fractional information. We need a quantization scheme and some extra metadata to reconstruct the values.

So the compression will not be exactly 2×.

But even after including that metadata, we can significantly reduce the storage requirement. ( around 1.78x)

now the main design decision was to compress **only the CPU-resident representation; **(no change on gpu side)

so this gives us an important separation. The attention kernels still consume their normal BF16 KV tensors. The compression and decompression happen only when transferring KV between the two tiers.

I wasn't trying to redesign how attention works or how HiCache decides which prefixes to retain. I only wanted to change how the L2 cache stores those prefixes.

Building the INT8 codec

so the first thing is to implement the encoder and decoder.

The encoder takes BF16 KV values and produces a compact representation. The decoder takes that representation and reconstructs approximate BF16 values.

Since KV values are floating-point numbers, we need a way to represent them using the limited integer range of INT8.

I used symmetric, per-head INT8 quantization.

Let's understand that with a small example. Suppose a KV head contains these four values:

INT8 cannot directly represent 0.5 or -0.2. So we first find the maximum absolute value in the head and calculate a scale:

We can think of the scale as the real-number value represented by one quantization step. Next, divide each value by that scale and round it to the nearest integer:

Original BF16 INT8 code

1.0 ─────► 127 0.5 ─────► 64 -0.2 ─────► -25 0.0 ─────► 0

Now we store those integers together with the scale.

During restoration, we multiply each integer by the scale:

Reconstructed value = INT8 code × scale

Actually the recovered values will be close to the originals, but not necessarily identical. Quantization is lossy.

And for the actual implementation, I calculate one scale independently for each KV head, for each token and layer, separately for keys and values.

Why per-head rather than one common scale for the whole row?

Because different heads can have different value ranges. A head with values mostly around 0.1 shouldn't have to share the same quantization step as another head containing values around 100.

Per-head scaling gives each head its own dynamic range, with relatively little metadata overhead.

And the core operations are:

amax = x.abs().amax(dim=-1).clamp_min(AMAX_FLOOR)
scale = amax / 127

q = (x.float() / scale.float().unsqueeze(-1)).round()
q = q.clamp(-127, 127).to(torch.int8)

# Dequantization
x_restored = q.to(torch.bfloat16) * scale.unsqueeze(-1)

There are a few details to understand.

The quotient is formed in FP32 because BF16 division would introduce avoidable rounding error when calculating the integer codes. I also explicitly clamp values to [-127, 127] because rounding the scale to BF16 can make the computed quotient slightly exceed the intended range.

And what happens when an entire KV head contains zeros?

Without protection, its scale would be zero, leading to division by zero. So I introduced a small nonzero floor for the maximum absolute value. An all-zero head still stores zero integer codes and restores to exactly zero.

These are small details, but they're important when the codec is going to sit inside an inference runtime.

Why exactly 1,152 bytes?

The next question was how to store the quantized values.

For Qwen3-8B at TP=1, one token has 8 KV heads, each with 128 dimensions.

For a single K or V row:

INT8 values = 8 × 128 × 1 byte
            = 1,024 bytes

BF16 scales = 8 × 2 bytes
            = 16 bytes

Raw encoded row = 1,040 bytes

At this point, 1,040 bytes looks like the natural record size. But I couldn't use it directly with SGLang's existing CUDA HiCache mover.

The mover configuration used by this path requires the transferred element size to be a multiple of 128 bytes. Internally, it uses 32-thread copy groups operating on 128-byte groups.

And 1,040 is not divisible by 128.

The next valid multiple is 1,152.

So the final record becomes:

That gives us:

1,152 = 9 × 128

The 112-byte padding doesn't contain useful KV data. It exists to keep the record compatible with the existing mover.

This was a deliberate trade-off. I could try to eliminate the padding by changing the transfer implementation, but that would introduce a much larger integration surface for a relatively small storage saving.

Now let's calculate what this means for the complete KV cache. Original BF16:

HiQCache INT8:

So the reduction is:

And the relative capacity becomes:

In other words, the same physical CPU memory can hold approximately 1.78× as many KV tokens.

And this record brings the actual result. The next question was whether I could integrate that representation into SGLang without rewriting the existing caching system.

Integrating the codec into SGLang

SGLang's HiCache already handles the difficult parts of hierarchical caching: prefix matching, cache allocation, eviction, backup, restoration, and scheduling.

I didn't want to reimplement any of that. Instead, I traced the path that transfers KV between the device pool and the host pool. And the interesting part was that the L1–L2 traffic goes through the host-pool interface.

That gave me two places to intercept the transfers:

backup_from_device_all_layer(...) load_to_device_per_layer(...)

By implementing a new host-pool class, MHATokenToKVPoolHostINT8, I could override these methods and introduce encoding and decoding at the transfer boundary.

The integration uses three main components:

The rest of SGLang still sees a host pool with logical KV slots. It doesn't need to understand the encoded values stored inside those slots.

One imp useful detail is get_size_per_token().

HiCache uses the host pool's reported bytes per token to determine how many tokens fit into the configured memory budget.

So the new pool reports:

backup_from_device_all_layer(...)
load_to_device_per_layer(...)

Instead of reporting the original BF16 token size, it reports the encoded size.

That connects the smaller representation directly to increased logical L2 capacity, without modifying the scheduler or radix-cache policy. However, changing the host representation exposed another problem. lets see that in next section.

The problem with SGLang's CUDA mover

My first thought was simple: quantize the GPU KV and transfer the smaller representation to CPU memory.

But the existing HiCache mover is designed to copy bytes, not perform quantization. It accepts separate source and destination strides, but a single transfer element size.

That means it cannot directly transform a 2,048-byte BF16 row into a 1,152-byte encoded row during the transfer.

So I introduced GPU staging buffers.

And the idea is to finish the compression on the GPU first. Once the compressed bytes are ready, the existing CUDA mover can copy them normally.

For backup, the implementation encodes the selected K and V rows across the model's layers and then issues an all-layer transfer using the existing mover.

CPU to GPU: restoration

Here the compressed rows are copied back to the GPU staging buffer, decoded, and written into the regular device KV pool

I used separate persistent staging buffers for backup and restoration because the two directions can have transfers in flight concurrently.

Sharing the same buffer would create the possibility of one transfer overwriting records that another transfer hadn't finished consuming. The buffers also grow when necessary, rather than being freshly allocated for every transfer.

Also this added some GPU memory overhead, but it allowed me to keep using the existing HiCache CUDA mover.

Making asynchronous restoration safe

But that's not it, there was one more correctness issue I had to account for.

CUDA operations are asynchronous. The CPU can enqueue a transfer and continue without waiting for the GPU to finish executing it.

Now imagine restoring a cached layer:

1. Copy compressed KV to GPU
2. Dequantize KV
3. Write reconstructed KV into L1
4. Attention reads that KV

What happens if attention starts before step 3 finishes?

It could read stale or partially written KV values. And that type of bug might not cause a crash. It might simply produce incorrect tokens.

SGLang already has synchronization mechanisms for its transfer streams, so I reused those rather than introducing a separate synchronization system.

And the essential requirement is that the compressed copy, dequantization, and scatter into the device KV pool are all enqueued on the H2D transfer stream before the layer is marked ready.

CUDA preserves ordering within a stream, and the completion event establishes the dependency between the transfer stream and the model's forward stream.

This means the model cannot consume the restored KV before the required work completes.

I also had to account for the lifetime of staging allocations when buffers grow. Replacing a GPU buffer while another stream still uses it can lead to memory reuse before the queued work finishes.

I think generally these details don't appear in the quantization formula, but matter just as much for a correct implementation.

Measuring whether it actually helps

Once the compressed path was working, I wanted to verify two separate things.

First, does the new representation really save the memory predicted by the byte arithmetic?

Second, does that extra capacity translate into more useful cache reuse?

so I ran the experiments with Qwen3-8B at TP=1 on an NVIDIA RTX A6000 48 GB GPU.

The workload used 32 shared-prefix groups, approximately 2,048 tokens per group, 128 requests, and concurrency 8. I capped the GPU KV pool at 16,384 tokens so that L2 capacity would matter.

I used two experiments, each holding a different quantity constant.

Experiment A: same token capacity

Here I fixed both configurations to the same logical L2 capacity: 54,254 tokens.

The question was: how much CPU memory does each representation require to store the same number of tokens?

So the same cache capacity fits into 56.25% of the original memory.

This is the most direct test of the storage-efficiency claim because the two configurations have identical token capacity.

One implementation detail: I used --hicache-ratio rather than --hicache-size to configure this experiment. The latter takes integer gigabyte values, while the ratio allows more precise control of the logical token count. I also checked the actual capacity reported by the running host pool rather than trusting only the configuration arithmetic.

Experiment B: same CPU memory budget

Now let's reverse the experiment. Instead of holding token capacity constant, give both configurations the same 8 GB CPU memory budget.

The question becomes: does the smaller representation allow us to retain more useful KV, and does that improve prefix reuse?

I think, I had expected this behaviour.

The larger L2 cache retained more of the reusable working set, which meant more requests could restore existing KV instead of recomputing their prefixes.

But I didn't want to draw broad latency conclusions from one favorable run.

Across four paired fixed-budget measurements, the cache-hit-rate improvement was relatively consistent:

BF16 HiCache: 56.8% – 57.6% HiQCache INT8: 72.7% – 73.4%

The latency results were more variable.

Three of the four paired measurements showed substantial TTFT improvement, while one was near parity or slightly worse.

So I wouldn't describe HiQCache as a universally faster inference system. The more defensible result is that under this workload, the larger effective L2 capacity produced consistently more cache reuse.

And that distinction matters.

The extra cache capacity is a direct consequence of the record size. The serving improvement depends on whether the workload actually benefits from retaining those extra prefixes.

For example, if every reusable prefix already fits in BF16 L2, compression may not increase the cache-hit rate at all. In that case, the extra encoding and decoding operations could make HiQCache slower.

So, what about quantization error?

There is one important trade-off that memory benchmarks cannot answer.

INT8 quantization is lossy.

When HiQCache restores a compressed row, it produces BF16 values, but those values are not necessarily identical to the original BF16 KV.

So how much error are we introducing?

I tested the codec against an elementwise numerical error bound that accounts for INT8 rounding, storing scales in BF16, and rounding the reconstructed values to BF16.

The bound is:

The numerical tests verified this bound across CPU, CUDA, and Apple MPS, with no violations in the tested inputs.

But there is an important distinction between numerical correctness and model-level quality.

Even when individual KV values are reconstructed with small errors, those errors can propagate through attention and subsequent transformer layers. Sometimes small changes in the attention output can change the final token selection.

I built a quality-comparison harness to measure token agreement and logprob differences between BF16 and INT8 restoration.

However, the recorded quality experiment did not actually trigger L2 restoration in either configuration. So those requests didn't exercise the codec on the read path, and the result couldn't establish the quality impact.

And rather than report that comparison as valid, I marked the result invalid and added a check requiring actual L2 restore activity.

So the current evidence establishes that compressed restoration works and that the codec satisfies the tested numerical bound. It does not yet establish that generation quality is unchanged.

That is still remaining validation to complete.

What I kept out of this V1

The current implementation is intentionally narrow.

It targets Qwen3-8B with TP=1, 8 local KV heads, 128 dimensions per head, page size 1, and the layer_first host layout.

I haven't generalized the compressed record to tensor-parallel configurations, MLA, larger page sizes, or L3 storage.

For example, increasing tensor parallelism changes the number of local KV heads. That would change the payload size, scale metadata, padding, and transfer geometry. The fixed 1,152-byte layout is no longer appropriate without redesign.

So, rather than silently support a configuration that could produce incorrect KV, the host pool rejects unsupported geometries at initialization.

I also kept the codec implemented with PyTorch GPU operations rather than immediately writing fused Triton kernels.

A fused implementation might reduce kernel launches and intermediate operations, but I first need reliable per-phase profiling to know whether encoding and decoding are actually the bottleneck.

And I think there is no reason to introduce another specialized kernel solely because it sounds faster.

So, what I learned from building HiQCache

I think, the quantization itself was probably the simplest part of the project.

And the more interesting work started when I tried to make that smaller representation fit into an existing inference runtime.

A 1,040-byte record seemt reasonable until the CUDA mover required 128-byte alignment. Directly copying BF16 into compressed storage looked straightforward until I traced the mover's single-element-width contract. And restoring a compressed row looked like a normal copy-and-decode operation until I had to account for transfer streams, completion events, and GPU allocation lifetimes.

Each of these constraints made the final design.

The result is a relatively small change to SGLang's host-cache representation, without requiring changes to the attention kernels, radix cache, scheduler, or existing HiCache CUDA mover.

I think the most important result is that I could store the same logical KV cache in 43.75% less CPU memory, or retain 1.78× as many tokens under the same memory budget. In the measured shared-prefix workload, that extra capacity also produced more cache reuse.

There is still a lot to do, particularly deterministic end-to-end quality comparisons and detailed codec profiling. But that is also part of what made the project useful for me: understanding not just how to implement an optimization, but what needs to be measured before claiming that it is actually better.

The complete implementation, experiment scripts, raw results, and design notes are available in the repository.

GitHub SGLang PR Experiment A Experiment B

So, that's pretty much about HiQCache! :)

The main goal was to see whether we could make the existing KV cache more memory-efficient without changing how the rest of the inference system.

There's still a lot to explore, especially the impact of lossy KV restoration on generation quality and the overhead of quantization/dequantization under different workloads.

Anyway, I hope you enjoyed reading through this little experiment. If you have any thoughts, feedback, or ideas on improving it, I'd love to hear them! :)