LMCache, Swift's s3api, and the Shim I Did Not Want to Write

I run inference nodes for Erebine. Each node is vLLM with
LMCache bolted to the side of it in multiprocess mode: L1 is a slab of CPU
memory the engine spills KV cache into, and L2 is whatever durable thing you
point an adapter at. For months L2 was a directory on the same disk as the
model store, which is the storage equivalent of keeping your spare tire in the
trunk of the car you are currently driving into a lake.

I wanted two things. A disk tier on its own block device that could not fill
that device, and a third tier behind it that outlives the node. I already had
the third tier. Every OpenStack cloud I touch has Swift, and Swift has an s3api
middleware, and LMCache has an S3 adapter. This was going to be a config
change. I want that sentence on the record, because it is the last optimistic
one in this post.

TL;DR

• LMCache 0.5.4's fs_native disk adapter cannot store a single salted key.
  The C++ side parses three or four @-separated fields; the Python side
  sends five. Every store fails and the log never says why. Upstream has known
  since June; four pull requests, none merged.
• The plain fs adapter works but declares no capacity, so the server's own
  eviction controller ignores it and the tier grows until the disk is full.
• The s3 adapter only speaks virtual-hosted addressing. Swift's s3api, as
  deployed almost everywhere, cannot answer that. Hand it a bare host and it
  sends PUT //<key> and gets an HTTP 500 for its trouble.
• All three are fixable from Python except the first. So I wrote a launcher
  that patches the adapters in place before running the stock lmcache CLI,
  ships in every agent image, and retires one patch at a time as upstream
  catches up.
• With it, L1 in RAM, L2 on a local disk, L3 in a Swift bucket is a JSON
  array. Reads prefer the disk, writes go everywhere, and there is one caveat
  you need to measure before you trust it.

Everything below was verified against lmcache 0.5.4, vLLM 0.27.1, and a
Qwen3.8-27B node running tensor parallel 2 with fp8 KV. Your numbers will
differ. Your bugs will not.

While this post is a bit of a rant, I will say upfront, I really like LMcache and think they're doing a great job with everything. I ran into problems for my setup, but I fixed them, and I did so durably, without needing to create a nonsensical fork.

The disk tier that had been writing nothing

The first sign was a warning. Thousands of them, actually, one per store
batch, each a screen wide:

warning erebine-eim-agent: component=lmcache lmcache_source=store_controller.py:776
  Store task 342 to adapter 0 failed for keys: [ObjectKey(chunk_hash=b'^\x00\xaa+...',
  model_name='/var/lib/inference/.cache/erebine/models/093C2BF8-...', kv_rank=33620481,
  object_group_id=0, cache_salt='b6cc774bcaf5cb3b...'), ...]

Note what is missing. There is no reason. The store controller learns that an
adapter reported failure and prints the keys, with the affection of a hospital
discharge form. I went looking for the error string, and here is the chain
that eats it: the native connector's batch-set path has no per-key exception
handler (the get and delete paths do), so the first bad key throws out of the
whole tile; the worker catches it and stashes e.what() in the completion;
and the Python thread that consumes completions builds its result from
the boolean and drops the string on the floor. Nothing reaches stderr. You
could tail that log until the heat death of the universe.

So I ran the connector by hand, inside the agent image, against a scratch
directory, with the exact key string the server builds:

from lmcache.lmcache_fs import LMCacheFSClient
client = LMCacheFSClient("/tmp/probe", 1, "", False, 0)
key = "/var/lib/.../093C2BF8-...@02010201@1@72ed8b...@b6cc77..."   # five fields
fid = client.submit_batch_set([key], [memoryview(bytearray(4095))])
# drain completions -> (fid, ok=False, "FSConnector: malformed key (expected 3 or 4 '@'-separated fields)", None)

There it is. Some time in June, upstream added object_group_id to the
ObjectKey (hybrid models like Qwen3.5 and 3.8 keep their linear-attention
state in a second object group) and taught the Python serializer to emit it.
Nobody taught csrc/storage_backends/fs/connector.cpp. Unsalted keys are four
fields, which the C++ side happily misreads as "salted" and writes to a
coincidentally correct filename. Salted keys are five fields and throw. If you
isolate tenants with cache_salt, and you should, your disk tier has been an
elaborate way of warming the page cache.

The fix is a few lines in key_to_filename. Upstream has it as
#4869, and before that as
#4034, and before those
#4396 and
#3837, the last two closed
without merging. Four attempts to make a string parser count to five. I have
opinions about that, and I am keeping them in the notebook.

Because the connector is compiled C++ inside a pip wheel, I could not fix it
from the outside without rebuilding the extension. So I stopped using it.

The adapter with no bouncer

The plain fs adapter is pure Python and parses the five-field key just fine.
It also tracks the bytes it stores, notes every load for LRU recency, and
implements delete. It has everything an eviction needs except a capacity.
Its constructor calls super().__init__() with nothing, and the storage
manager, on seeing max_capacity_bytes == 0, logs that the adapter "does not
support global eviction" and declines to wire it into the L2 eviction
controller. The tier grows. Forever. On the disk with the models.

That one is a Python patch, and it is the smaller half of the shim.

S3 on Swift, or how to get a 500 for every chunk

Swift's s3api middleware is a good piece of software with three properties
that matter here, none of them bugs:

1. It expects the bucket in the path: /<bucket>/<key>. Virtual-hosted
   requests, where the bucket is a hostname prefix, need storage_domain set
   in the proxy config, and almost nobody sets it.
2. Your cloud probably has no wildcard DNS under the Swift endpoint, so
   <bucket>.swift.example.net does not resolve anyway.
3. Your TLS certificate names the endpoint and maybe www, not *., so even
   with DNS the handshake fails hostname verification.

LMCache's S3 adapter is built on the AWS Common Runtime and is virtual-hosted
only. It sets the Host header to <bucket>.<host> and puts the raw object
key in the path. The docstring says so; "path-style addressing is not
supported." I read that and sighed...

What happens if you give it the bare host instead? The object key for a KV
chunk starts with the absolute model path, so "/" + url_quote(key) becomes
PUT //var/lib/inference/.cache/...%40.... Empty bucket segment. s3api
answers 500. The CRT maps a 500 to AWS_ERROR_S3_INTERNAL_ERROR, and the
adapter raises it inside a completion callback, so you get a traceback per
object with the words "internal server error" and no hint that the request
was malformed. I reproduced it byte for byte with a 1 MiB payload and again
with a 34 MiB one, which is the size of a real chunk on this model.

The good news, all of it measured against a live Swift endpoint:

So the store is fine, the client library is fine, and the only thing between
them is a Python function that builds a path. Upstream has that too:
#3896 adds an s3_bucket
key for path style. Open since June. There is a pattern forming.

The shim, and how LMcache got its grove back

I did not want a fork, and I did not want sed running over site-packages at
image build time, invisible at runtime and fragile across upgrades. What I
wanted was one file, with a header that says what it patches and when each
patch can be deleted, that runs the stock CLI on the same argv. So the agent
images now launch erebine-lmcache instead of lmcache. It is about 200
lines of Python, and the interesting parts fit here.

The S3 half wraps the config parser to learn one key and wraps the module's
request constructor so every path the adapter ever builds, stores and bucket
listings alike, gets the bucket in front of it:

def patch_s3_path_style(s3):
    state = {"bucket": ""}
    orig_from_dict = s3.S3L2AdapterConfig.from_dict.__func__

    def from_dict(cls, d):
        cfg = orig_from_dict(cls, d)
        state["bucket"] = str(d.get("s3_bucket", "")).strip("/")
        return cfg

    s3.S3L2AdapterConfig.from_dict = classmethod(from_dict)
    orig_request = s3.HttpRequest

    def http_request(method="GET", path="/", headers=None, body_stream=None):
        if state["bucket"]:
            path = "/" + state["bucket"] + path
        return orig_request(method, path, headers, body_stream)

    s3.HttpRequest = http_request

The Host header stays the bare endpoint, the object name keeps its leading
slash (Swift does not mind, and it means listings round-trip), and the stock
parser ignores the extra key, so the same JSON through the unpatched binary is
merely broken instead of rejected.

The fs half has one subtlety. Setting the capacity is easy; you wrap
__init__ and assign _max_capacity_bytes before the storage manager looks.
Seeding is the part that bit me. The objects already on disk need to be
registered so a restart resumes from real usage, and they need to reach the
LRU policy, but the policy is attached as a listener after construction. So
the seed runs lazily on the first get_usage() call, which the eviction
controller makes once per second once everything is wired. And then the LRU
inserts a batch in reverse order, on purpose, so the later chunks of one
request evict before the earlier ones. Pass an age-sorted directory listing
into that and you have built a most-recently-used cache. Newest first fixes
it. I found that one with a test, not in production, which I mention only
because it is the one time in this story that happened.

def get_usage(self):
    if not self._capacity_seeded:
        with self._capacity_seed_lock:
            if not self._capacity_seeded:
                self._capacity_seeded = True
                _seed_fs_from_disk(self, fs)      # newest first; the LRU reverses a batch
    return orig_get_usage(self)

I verified the whole thing inside the real image against the real classes, not
stubs: an fs tier with a 4 MiB cap and 6 MiB already on disk, a real
L2EvictionController started against it, and within two seconds the oldest
object was gone and usage sat at 0.75. Then I started the actual server with
two --l2-adapter flags and read its status page. Both tiers present, each
with its own capacity, both healthy.

L1, L2, L3

LMCache has no L3. What it has is L1 plus an ordered list of L2 adapters, and
the docs call the list a cascade. Repeat --l2-adapter and the default
policies do exactly the sensible thing: every chunk written to L1 is stored to
every adapter, and a lookup loads each chunk from the lowest-indexed adapter
that has it. Put the disk first and the bucket second and you have a local
read preference with a durable backstop. Each adapter evicts on its own
eviction block.

The agent takes that as one environment variable. An object is one adapter;
an array is a cascade, one flag per element in order, each element copied out
of your text byte for byte because I have been burned by JSON re-encoders
before:

EREBINE_AGENT_LMCACHE_DISK_PATH=/var/lib/inference/.cache/erebine/lmcache
EREBINE_AGENT_LMCACHE_DISK_SIZE_GB=150
EREBINE_AGENT_LMCACHE_L2_ADAPTER=[{"type":"fs","base_path":"/var/lib/inference/.cache/erebine/lmcache","max_capacity_gb":150,"eviction":{"eviction_policy":"LRU","trigger_watermark":0.8,"eviction_ratio":0.2}},{"shared":true,"type":"s3","s3_endpoint":"swift.example.net","s3_bucket":"kv-cache","s3_region":"<whatever your s3api location is>","s3_prefer_http2":false,"aws_access_key_id":"...","aws_secret_access_key":"...","max_capacity_gb":2000,"eviction":{"eviction_policy":"LRU","trigger_watermark":0.8,"eviction_ratio":0.2}}]

The credentials are the EC2-style pair from openstack ec2 credentials create. s3_prefer_http2 is off because the endpoint negotiates no ALPN and
there is no reason to let the CRT try. shared goes on the bucket, which
several nodes may write, and not on the disk, which they may not.

Two things the docs will not tell you and the code will:

• Nothing promotes. A chunk that only survives in the bucket is served
  from the bucket every time until the engine regenerates it. The store
  controller deliberately ignores prefetch writes, so a load from tier two
  never lands in tier one.
• The slowest tier pins L1. The store controller holds a read lock on the
  chunk's L1 entry per adapter until that adapter finishes. At 10 MiB/s
  upstream, a 67 MiB chunk keeps its 67 MiB of L1 locked for seven seconds
  while Swift catches up, and L1 cannot evict a locked entry. Measure the
  upload rate from the node before you put a remote tier second. From the
  wrong side of a WAN this is a way to fill L1 with things you cannot evict,
  which is a novel failure mode and not a feature.

Things that bit me so they do not have to bite you

• The region string is not decorative. SigV4 puts it in the credential
  scope, and s3api compares it to its configured location. Send the wrong
  one and you get AuthorizationHeaderMalformed with the expected value in
  the message. Read the message.
• The object names carry the tenant salt, and the objects are plaintext.
  That is fine on a disk you own and a decision on a bucket you rent. LMCache
  ships an aesgcm serde that can wrap the S3 tier if it matters to you.
• The agent logs the full server command line at debug, credentials and
  all. Do not run production at debug. I say this as someone who is/was running
  production at debug :).
• raw_block, if you go that way, wants a slot big enough for a whole object.
  On this model the linear-attention state object is 37.4 MiB per rank, so
  the slot is 40 MiB, a multiple of 4096, and a 1 MiB slot fails every store.
  It also wants to open the device writable, and your container runs as an
  unprivileged user, so /dev/vdb answers EACCES until you add the disk
  group or, easier, point it at a fallocated file on the mounted volume.

FIN

The disk tier had been faithfully storing nothing for weeks and every health
signal was green. That is the part I keep coming back to, more than the C++
parser or the path style. A cache that fails closed and says so is a cache. A
cache that fails open and smiles is a page-cache warmer with a monthly
invoice.

All of it now lives in one launcher with a header that names each patch, the
upstream change that retires it, and the version to check. When 0.5.5 lands
with path style and someone teaches the native connector to count to five,
I get to delete code, which remains my favorite kind of commit.

If you run LMCache against Swift and this saved you a night, I would like to
hear about it. If you found a fifth field I did not, I would like to hear
about that too. If you also rage code at 01:56 and want to talk about it, I want to talk about that even more.