Expert cache, live connecting

Lookup hit rate · last 10 s (whole run below)

L0 GPU hit
–
arriving / read in flight
–
L1 RAM hit
–
L2 SSD miss
–
Lookups / s
–

Throughput and links · last 10 s

Tokens / s (total)
–
L2 → L1 · SSD read
–
L1 → L0 · RAM to GPU
–
L0 → L1 · GPU to RAM
–
GPU decode (FP8 out)
–
L0 usage · L1 usage
–

Per GPU

GPUmodelbusymemorypowertempSM clockPCIe inPCIe outexperts on GPUtok/s
waiting for data

Why RAM → GPU isn't at 100%

last 10 s
whole measured run

Why SSD → RAM isn't at 100%

last 10 s
whole measured run

GPU 0 busy and PCIe

busy %PCIe in, GB/s × 10PCIe out, GB/s × 10

Tokens / s

per second10 s average

Lookup outcome, 5 s windows

L0 GPUarriving / in flightL1 RAML2 SSD

Link throughput, GB/s

SSD → RAMRAM → GPUGPU → RAMGPU decode

Queue

experts missingSSD reads in flightuploads pending

Cache usage

L0 GPU used / slotsL1 RAM victims / slots

Events / s

SSD readsRAM promotionsGPU decodesscratch hits

Expert lifecycle (seconds, whole run)

Time on GPU before eviction

Idle on GPU before eviction

Gap between uses

Eviction to next load

No free GPU slot (upload retried)
–
RAM hit, no GPU slot yet
–
No free RAM slot (read retried)
–
Busiest layer now
–

Finished runs (from the pod)

runmodeGPU store / poolguessestok/sL0 hitL1 hitL2 missSSD GB/sRAM→GPU GB/slongest layer wait

API: /metrics (Prometheus) · /api/v1/summary · /api/v1/series · /api/v1/gpus · /api/v1/runs