Generate workloads, compare schedulers, model KV pressure and prefix reuse, separate prefill from decode, and search the SLO-constrained design space. The simulator is Python running locally in a Pyodide Web Worker.
The Python simulator executes in a background Web Worker and returns request-level virtual timestamps.
Run a simulation to generate a bottleneck explanation.
Run every colocated v0.3 scheduler against the current Serving Lab workload and rank by SLO attainment, then goodput.
Your Serving Lab workload and model controls are reused automatically.
| Scheduler | Goodput | SLO attainment | p95 TTFT | p95 E2E | KV peak | Unfinished | Diagnosis |
|---|
The planner repeatedly runs the simulator at different offered loads.
| Rate | Pass | Mean SLO | Worst repetition | Target | SLO range | Goodput | p95 TTFT | p95 E2E |
|---|
Compare colocated and prefill/decode-disaggregated serving, each with and without the current shared-prefix reuse scenario. P/D worker counts and interconnect settings come from Serving Lab.
Set shared-prefix and P/D parameters in Serving Lab, then run this controlled comparison.
| Scenario | GPU instances | Goodput | Goodput / GPU | SLO attainment | p95 TTFT | p95 E2E | p95 KV transfer | Cache hit | Prefill saved | Diagnosis |
|---|
Sweep 10 colocated candidates plus six P/D/cache variants, then identify the non-dominated goodput/TTFT frontier. This is an interactive design study, not an exhaustive optimizer.
The current workload, model, SLO, and P/D parameters are reused from Serving Lab.
| Candidate | Perf Pareto | Efficiency Pareto | SLO pass | GPU instances | Goodput | Goodput / GPU | p95 TTFT | p95 E2E | KV peak | Diagnosis |
|---|
Requests are generated from deterministic workload distributions and advanced through virtual time. Colocated runs model admission, prefill, paged KV allocation, dynamic decode batches, and completion. P/D runs use separate prefill and decode worker pools plus an explicit serialized KV-transfer link.
v0.3 models a single shared prompt prefix with configurable length and reuse fraction. Cache hits avoid redundant prefill work and share one persistent KV allocation. It is deliberately a controlled what-if abstraction, not a claim to reproduce SGLang's full RadixAttention policy.
Prefill and decode have separate accelerator profiles and worker counts. Prompt KV state crosses a modeled interconnect before decode admission. The simulator reports role utilization, transfer latency, and transfer pressure so the benefit of isolation can be weighed against data-movement overhead.
The Design Explorer reports both a raw-performance frontier and a resource-normalized frontier using goodput per accelerator. That prevents a multi-GPU P/D layout from looking unconditionally better merely because it uses more simulated hardware.
Vidur established the value of simulation for avoiding expensive deployment sweeps. Recent systems have pushed toward heterogeneous and disaggregated serving, communication-aware modeling, runtime optimizations, and SLA-dependent design-space exploration. InferScale-Sim remains intentionally smaller and inspectable.