UltraCompare

Single-stream decode on a roofline model. Weight bytes: W = P_total × q_w; active per step A_eff(B) = P_shared + P_experts_pool × (1 − (1 − k/n)^B) (dense: A_eff = P_active); KV per token C_kv = 2 × layers × n_kv_heads × head_dim × b_kv bytes (2 = K+V).

• Prefill (TTFT) = max( 2·N·P_active + 4·N²·L·H compute, W_resident / BW ) + W_offload / BW_offload + interconnect stall. The one-time transfer moves the weights that live in system RAM; the stall is the N-token block’s all-reduce (see below).
• Decode, single = (A_eff + L_ctx × C_kv) / BW_gpu.
• Decode, batch B = max( (A_eff(B) + B·L_ctx·C_kv)/BW, B·2·P_active / TF_eff ). MoE note: concurrent sequences activate different experts, so the weight bytes follow the expert union A_eff(B) (saturating at the whole pool), while FLOPs stay proportional to B × P_active. The MoE advantage decays to a dense model of the same size.
• Offload = the fraction f of the model that does not fit VRAM is streamed from system RAM every step over a two-stage path: DDR5 RAM in series with the PCIe link. Effective throughput is the harmonic mean 1/(1/BW_ram + 1/BW_pcie); DDR5-65 ∥ PCIe4 (25.2 GB/s effective) = 18.3 GB/s, not 26. If weights + KV + overhead exceed VRAM + RAM the page says so instead of guessing.
• Tensor parallel (2 cards): BW ×2, TF ×2, plus a per-layer all-reduce with two regimes: t_sync = max( 2·L·2·B·H·2 bytes / link, 2·L·lat_link ) with lat_link ≈ 10 µs over PCIe and ≈ 1 µs over NVLink. Small batches are latency-bound (70B, B=1: 160 syncs × 10 µs = 1.6 ms, negligible against a ~70 ms step, so TP over PCIe is fine for single-stream decode); a 2000-token prefill block moves ~10 GB, which is ~0.42 s of stall over PCIe 4 versus ~0.09 s over NVLink. That is the real killer.
• Pipeline parallel (2 cards): no all-reduce; the hidden-state hand-off 2·L·H·2 bytes / link is added to the step and half-overlapped with compute (prefill pays it per token).
• Interconnect: NVLink when the card has it (RTX 3090: 112.5 GB/s), otherwise the PCIe link × 0.8.

Quantization is bytes per weight parameter, metadata included: FP16/BF16 2.0 · FP8 1.0 · INT8/Q8_0 1.0 · Q5_K_M 0.7 · Q4_K_M (GGUF/AWQ) 0.6 · pure INT4 (no metadata) 0.5 · Q3_K_M 0.45. On a 70B model, Q4_K_M versus pure INT4 is a 20% difference in the headline number.

Efficiency constants, all visible here and tunable in one place (ETA in llm-calc.js):

• η_bw = 0.97: streaming-read efficiency, calibrated so the known ballparks reproduce: RTX 4090 + Llama 3 70B FP16 → 6.8 tok/s (recall ≈ 7) and Mixtral 8x7B FP16 → 36.4 tok/s (recall ≈ 38). Real llama.cpp / vLLM typically lands 10–15% below these numbers; scale every result by ~0.87 for a conservative estimate.
• η_compute = 0.65: matmul utilization vs peak FLOPS. FP8 runs on the card’s FP8 tensor rate when it has one; otherwise the FP16 rate (weights still 1 byte/param).
• η_link = 0.8: protocol factor on PCIe links; lat_pcie = 10 µs and lat_nvlink = 1 µs: all-reduce link latency; 0.5: pipeline hand-off overlap; 1.5 GB: driver context, activations and fragmentation overhead.

Hardware numbers (VRAM, bandwidth, FP16/FP8/FP32, NVLink, PCIe, TDP) are read from this repository’s comparison/ JSON specs, the same entries the hardware comparison page plots. Apple FP16 = FP32 1:1 on M1 and later (Apple publishes no GPU clocks or TFLOPS; the M4/M5 rates are derived from measured clocks), so Apple capability is bandwidth-first. Model geometries come from the public model cards: Llama 3 8B/70B (32/80 layers, 8 KV heads), Mixtral 8x7B (46.7B total, 12.9B active at batch 1; the shared/expert split is backed out so that holds), Qwen3 30B-A3B (30.5B total, 3.3B active, 128 experts × 8 active), DeepSeek-V3 (671B total, 37B active, 256 experts × 8, MLA KV approximated as one 576-dim head: conservative).

Shareable URL state: every control is encoded in the query string (?m=&q=&kv=&g=&n=&par=&ram=&rambw=&pcie=&pt=&ot=&b=&ctx=), same pattern as the main page.