Newton ASV benchmark check — 2026-08-06

Daily automated regression check across the Newton benchmark suite (ASV). Findings reference public newton-physics/newton PRs; fleet-internal machine detail is omitted.

Summary

• Nightly-wheel publish-race flake: resolved. Structurally closed by #3780 — it pins the ASV install step to a stable warp-lang release and raises the pyproject floor so pip can no longer resolve a pre-release nightly at all. It re-arms once main re-pins to a dev nightly after the 1.5 cut, so the durable belt-and-suspenders is --only-binary=warp-lang.
• The fleet-wide NotifyDRLegs 2–7× jump is real on the dashboards but NOT a product regression. A controlled A/B (newton × warp, at a fixed commit) came back flat on both axes — exonerating the warp wheel and every newton commit in the bracket. notify(MODEL_PROPERTIES) is a code no-op (a device sync), so the extra cost is added launch/sync overhead: a benchmark-environment event at the 08-04 boundary, not the code. It belongs with the perflab/infra side, not the solver.
• One genuine product regression: KpiInitializeSolver (SolverMuJoCo construction) +12–24%, scaling with worlds × shapes. Isolated to solver construction (KpiInitializeModel is flat). Prime suspect #3714 (the new collision-mask graph compiler). It shipped in v1.5.0rc1 — worth a deliberate author sign-off (likely accepted-cost-for-correctness, but it also hits end-user solver creation, not just the benchmark).
• KpiDRLegs [4096] shipped dark into v1.5.0rc1 — a 5th consecutive null-run day, null even on a fresh high-memory (122 GB, unified) node, which argues graph-capture failure rather than OOM. The only 4096-world KPI has zero data in the RC — repeat escalation, now with release urgency.
• Teleop metrics reclassified: the CPU-backend quality numbers are deterministic per (arch, python) and step only at dependency bumps (mujoco-warp / the mujoco C-engine) — not the "noise class" they'd been labelled. Nothing moved in-window.
• Clean signals: hash-rewrite holding at 0 mismatches; two small (+32 MB) memory steps that are footprint increases, not leaks; the real-workload benchmarks (FastDRLegs, bench_mujoco) flat across the warp boundary — so warp-1.16.0-final is perf-neutral for real work.

What this says for benchmark monitoring / regression notification

This is the shape of output an automated daily check produces: a deterministic sweep + a classification of every flag (workload / dependency / product / infra / environment), each substantive finding independently re-derived, and the headline settled by a direct A/B rather than a guess. The natural automation is to wire that classification into the notification — auto-file an issue for the clear P0s, list the ambiguous ones with a prefilled create-issue link — with the deterministic results feeding an always-present, plotted section of a dated report, and a free-form section for commentary and root-cause.