Outage fix holding, no real regressions, one hash-rewrite action pending. Daily monitoring digest for the public Newton ASV benchmark fleet — for review, not a decision report.
Benchmark check 2026-07-24 (Dylan)
TL;DR: no perf regressions — both headline movers are expected (a deliberate KitchenG1 workload fix + ~95 new metric series landing). Outage fix holding: every machine reporting through 07-23 except GB10. One action pending: run the hash-rewrite, now across two boundaries (:exclamation:). 1 machine watch (:warning:), 2 perf watches (:eyes:).
:exclamation: hash-rewrite still pending — now across TWO boundaries: #3566 (a051c39a, 07-22, all refreshed suites) + #3575 (38c44beb, 07-23, drops custom solver-iteration overrides from bench_mujoco + teleop). No rewrite commit exists in newton-asv; the sweep's mismatch count only fell ~25→2 because benchmarks.json caught up as machines ran newer snapshots. Values are continuous at both boundaries (FastG1, FastAllegro, FastHumanoid, NewtonOverhead, KpiDRLegs) → recommend running rewrite-hashes to merge both, EXCLUDING FastKitchenG1.track_simulate (intentional version="2" split — add an exclusion comment in replace_hash.sh, like the tiled-camera one).
:warning: GB10 Spark (SC-PV-SPARK-PS-07) still silent 30 days — last data 2026-06-23. Predates the pip-check outage, so it's a separate issue; needs its own attention.
:white_check_mark: outage fix holding — Santhosh's #3582 recovery is sticking: SC-PV-02, SC-PV-10, orin, thor all have data through 07-23; L40S through 07-22.
:white_check_mark: FastKitchenG1.track_simulate +36–50% median (per-point +73–107%) on every machine is NOT a regression — #3566 bumped it to version="2"; the pre-v2 series accidentally omitted the kitchen scene, so the old numbers were simulating without it. The new values are the first correct ones. Exclude KitchenG1 from any history merge.
:white_check_mark: Jul-22 orin ContactPyramidDefaults transient resolved — back to 0.196 at 77b846e0 (within ~2.5% of the 0.191 baseline). Closing as a post-outage runner transient.
:information_source: Viktor + Tobias's ~95 new metric series have fully landed (#3566 + teleop #3269 + implicit MPM #3411 + NotifyDRLegs #3577): track_p95_step_time, track_real_time_factor, track_steady_state_gpu_memory, track_solver_niter_*, FastMetrics* classes, etc. All pre-labeled and start fresh — none are regressions; appearance/level is not a perf signal.
:eyes: CpuMuJoCoAnt slow creep on orin, +9% cumulative (two small steps, 3.55→3.87; latest 3.82); weak echo on thor (+6%) and SC-PV-02 py3.13. Not yet classifiable — still watching.
:eyes: FastNewtonOverheadHumanoid.track_simulate [8192] thor-only +8% (6.28→6.77, 4 points); SC-PV machines dead flat (13.3–13.5). Holding for more data.
Details:
Freshness (all UTC): SC-PV-02 and SC-PV-10 last data 07-23 (6de960ac py3.12 / 9e85e577 py3.13); orin 07-23 (77b846e0); thor 07-23 (ff9b71d5); L40S 07-22 (4a6e036f); GB10 stuck at 2026-06-23.
Hash boundaries: #3566 a051c39a (07-22) refreshed all suites; #3575 38c44beb (07-23) removed custom solver-iteration overrides from bench_mujoco + teleop, so mujoco/teleop hashes changed twice. Residual HASH MISMATCH is just 2 series / 1 env — FastNewtonOverhead{G1,Humanoid}.track_simulate on L40S, whose 4a6e036f run predates #3575; self-clears on L40S's next run.
Carry-over improvements (unchanged, still expected, not regressions): KpiDRLegs.track_simulate −5 to −16% across machines (warp 1.16 d4b8db4c / #3557 the prime suspect) and the #3415 "Batch ModelBuilder replication" model-init cluster, now confirmed for humanoid too (KpiInitializeModel humanoid −28% on L40S, echoed on thor / SC-PV-02 py3.13).
Standing pre-labels: warp still 1.16.0.dev20260716 (no further bump this cycle); new track_* series start fresh (level is not a perf signal); KitchenG1 v2 is a deliberate workload fix.
SC-PV-02, SC-PV-10, orin, thor all
have data through 07-23 snapshots; L40S through 07-22.
GB10 Spark still silent (30 days) — unchanged, a separate issue.FastKitchenG1.track_simulate is up +36–50% median (per-point +73–107%) on
every machine. The benchmark source explains it:
#3566 bumped it to
version = "2" with the comment “The pre-v2 series accidentally omitted the
kitchen environment.” The old series simulated without the kitchen scene; the new one
actually loads it. Deliberate, hash-split workload fix — do not read as a regression, and
exclude KitchenG1 from any history merge.track_p95_step_time,
track_real_time_factor, track_steady_state_gpu_memory,
track_solver_niter_*, the teleop suite
#3269, implicit MPM
#3411). All start fresh —
pre-labeled, none are regressions. As expected this added hash churn: mujoco/teleop
hashes changed twice (again at
#3575, 07-23).benchmarks.json caught up as machines ran newer snapshots. Old histories
remain split under old hashes. Values are continuous at both the #3566 and #3575 boundaries on
everything checked → recommend running rewrite-hashes to merge across both,
excluding KitchenG1. (See the action item below.)ContactPyramidDefaults
came back down (0.213→0.196, within ~2.5% of baseline) — closing as a
post-outage transient. CpuMuJoCoAnt on orin has instead firmed up as a slow creep
(+9% since early July, two small steps) — still watching. Warp is still
1.16.0.dev20260716; no further bump.| Machine | Last data | Status |
|---|---|---|
| SC-PV-02 | 07-23 (6de960ac py3.12, 9e85e577 py3.13) |
OK |
| SC-PV-10 | 07-23 (6de960ac py3.12, 9e85e577 py3.13) |
OK |
| jetson_agx_orin | 07-23 (77b846e0) |
OK |
| jetson_agx_thor | 07-23 (ff9b71d5) |
OK |
| adenzler-horde-L40S | 07-22 (4a6e036f) |
OK |
| SC-PV-SPARK-PS-07 (GB10) | 2026-06-23 | still silent 30 days — predates the pip-check outage; needs separate attention |
Hash churn from the metrics work has left ~25 benchmark series split at old-hash boundaries. No
rewrite commit exists in newton-asv; benchmarks.json merely caught up,
which is why the sweep’s HASH MISMATCH count fell to 2. There are now two
boundaries:
a051c39a, 07-22) — all refreshed suites.38c44beb, 07-23) — removes custom solver-iteration overrides from
bench_mujoco + teleop, so mujoco hashes changed again.Checked values are continuous across both boundaries (FastG1, FastAllegro, FastHumanoid, NewtonOverhead, KpiDRLegs), so the histories should merge cleanly.
FastKitchenG1.track_simulate, which is an intentional
version="2" split. Record that as an exclusion comment in
replace_hash.sh (same as the tiled-camera exclusion).
FastNewtonOverhead{G1,Humanoid}.track_simulate on L40S — its latest run
(4a6e036f) predates #3575. Self-clears on L40S’s next run; no action.track_p95_step_time / track_real_time_factor / track_sim_dt /
track_sim_substeps / track_steady_state_gpu_memory /
track_solver_niter_* and FastMetrics* classes across
anymal/kamino/mujoco/quadruped_xpbd (#3566, now on all machines);
bench_teleop_mujoco.* (#3269); bench_implicit_mpm (#3411,
cde70e27); NotifyDRLegs.*
(#3577).bench_mujoco.Fast*.time_simulate (removed by #3416, track history already merged) and
FastSensorTiledCamera.time_rendering_pixel_priority_* (documented exclusion, 07-16 /
recorded 07-18). No action.Six findings. The first three are expected / not regressions (a deliberate workload fix plus two carry-over improvements); two are genuine watch items; one is resolved.
FastKitchenG1.track_simulate [512] step up ~2×, all machines
SC-PV-02 0.00195→0.00338, SC-PV-10 0.00194→0.0034–0.0040, L40S 0.00222→0.00420, orin
0.0108→0.0223, thor 0.00718→0.0145; spreads ≤1.08, 2–4 post-step points each. Cross-machine bracket
(0b457032, a051c39a] — 5 commits — and the classification is definitive
from the source: #3566 set version="2" because the pre-v2 benchmark accidentally
omitted the kitchen environment. The new numbers are the first correct ones for this scene;
persisting correctly. Action: exclusion entry in replace_hash.sh; sweep flags will age
out.
KpiDRLegs.track_simulate improvement, persisting
Carry-over from 07-22, still at the improved level on all machines (SC-PV-02 py3.12 −16%, py3.13
−5%, SC-PV-10 −16%/−5%, thor −6%), now 4–5 post-step snapshots. Bracket
(e6ea232a, cbf32c55]; warp 1.16 bump d4b8db4c (#3557) is the
prime suspect, Kamino #3564/#3549 the alternates.
L40S pins KpiInitializeModel ['humanoid', 8192] 5.57→3.99 (−28%) at exactly
(26327db73, 3ecf0685] on 07-17 — the same #3415 “Batch
ModelBuilder replication” boundary reported for g1 on 07-22. Echoed with tight
spreads on thor (5.20→3.76) and SC-PV-02 py3.13 (3.58→2.45) across their outage gaps, plus
FastInitializeModel ['humanoid', 256] −26% and jetson
peakmem_initialize_model_cpu −7% (the memory part may instead trace to fewer
collision-filter pairs from 1d317fb7 #3574). Improvement; persisting.
CpuMuJoCoAnt slow creep on orin, +9% cumulative
Clean series shows two small steps: 3.55→3.71 at (647ebc0b, ffd116e7c]
(07-14) and 3.71→3.87 across the outage gap (07-15→07-20, which contains the warp 1.16 bump);
latest 3.82. Echoed weakly on thor (+6%) and SC-PV-02 py3.13 (+6%, oscillating). Small but
consistent direction on a CPU benchmark — keeping open, not yet classifiable
(orin-only bracketing is wide; other machines too noisy to intersect).
FastNewtonOverheadHumanoid.track_simulate [8192] thor-only +8%
6.28→6.77 over 4 recent points on thor; SC-PV machines dead flat (13.3–13.5). Noisy jetson track metric, one machine class — holding for more data.
FastExampleContactPyramidDefaults transient
The Jul-22 watch item (+9–11% on two post-outage points) dropped back to 0.196 at
77b846e0 (07-23), within ~2.5% of the 0.191 baseline. Reading the 07-20/21 elevation as
a post-outage runner transient; closing.
setup.bench_model.{Fast,Kpi}Initialize{Solver,Model,ViewerGL} py3.12 regression-side
flags up to 1.40× (SC-PV-02/10) — known bimodal series; SC-PV-02 py3.12 windows still contain the
degraded cbf32c55 cold-start run (e.g. 27.75 s KpiInitializeModel humanoid vs ~2.5 s
normal), which inflates n=5 medians.CpuMuJoCoAnt 1.24× SC-PV-02 py3.12 — same cold-start contamination; the genuine
residual signal is tracked as finding 4.SlowExample* compile-time flags 1.05–1.12× / 0.94× — single-shot compile
benchmarks, ±15% jitter, oscillating, directions disagree across machines/envs.CpuIKFranka 1.21× SC-PV-10 py3.12 — bimodal 0.26/0.31, recent window at high mode;
py3.13 flat.time_render_* 0.85–0.95× improvements everywhere — the 07-17 #3415
improvement (finding 3 family) aging through short baselines; no new movement.FastExampleContactSdfDefaults 0.64–0.86× — baseline artifact of the 07-09→07-14
resolved regression; current values at recovered level.FastAllegro.track_simulate 0.56–0.72× — aging step from ~07-14 plus 1.7–1.8×
spread; current values flat since 07-15 everywhere.FastHumanoid.track_simulate 0.93×, KpiViewerGL orin 0.90× (n=2),
FastDRLegs orin 0.94× — small, short windows, parts of already-classified clusters
or awaiting more points.