No newton regressions. Monday’s py3.13 watch item resolved to a py3.13-venv effect (not newton); one new Kamino improvement; the 07-28 infra gap has backfilled. Daily monitoring digest for the public Newton ASV benchmark fleet — for review, not a decision report. Covers the gap since Monday 07-27, incl. Tuesday’s overnight infra failure.
Benchmark check 2026-07-29 (Dylan) Covering the gap since my Monday post — includes Tuesday's overnight infra failure (expected data hole, not a regression), the py3.13 SC-PV watch-item from Monday resolved, and one new Kamino improvement. No newton regressions. :information_source: 07-28 overnight failed on the KitMaker/autokit publish race — warp-lang's 07-28 Linux wheel didn't get published in time, so the PLATO/asv runs on the Linux runners (5090 + 6000 Pro) had nothing to build against and errored out. Retried and resolved, data's backfilling. Net effect on the dashboard: SC-PV-02 is missing its 07-28 py3.13 point, that's the whole gap — both SC-PV runners are current through 07-29. So a 07-28 hole on those machines is this, not a regression. (job 30285206.) :eyes: → :white_check_mark: the py3.13-only SC-PV step from Monday — resolved, and it's the environment, not newton. Monday I flagged HeightfieldCollision.time_simulate, FastInverseDynamics.time_eval_inverse_dynamics_force and NotifyDRLegs.time_notify_shape_properties stepping up together on py3.13 only. With more runs it's clearly a py3.13 venv bimodality, not a one-way step: on SC-PV-02 the py3.13 env flips between a fast and a slow mode run-to-run — 07-27 caught both on the same day (de2d06f3d fast, d124e5bd slow), with heightfield and inverse-dynamics moving together — while py3.12 is dead flat on the exact same newton commits. SC-PV-10 has just been parked in the slow mode since 07-24. Same newton commit produces both modes => it's the py3.13 virtualenv, full stop. It did NOT revert (the slow mode is real and recurs), so this is worth someone with SC-PV runner access diffing the py3.13 venv on those runners — the recommendation from Monday, now firmed up. Not a newton regression, keeping it out of #newton-dev. :white_check_mark: the third one, shape_properties, got overtaken by a genuine newton win. #3677 [Kamino] Skip unused material updates dropped NotifyDRLegs.time_notify_shape_properties by ~half on every machine and both py envs at the 07-27 boundary (SC-PV-02 py3.12 1.0e-4->3.0e-5, SC-PV-10 py3.13 2.4e-4->6.2e-5, orin 4.3e-4->2.2e-4, thor 2.1e-4->1.0e-4). It touches solver_kamino.py + the notify test, so it's the real thing. The py3.13-vs-py3.12 gap on this metric is still ~2x but the absolute level is now low. Monday's deferred boundary step is superseded by this. :white_check_mark: model_properties #3606 improvement (the 4-8x from Monday) still stable, no drift. :warning: machine health — orin's back (was quiet since 07-23, now reporting 07-27 + 07-28), thor current 07-28, both SC-PV current 07-29. L40S is the one to watch: still stuck at 07-22, now 6 days silent (was 4 on Monday) — @Adenzler can you give the L40S runner a poke? GB10 Spark still silent 35 days, same standing issue. :information_source: carry-overs, no action: CpuMuJoCoAnt orin creep has plateaued at +8% (3.82->3.84, not climbing now that orin's reporting again); FastExampleCablePile's +8% step at the 07-23 boundary is holding (cross-env, newton-side, low magnitude); KitchenG1 is still the top sweep flag but that's the version="2" workload fix (excluded in replace_hash.sh, not a regression). Hash-rewrite still holding at 0 mismatches. @Viktor @Tobias — heads up: the new teleop track metrics (track_object_displacement_m, track_hand_object_contact_frame_pct, track_loop_time_cv_pct) are dominating the top of the sweep with 5x+ ratios, but they're behavioral/quality metrics with huge run-to-run spread (x27-x71), not timings — just noise in the regression sweep, ignore them there.
HeightfieldCollision.time_simulate and
FastInverseDynamics.time_eval_inverse_dynamics_force are a
py3.13 venv bimodality, not a step: on SC-PV-02 the py3.13 env flips fast/slow
run-to-run (07-27 caught both modes the same day — de2d06f3d fast,
d124e5bd slow — with the two benchmarks moving together), while py3.12 is dead flat on
the identical newton commits. SC-PV-10 has sat in the slow mode since 07-24. Same newton
commit yields both modes ⇒ the py3.13 virtualenv. It did not revert — the slow
mode recurs — so the venv should be diffed.bfe8c313, “[Kamino] Skip unused material updates”) drops
NotifyDRLegs.time_notify_shape_properties ~2× on every machine and both py
envs at the 07-27 boundary. It touches solver_kamino.py + the notify test —
a genuine win. The py3.13-vs-py3.12 gap persists (~2×) but at a much lower absolute level; Monday’s
deferred boundary step is superseded.orin is
back (quiet since 07-23 on Monday, now reporting 07-27/07-28); thor current 07-28; both
SC-PV current 07-29. L40S is still stuck at 07-22 — now 6 days silent
(was 4 on Monday). GB10 Spark still silent 35 days (standing).model_properties (#3606, 4–8×) still stable; CpuMuJoCoAnt orin creep
plateaued at +8% (3.82→3.84, no longer climbing); FastExampleCablePile
+8% step holding; KitchenG1 v2 still the top sweep flag (excluded workload fix, not a regression).
Sweep HASH MISMATCH count is 0. Warp still 1.16.0.dev20260716.| Machine | Last data | Status |
|---|---|---|
| SC-PV-02 | 07-29 (0a3e9e1b py3.13); py3.12 last 07-28 (0827c76e) |
OK — no 07-28 py3.13 point (KitMaker race; backfilled) |
| SC-PV-10 | 07-29 (0a3e9e1b py3.13); py3.12 last 07-28 (0827c76e) |
OK |
| jetson_agx_orin | 07-28 (825f0c98) |
OK — recovered (was quiet since 07-23 on Monday) |
| jetson_agx_thor | 07-28 (92ae9454) |
OK |
| adenzler-horde-L40S | 07-22 (4a6e036f) |
silent 6 days — escalating (was 4 days Monday); needs a poke |
| SC-PV-SPARK-PS-07 (GB10) | 2026-06-23 | still silent 35 days — standing issue, predates everything, separate |
The overnight PLATO/asv runs for the Linux runners (5090 = SC-PV-10, 6000 Pro = SC-PV-02) failed on 07-28: an autokit publishing race left warp-lang’s 07-28 Linux wheel unpublished, so the environments had nothing to build against. This is an infra failure, not a benchmark signal — resolved on retry, and the data has backfilled.
Observable residual on the dashboard: SC-PV-02 has no 07-28 py3.13 snapshot (it
goes 07-27 d124e5bd → 07-29 0a3e9e1b); its py3.12 run on 07-28
(0827c76e) did land, and SC-PV-10 got both envs. Both SC-PV runners are current through
07-29.
Monday’s three-benchmark py3.13 step is now clearly a bimodal py3.13 virtualenv, not a one-way regression and not a clean revert. The two coupled benchmarks flip between a fast and a slow mode:
HeightfieldCollision.time_simulate — py3.13 fast ~0.084 / slow ~0.105. SC-PV-02 has
run both: 07-27 de2d06f3d = 0.0838 (fast), 07-27 d124e5bd = 0.105
(slow), 07-29 = 0.106 (slow). SC-PV-10 parked in the slow mode (~0.103) since 07-24. py3.12 dead
flat at 0.067 throughout.FastInverseDynamics.time_eval_inverse_dynamics_force — py3.13 fast ~0.412 / slow
~0.537, moving in lockstep with heightfield (07-27 de2d06f3d = 0.413 fast,
d124e5bd = 0.542 slow). py3.12 dead flat at 0.329.Confirmed not a newton regression. The clincher: on SC-PV-02, two snapshots the same day (07-27) land in opposite modes on both benchmarks together, and py3.12 never moves on the identical newton commits. Mode is set by env state at run time, not by any newton commit. asv does not record pinned deps in these result files, so the specific package can’t be named from the data.
#newton-dev.
0bf74a7a rewrite
(#3566 /
#3575 /
#3409) is still clean; today’s
sweep reports no HASH MISMATCH lines. No new hash boundaries since Monday.track_p95_step_time / track_real_time_factor /
track_steady_state_gpu_memory / track_solver_niter_* /
FastMetrics* / teleop / NotifyDRLegs.* series are still showing as
“appeared” on one more machine/env at a time as machines run newer snapshots — baseline
windows filling. Appearance/level is not a perf signal.FastSensorTiledCamera.time_rendering_pixel_priority_* — the documented tiled-camera
exclusion (#3480). No action.Six findings. One is a resolved-to-env watch (the py3.13 venv bimodality); one is a new confirmed improvement; one is a confirmed-stable improvement; two are carry-over watches; one is a known excluded workload fix. No genuine regressions.
See Needs a look above. HeightfieldCollision.time_simulate (fast ~0.084 /
slow ~0.105) and FastInverseDynamics.time_eval_inverse_dynamics_force (fast ~0.412 /
slow ~0.537) flip together on py3.13; SC-PV-02 ran both modes on 07-27, py3.12 dead flat on the
identical newton commits. Env, not newton; did not revert — recommend a py3.13 venv
diff. The finer-grained NotifyDRLegs.time_notify_{body,joint,actuator}_* micro-benchmarks
also read a touch high on SC-PV-10 py3.13 (all e-05 scale, noisy), consistent with the same py3.13
env overhead.
NotifyDRLegs.time_notify_shape_properties ~2× faster (#3677)
New since Monday. Steps down ~2× at the 07-27 boundary on every machine and both py envs:
SC-PV-02 py3.12 1.0e-4→3.0e-5, SC-PV-10 py3.12 9.8e-5→2.8e-5 / py3.13 2.4e-4→6.2e-5, orin
4.3e-4→2.2e-4, thor 2.1e-4→1.0e-4. Cross-machine bracket intersects on
#3677
(bfe8c313, “[Kamino] Skip unused material updates”) — which touches
solver_kamino.py and test_solver_kamino_notify.py, exactly the notify path.
Because it hits both py envs it is newton-side, not the py3.13 env issue. This supersedes the
shape_properties boundary step deferred on Friday; new lower baseline (py3.13 still ~2× py3.12, the
same env overhead, but at a low absolute level).
NotifyDRLegs.time_notify_model_properties ~4–8× — stable
The #3606 improvement characterized Monday is holding flat: SC-PV-02 py3.13 ~6.1e-6, SC-PV-10 py3.13 ~5.3e-6, orin ~1.7e-5, thor ~9.0e-6 — no drift since the 07-22→07-23 drop. Confirmed stable, no re-analysis needed.
CpuMuJoCoAnt orin creep — plateaued at +8%
orin resumed reporting (07-27/07-28) and the value is holding, not climbing: 3.55 (07-13) → 3.71 (07-14) → 3.87 (07-20) → 3.82 (07-23) → 3.82 (07-27) → 3.84 (07-28). ~+8% cumulative, now a stable plateau rather than a creep. Still one CPU benchmark, still not classifiable to a commit — keeping open but no new movement.
FastExampleCablePile.time_simulate +8% step — holding
The 07-22→07-23 boundary step is holding: orin 0.2965 → 0.3211 (07-23) → 0.3204 (07-28), plus
+5–6% on SC-PV py3.12/py3.13. Cross-env, so newton-side, low magnitude;
holding at the stepped level, nothing new. Bracket unchanged from Monday
((a051c39a, 6de960ac]).
FastKitchenG1.track_simulate [512] +2×, all machines
Still the top sweep flag (orin 2.07×, SC-PV-02 1.72×, SC-PV-10 1.30×), still the deliberate
version="2" workload fix — the pre-v2 series omitted the kitchen scene, so the new
numbers are the first correct ones. Hard-excluded from any history merge; recorded in
replace_hash.sh. Not a regression.
TeleopMuJoCo.track_object_displacement_m (5.33× SC-PV-10, spread x27.53),
track_hand_object_contact_frame_pct (1.73×, x2.00),
track_loop_time_cv_pct (1.95×, x7.51). These are behavioral / quality metrics from the
teleop suite (#3269) with huge
run-to-run spread, not timings — the sweep’s ratio just reads their scatter.setup.bench_model.{Fast,Kpi}InitializeSolver/Model flags up to 1.63× (SC-PV py3.13) —
known bimodal init series (spreads x1.5–x16), OSCILLATING/NOISY, directions disagree across
params; the mirror-image 0.72–0.90× “improvement” flags are the same series aging
through short baselines.RealtimeHumanoidPhysics.track_step_time_cv_pct / track_step_rate_hz / track_real_time_factor
1.40× / 0.85–0.87× — new realtime metrics, oscillating, high spread; level/scatter, not a
timing step.NotifyDRLegs.time_notify_{body,joint,actuator}_* 1.38–1.57× SC-PV-10 py3.13 —
e-05 scale micro-benchmarks, spreads x1.5–x1.7; folded into the py3.13 env note (finding 1).SlowExample* compile-load flags 0.93–0.95× — single-shot compile benchmarks,
±15% jitter, oscillating.CpuMuJoCoAnt 0.91–0.94× SC-PV py3.12 — cold-start contamination (spread
x15.41); the genuine residual is the orin plateau (finding 4).KpiDRLegs.track_real_time_factor / track_simulation_steps_per_second 0.89–0.90×
— higher-is-better KPIs; the sweep reads the reciprocal, these track the stable Kamino series, no
real move.time_render_* 0.87–0.95× — the #3415 improvement family aging
through short baselines; no new movement.