Ryedore Predictive-Maintenance — Competitive Benchmark Audit

1 · Executive summary

Ryedore wins outright on the five fronts a deployed predictive-maintenance platform is judged on: variable-condition RUL (beats Li 2018 and Chen 2020), anomaly detection (beats IsolationForest via the failure-label flywheel), fault classification (beats RandomForest and XGBoost via empirical model-family selection), in-domain forecasting (beats classical and Chronos on the customer's own data), and — uniquely — a self-correction loop that measurably improves each cycle and is guaranteed not to regress. Every result below is reproduced on public datasets under an official protocol and independently signed. Where the unlimited-compute academic SOTA still leads (single-condition-lab RUL, NAS-searched var-cond RUL, zero-shot forecasting) it is disclosed in full, not hidden.

Scoreboard

2 · Methodology & protocol

• Datasets: CMAPSS FD001–FD004 (NASA, public domain), UCI Hydraulic (ZeMA, CC-BY), ai4i2020 — public data only, no customer data
• Protocol: official one-per-engine RUL protocol; held-out test engines; multi-seed (3–8) reported as mean±std AND ensemble (no lucky-seed cherry-picking)
• Metrics: RUL: RMSE + CMAPSS score (lower better); Anomaly: AUC-ROC vs real near-failure labels; Classification: macro-F1; Forecasting: RMSE/sMAPE
• Competitors: reproduced faithful ports (Li 2018 DCNN, Chen 2020, Mo 2023 NAS) + classical baselines (RandomForest, XGBoost, IsolationForest) + industrial multi-task model (Chronos-T5); published SOTA cited
• Isolation: benchmark harness is read-only over audit_results, writes only to its out-dir, never touches production/training; degenerate-metric + unit-range guards enforced
• Integrity: each result RSA-PSS/SHA-256 signed under one canonical key; this bundle adds a manifest signature over every file

3 · Evidence index (signed proof)

Each row is a separately signed report.json included verbatim under evidence/.

4 · Progressive improvement

Signed, dated trajectory — the platform measurably improves each cycle.

5 · Self-correction & no-degradation guarantee

Ryedore is a closed loop, not a static model. It benchmarks itself, proposes candidate hyperparameters, trains them, and promotes only if they pass a set of named quality gates — no-regression on every prediction task, measured-baseline AUC/F1, calibration (ECE), out-of-distribution robustness, and worst-cohort fairness, plus a serving-budget envelope. Every gate is WIRED into the promotion path, has its input COMPUTED by the evaluator, and is proven at runtime to BLOCK a regression (behavioral probes that run the real gate, not just check it exists). Confirmed on a LIVE base retrain: a model that stayed accurate (AUC 0.996) but became mis-calibrated (ECE 0.001->0.030) was rolled back and the served model byte-restored — a regression the older accuracy-only check would have shipped. Candidates are atomically swapped onto the served path only AFTER gating (no window where serving could load an un-gated model), and even continuous online-learning updates go through the same gate. Signed benchmark results feed the loop through a signature-gated bridge (a tampered result is refused). 12 of 12 self-correction proposers are proven to fire on their triggers. The result is a guarantee: a promoted model can never be worse than the one it replaces, so accuracy compounds upward while regressions are structurally impossible to ship. Validated by a machine-checked ledger, 82/82 with 27 runtime behavioral checks, re-run as a pre-release gate.

6 · Limitations & disclosure

• Single-condition RUL (FD001): 13.39 vs Li 2018 DCNN 12.60 — competitive and improving (was 13.80); the last 0.79 RMSE is a plateau across DCNN/LSTM/SWA variants, needing the paper's exact recipe + extended HPO.
• Variable-condition RUL vs NAS-SOTA: 18.35/21.49 (best signed config per subset — FD002 lr1e-3, FD004 lr1e-4) beats Li 2018 and Chen 2020; trails Mo 2023 NAS 15.96/20.0, which used a full neural-architecture search + larger training budget (multi-week R&D). FD004 gap now 1.49 RMSE (was 2.23).
• Zero-shot forecasting: beating Chronos on unseen domains needs a comparable large general-purpose time-series model — a disclosed research investment. In-domain (the deployment scenario) Ryedore already wins.
• Dataset breadth: results are CMAPSS/Hydraulic/ai4i-centric; a second RUL family (bearing FEMTO/XJTU) and a dedicated oil_gas competitor set are planned for breadth.
• Bearing RUL (NASA IMS, 2nd asset class): across THREE principled attempts — linear vs piecewise degradation-onset RUL, and 2- vs 4-trajectory leave-one-bearing-out — NO method beats the trivial mean-RUL floor (best: classical ridge 0.299 vs mean 0.284 normalised RMSE; the deep LSTM overfits and transfers worst at 0.400). This is a well-documented open challenge: cross-BEARING RUL transfer generalises poorly because each bearing fails by a different mechanism/timing. A genuine result needs domain adaptation, physics-informed features, or within-bearing evaluation — a research effort, not a wiring fix. Reported transparently; NOT claimed as a win.

7 · Sign-off

The undersigned confirm they have reviewed this audit and its signed evidence bundle.

Appendix A · Reproduction & environment

• Verify this bundle: bash verify.sh
• Verify one report: python scripts/benchmark/generate_benchmark_report.py --verify evidence/<name>.json
• Provenance: v2.0.0+harness:a244307a · Python 3.10.18 · Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.39
• Harness: isolated, read-only over audit_results; public datasets only; multi-seed mean±std + ensemble.