133 · Strategy / Scoring / Backtest Integrity Audit (2026-06-04)
Reviewer: independent read-only quant/integrity audit
Scope: src/lib/winning-strategy.ts (WINNING_STRATEGY, STRATEGY_PROOF), src/lib/signals.ts (computeV13Score / V14e, computeScore / V3), src/lib/backtest-compute.ts, src/lib/quant-stats.ts, src/lib/backtest-methodology.ts, src/lib/strategy-proof.ts, the weekly/OOS crons, and the public surfaces that render these numbers (landing, /performance, /backtest, MCP, REST).
Method: code read + live read-only DB queries against DATABASE_URL_DIRECT (Neon prod), 2026-06-04. No code, data, or scoring changed.
TL;DR: The honesty machinery is unusually good (three-Sharpe disclosure, DSR, Wilson/bootstrap CIs, PIT policy, winsorization helpers, an explicit "fake sparkline" label). But several headline numbers shown to users are stale, non-reproducible, or contradicted by the current DB. Two are serious: the relatedInsider 85.4%/+268% claim is inverted in the live data (and it drives a live scoring bonus), and the headline OOS Sharpe 1.36 / CAGR 38.6% does not reproduce from a straightforward reconstruction (I get ≈0.55 / ≈+23%). Plus the whole STRATEGY_PROOF "universe" block is ~85% stale on sample size.
This audit changes nothing. Any scoring fix is gated by the AGENTS.md ship-gate.
A. INTEGRITY VERDICT: per displayed claim
Legend: REPRODUCED (DB matches) · STALE (was right, DB has moved) · UNVERIFIABLE (no reproducible generator in-repo) · FABRICATED/CONTRADICTED (DB disagrees materially, sometimes sign-inverted).
| # | Displayed claim | Source (file:line) | DB / reproduction (2026-06-04) | Verdict |
|---|---|---|---|---|
| 1 | Win rate 77% (landing headline, badges, MCP, REST) | winning-strategy.ts:961 (winRate: 77) |
Hand-entered for the n=173 "retail-realistic" subset. No in-repo generator; the n=173 cohort cannot be reconstructed from the DB (the filter that produced it isn't materialized). | UNVERIFIABLE |
| 2 | avgReturn 13.2%/yr | winning-strategy.ts:952 |
Same n=173 subset, hand-entered. Not reproducible. | UNVERIFIABLE |
| 3 | 3/4 winning years, 2025 = +48.3%, 2024 = −3.7%, 2023 = +16.0%, 2022 = −7.9% | winning-strategy.ts:878-883 (_proofYears) |
Hardcoded table, comment says "REFRESHED 2026-05-14 against actual backtest universe (n=173)". No script in scripts/ regenerates it. The /performance page computes its own yearly strategies live (performance-data.ts::runStrategy with a real CAC benchmark), but the landing track-record table is a frozen, hand-maintained array decoupled from that. |
UNVERIFIABLE (and structurally drift-prone) |
| 4 | Sharpe 1.87* (cross-sectional) | winning-strategy.ts:888 |
Frozen legacy figure on n=173 trade-level dispersion. Honestly qualified in UI (config/strategy-proof.ts::sharpeDisclosure, asterisk + footnote + "NEVER cite alone"). Not reproducible but correctly framed as not-a-portfolio-Sharpe. |
UNVERIFIABLE but honestly disclosed |
| 5 | sharpeDeflated = null + DSR-negative note | winning-strategy.ts:899-907 |
computeDeflatedSharpe(1.87, 583200, 4) → penalty √(2·ln(583200)/4) ≈ 2.577 → negative → null. Recomputed by hand: matches. Honest. |
REPRODUCED |
| 6 | liveOosUniverse: n=20,688 · WR 52.96% · mean T+90 +8.59% | winning-strategy.ts:1008-1014 (the in-code fallback seed) |
The live cron query is correct and current: latest StrategyProofSnapshot row (2026-06-01) = n=38,384 · WR 52.47% · mean 6.86%; my live re-run of the exact cron SQL today = n=39,464 · WR 52.75% · mean 6.88%. The hardcoded fallback (20,688 / 8.59) is ~48% stale on n and ~25% high on the mean. Risk: any lambda where the DB read fails renders 20,688/8.59. |
STALE fallback (live path OK) |
| 7 | crossMarketUniverse / universeSize: n=20,688 · WR 46.8% · mean T+90 +0.78% · cross-sectional Sharpe 0.027 | winning-strategy.ts:990-1002 |
Live full BUY priced universe = n=253,447 · WR 42.16% · RAW mean +14.48%. Sample size is ~12× larger than displayed. The "universe=20,688" shown in the public headline eyebrow (strategyHeadline.sampleLong) is wrong by an order of magnitude. |
STALE / CONTRADICTED |
| 8 | srMarkets.FR: n=16,101 · WR 44.3% · mean −0.25% (and all 6 per-market rows) | winning-strategy.ts:1086-1092 |
Query by Company.market='AMF' returns n=0 , the DB stores the MIC code XPAR, not AMF. The whole srMarkets map keys on regulator codes that no longer exist in the market column. Frozen 2026-05-17 snapshot; not reproducible against current schema. |
STALE / UNVERIFIABLE |
| 9 | relatedInsider: related n=111 · WR 85.4% · +268% 1Y vs direct n=4,075 · WR 46.3% · +227% 1Y (drives the V12 +5pt bonus AND the V14e +0.20 family multiplier) | winning-strategy.ts:1119-1124; bonus at signals.ts:356-362; multiplier at signals.ts:1101-1107 |
Reproduced on the same 2023-2024 BUY window: related (true) n=19,846 · WR 49.5% · avg 1Y +1.1% · median 0%; direct (false) n=26,954 · WR 54.3% · avg 1Y +41.4%. The cohort grew ~180× and the effect inverted: related now underperforms direct (49.5% vs 54.3% WR; +1.1% vs +41.4% 1Y). Claimed +39pt related edge is not present and sign-flipped in the live data. Max related 1Y is +455% (the +268% "average" was clearly outlier-driven on the tiny old n). | CONTRADICTED (critical , and it's wired into the live score) |
| 10 | OOS: Sharpe 1.36 · DSR 0.43 · CAGR +38.6% · MaxDD −10.6% · WR 55.0% · n_OOS 12,384 (performance hero, methodology Ch06, MCP, landing Sigma) | winning-strategy.ts:1042-1065 (oosResults); audit 103 |
The universe row reproduces exactly: EU_strict (XPAR/XAMS/XWBO/XBRU/XHEL/XOSL/XSTO/XETR), pubDate≥2025-01-01, n=12,384 ✓ (raw mean +2.66%, winsor +2.00%, WR 50.2%). But my approximate reconstruction of the top-10/month V14e picker (rank by signalScoreV13, T+90, winsor ±50%, 14 months) yields Sharpe ≈ 0.55, CAGR ≈ +23.4%, σ_monthly 9.48% , vs claimed Sharpe 1.36, CAGR 38.6%, σ 6.41%. Claim is ~2.5× the Sharpe and ~1.6× the CAGR of a straightforward reconstruction; the gap is mostly an implausibly low claimed volatility (6.41% vs realized 9.48%). My reconstruction is approximate (rebalance timing, exact hold accounting, weighting, the V13→V3 COALESCE fallback), so this is a strong flag, not a precise refutation. |
NOT REPRODUCED (materially optimistic) , re-run the bake with the audit-103 harness before continuing to display |
| 11 | Landing 24-month win-rate sparkline | LandingTrackRecord.tsx:27 |
22 hardcoded points (seed = [0.58, 0.62, …]) + 1 live point. Comment literally says "fake historical win rate". Labelled "illustratif/illustrative" in section copy (prior audit HARDCODED-AUDIT-2026-05-20 noted this). |
FABRICATED but disclosed (recommend removing or replacing with real data , a "fake" curve on the landing track-record section is a credibility liability even when labelled) |
| 12 | OOS live tracker (RecoSnapshot) | oos-tracker.ts, /api/cron/oos-snapshot |
RecoSnapshot table has 160 rows, 2026-05-27 → 2026-06-03 (7 days). Genuinely live and forward-looking, but far too young to have matured T+90 returns , it cannot yet corroborate any forward claim. |
REPRODUCED (but immature) |
Integrity bottom line
- The architecture is honest by design (three-Sharpe split, DSR null, CIs, PIT, winsorize helpers, asterisk tooltips, explicit "fake" labels, a working live-OOS cron). Whoever built
STRATEGY_PROOF's disclosure block understood the failure modes. - But the frozen constants have drifted hard and at least one (#9 relatedInsider) is now contradicted and sign-inverted in production data while still feeding the live score. The headline OOS Sharpe/CAGR (#10) does not reproduce and looks optimistic.
- A prior internal audit (
HARDCODED-AUDIT-2026-05-20.md) already caught one fabricated copy claim ("67% at score≥65") and the fake sparkline. This audit extends that: the data-layer constants (universe size, per-market table, relatedInsider, OOS headline) are the next liability, and they're higher-stakes because they're the proof, not the prose.
B. METHODOLOGY FINDINGS
B1. Outliers dominate every RAW mean (CRITICAL, still open from audit 28/C3)
returnFromPub90d on the full priced BUY universe: min −99.9%, max +908,990%, median −2.71%, p99 +73.9%, 43 rows >+1000%. RAW mean +14.48% but winsorized (0.5/99.5) mean −2.24% and median −2.71%. So the unfiltered universe is negative once de-outliered.
backtest-compute.tsdoes winsorize (0.5/99.5) before mean/Sharpe →/backtestpage is robust. Good.- The
liveOosUniversecron andcrossMarketUniversedo NOT winsorize → the published "mean T+90" is the inflated raw figure. Live OOS: raw +6.88% vs winsorized +3.23% vs median +1.19%. The displayed mean is ~2× the honest central tendency. - Recommendation: winsorize (or publish median + winsor-mean) in the cron query and the frozen universe block. The win-rate is outlier-immune and fine to keep.
B2. "Universe mean" is the wrong honest statistic
Given the fat right tail, median and win-rate are the defensible central stats; the mean is not. Publishing a single mean (raw or winsor) without the median invites exactly the over/under-statement seen in #6/#7. Recommend always pairing mean with median + n on every universe surface.
B3. Look-ahead / entry-timing: LOOKS CLEAN
Entry is the close after pubDate (returnFromPub*, "retail-view"), not the insider transaction date , the honest, no-look-ahead choice. pubLeakPct median ≈ 0, mean +1.34% confirms only mild pre-publication drift. The track-record feature (priorAlphaForInsider) filters strictly to pubDate < currentPubDate (look-ahead-free) with k=5 Bayesian shrinkage. PIT policy (allowsLiveMarketSnapshot) gates mcap/Yahoo/analyst bonuses to a live window so historical rows don't get future fundamentals. This is the strongest part of the methodology.
B4. Survivorship bias: disclosed, not corrected
Delisted 2015-2026 names are absent from Company, so their trades never enter the universe (disclosed at winning-strategy.ts:1018). Real upward bias, honestly flagged, structurally hard to fix without a delisted-securities feed. Acceptable as a disclosed caveat; do not let copy imply it's eliminated.
B5. Multiple-testing / DSR: applied, but unevenly
DSR is genuinely applied (583,200-trial penalty drives the public Sharpe to null , commendable). But the production V14e headline (1.36) uses DSR with N=18 trials (disclosure.multipleTestingRisk), which undercounts: the true search included 583,200 filter combos + a 314-weight grid + a 28-variant bake-off + the EU-vs-full + the family/entity multiplier sweep. Using N=18 for the deflation of the shipped number is generous. The OOS window is 14 monthly buckets , DSR with T=14 and an honest N (hundreds to thousands) would bite much harder than the displayed +0.43.
B6. OOS window genuinely held out: but short and singular
Test window 2025-01-01 → 2026-02-27 (max pubDate), T+90 matured (today 2026-06-04). It is out-of-sample vs the 2021-2024 discovery. But it's one 14-month window, one regime. The disclosure.multipleTestingRisk says this plainly ("not equivalent to repeated forward validation"). Keep that caveat adjacent to the 1.36 number everywhere it appears.
B7. EU_strict restriction: defensible, but it IS a universe search
Audit 101 (XNAS inverts signs) makes the EU-only filter intellectually honest if US-tape inversion was discovered, not fitted. But note: XNAS is 131,439 of 253,447 priced BUY rows (52%) , excluding the majority of your data to rescue the Sharpe is a large degree of freedom. The claim "honest-tape filter, not a parameter search" is partly true and partly a post-hoc universe selection. Treat it as a tested hypothesis, not settled fact, until a second OOS window confirms.
B8. Win-rate baseline honesty
Filtered subset 77% vs universe 42-53% is a real, large gap. Win-rate is the most trustworthy headline you have (count-based, outlier-immune). The 77% itself is on the unverifiable n=173 subset (#1), so re-derive it on the current DB with the materialized Sigma filter before continuing to lead with it.
B9. Two scoring formulas coexist; the displayed budget is the legacy one
computeScore (V3, the 100-pt budget shown on /methodologie) and computeV13Score (V14e, the actual production ranker) are different models. The pretty 100-pt weight table (SCORE_V3_WEIGHTS) is not what ranks production recommendations (V14e is a continuous score with EU gate + multipliers). Users shown the V3 budget may think that's the live formula. Not dishonest, but a transparency gap worth a one-line note.
C. SIGNAL-QUALITY ASSESSMENT (which components carry weight)
From the live DB cross-checks and the audit trail. "Tests" = what to run before trusting; none applied here.
| Component (V14e / V3) | Live evidence | Verdict | Recommended test |
|---|---|---|---|
Directional cluster (clusterStrengthScore, V13 +2, wide-cluster ×1.30) |
Repeatedly validated as alpha source across audits; directional fix removed mixed-activity inflation | Real signal | Re-confirm wide-cluster ×1.30 (≥5 insiders) lift on EU_strict OOS; small-n risk |
Senior role CEO/CFO (functionScore ×1.3/1.2; V13 +2.5) |
Audit 53/56 OOS bake gave the tilt; sign-consistent on EU | Real on EU, INVERTS on XNAS (audit 101) | Per-tape role-sign test before any market re-inclusion |
% market cap (pctMcapScore, V13 ×1.4 capped 5) |
Conviction proxy; conceptually sound | Likely real | Decile-monotonicity test on winsorized returns |
Track record shrinkage (trackRecordScore, k=5, prior-only) |
Look-ahead-free, well-built | Plausible, low risk | IC of shrunk alpha vs forward 90d; watch thin per-insider n |
| relatedInsider +5 (V12) AND family multiplier +0.20 (V14e) | DB CONTRADICTS the founding stat (#9): related now 49.5% WR vs direct 54.3%, +1.1% vs +41.4% 1Y | Likely NOISE / negative now , leaking a stale, inverted prior into the live score | Re-bake related & family/entity multipliers on full current cohort; expect to zero or flip them. Ship-gated. |
| Earnings proximity ×, sector momentum × (V13.2/V13.3) | 17.5% coverage; +0.13 / +0.61 same-run lift claimed (docs 78/80) | Unconfirmed, thin coverage | Leave-one-month-out already done; re-run on current cohort, check coverage hasn't collapsed |
| Analyst-contrarian, DCA, conviction (cumNet>0), composite Yahoo, fundamentals | Small fixed weights; mostly public info | Low / marginal | Ablation: drop each, measure ΔSharpe on EU_strict OOS , candidates to prune |
| CMP-2012 pattern bonus (−8..+12) | FR data already contradicted US finding (WINNING_STRATEGY comment: routine outperforms here); shipped opt-out |
Suspect / region-dependent | Re-estimate sign on EU_strict; the comment already admits the US prior doesn't hold |
Dead/leaking/redundant flags: the relatedInsider/family path is the clear leak (stale inverted prior, live in the score). The V3 composite Yahoo sub-bonuses and fundamentals are the most likely noise/redundant (small weight, public info, already PIT-gated off for most historical rows).
D. IMPROVEMENT HYPOTHESES (ranked · expected lift × feasibility · NOT applied)
Each is a hypothesis to backtest later. Bold = needs the AGENTS.md ship-gate (it changes scoring/universe/filter/sizing).
- Re-estimate (or kill) the relatedInsider +5 and family +0.20 multipliers. [high lift-risk × high feasibility] The founding stat is inverted in current data (#9). Removing a negative/noise signal should raise OOS quality immediately. Ship-gated. Likely the single highest-value change.
- Winsorize + publish median on every universe surface (cron + frozen block). [high integrity × trivial feasibility] No scoring change, no ship-gate , it's a display/measurement fix. Removes the #6/#7 overstatement. Do first.
- Regenerate
STRATEGY_PROOFconstants from a committed script and wire freshness. [high integrity × medium]_proofYears,universeSize,srMarkets,relatedInsiderare all hand-frozen and 2-180× off. Buildscripts/_regen-strategy-proof.ts, keysrMarketson MIC codes, and have the page read the live snapshot (already exists forliveOosUniverse) instead of constants. Not a scoring change. - Re-run the V14e OOS bake with the audit-103 harness and reconcile #10. [high credibility × medium] My reconstruction (≈0.55/≈23%) vs claimed (1.36/38.6%) must be reconciled before continuing to display the headline. If the harness can't reproduce 1.36 on today's DB, restate the number. Effectively ship-gate-adjacent (it's the headline proof).
- Honest DSR with the true trial count. [medium × easy] Recompute the V14e deflation with N reflecting the full search (not 18). Likely pushes DSR toward/under 0 , display it anyway; it's more defensible than an optimistic +0.43.
- Second OOS window (walk-forward, ≥2 disjoint regimes). [high robustness × medium] One 14-month window isn't repeated forward validation (B6). Add a rolling-origin evaluation. Ship-gated if it changes the published metric.
- Ablation sweep on EU_strict OOS to prune low-weight V3/composite features. [medium × medium] Drop fundamentals, Yahoo composite sub-bonuses, CMP pattern one at a time; keep only ΔSharpe-positive ones. Ship-gated.
- Per-tape sign validation before re-including any non-EU market. [medium lift × medium] XNAS (52% of data) is excluded; a validated US-tape sub-model (audit 101 Filter A/B) could recover breadth. High value but high risk. Ship-gated, do not rush.
- Replace the fake landing sparkline (#11) with the real
RecoSnapshotforward curve once it matures (~90d). [credibility × low] Removes a "fake" data artifact from the most-viewed page. - Liquidity/turnover realism on the picker. [low-medium × medium] EU small-caps are conviction-rich but thin; top-10/month at size may not be executable. Add an ADV cap to the backtest. Ship-gated (changes reported returns).
E. WHAT I COULD NOT VERIFY (stated plainly)
- The n=173 / n=196 Sigma subset (#1, #2, #4): no in-repo generator materializes it; could not reconstruct the exact filter cohort, so 77% / 13.2% / 1.87 are UNVERIFIED, not disproven.
- The
_proofYearsyearly table (#3): no generator;/performancecomputes different live yearly numbers, so the landing table is UNVERIFIED. - The exact OOS Sharpe 1.36 (#10): my picker reconstruction is approximate (rebalance timing, weighting, hold accounting, V13/V3 fallback). I can say it does not obviously reproduce and looks ~2.5× optimistic, not that 1.36 is definitively wrong. Re-run the canonical bake to settle it.
Appendix · live DB figures (read-only, 2026-06-04, DATABASE_URL_DIRECT)
BacktestResult by direction: BUY 345,148 (253,447 w/ 90d) · SELL 96,716 · OTHER 5,748
Full BUY priced universe: n=253,447 · WR 42.16% · RAW mean +14.48% · WINSOR mean −2.24% · median −2.71%
min −99.9% · max +908,990% · p99 +73.9% · 43 rows >+1000%
Live OOS (pubDate≥2025): n=39,464 · WR 52.75% · RAW mean +6.88% · WINSOR +3.23% · median +1.19%
(cron StrategyProofSnapshot 2026-06-01: n=38,384 · WR 52.47% · mean 6.86%)
EU_strict OOS (8 MICs, ≥2025): n=12,384 ✓ · WR 50.19% · RAW mean +2.66% · WINSOR +2.00% · median +0.10%
EU_strict top-10/mo picker (approx, winsor ±50, 14mo): Sharpe ≈0.55 · CAGR ≈+23.4% · σ_m 9.48% · mean_m +1.77%
relatedInsider 2023-2024 BUY 1Y:
related(true) n=19,846 · WR 49.47% · avg +1.09% · median 0% (claimed: n=111 · WR 85.4% · +268%)
direct(false) n=26,954 · WR 54.34% · avg +41.39% · median +3.29% (claimed: n=4,075 · WR 46.3% · +227%)
FR market by 'AMF': n=0 (DB uses MIC 'XPAR' → srMarkets keys are stale)
pubLeakPct: mean +1.34% · median ≈0 (entry-timing clean)
Priced BUY by market (top): XNAS 131,439 · BVMF 42,513 · XBOM 18,856 · XPAR 16,069 · XTKS 8,919 · …
Score coverage (DIRIGEANTS, pdfParsed): total 751,229 · signalScore 751,187 · signalScoreV13 751,182
RecoSnapshot live tracker: 160 rows · 2026-05-27 → 2026-06-03 (immature)
No code, data, or scoring was modified. Temporary verification scripts were deleted after use. Reproduction queries are inlined above so any reviewer can re-run them read-only.