87. Backfill status audit, 2026-05-19
Snapshot at 23:30 CEST. Audit ran against prod Neon DB.
Active backfills
| PID(s) | Script | Elapsed | Status | Decision |
|---|---|---|---|---|
| 28940 / 28967 / 28968 | scripts/backfill-sec-form4-history.ts |
3d 00h | Running, 20y long-haul | LET RUN |
| 51809 / 51854 / 51865 | scripts/backfill-sec-form4-historical.ts --start-year=2016 --end-year=2020 |
11h 07m | Running, at 2016-02-09 after 35644s, rows=33000 filings=48814 errors=0 | LET RUN |
| 77393 / 77426 / 77432 | scripts/_gtm-provision.ts --publish |
retry loop | Unrelated to backfills | LET RUN |
ETA SEC 2016 to 2020: at current pace (~2 trading days / hour after warmup), 5 years ~= ~25 days wall-clock. Acceptable per design.
Note: backfill-sec.log shows transient P1017 Prisma connection close + ECONNRESET, but loop keeps progressing (rows counter advancing). No restart needed.
Completed (looked "stale" but actually DONE)
| Log | Last line | Verdict |
|---|---|---|
backfill-adjclose.log |
done. updated=1375 skipped=2 anomalies=159 errors=0 |
DONE |
backfill-oslo.log |
done windows=13 seen=609 new=8 elapsed=7.4s |
DONE |
backfill-consob.log |
done pages=236 seen=5635 parsed=19 new=16 |
DONE |
backfill-cvm.log |
summary 2021 to 2025, all years done | DONE |
fast-rescore.log |
ALL DONE |
DONE |
merge-jp-watchdog.log |
`done | companies=608 insiders=292 declarations=2238` |
revive-jp-edinet.log |
[jp-backfill] DONE |
DONE |
revive-dart-kr2.log |
DONE chunks=23 totalNew=18332 totalSeen=19004 |
DONE (rate-limited at chunk 23 by DART API quota, acceptable) |
Stalled / dead
| Log | Last mtime | Issue |
|---|---|---|
rescore-v13.log |
19:58 (3.5h ago) | No "done" marker, 79 lines, no active PID. Likely killed manually or crashed silently. Not restarted, falls under future V13 wiring work. |
No restart performed — all non-SEC backfills completed cleanly.
XNAS unpriced gap analysis
Query: declarations on market=XNAS since 2023-01-01 where BacktestResult.return90d IS NULL.
| Year | Unpriced | Mature unpriced (pubDate < NOW - 90d) |
|---|---|---|
| 2023 | 9 | 9 |
| 2024 | 17 | 17 |
| 2025 | 264 | 264 |
| 2026 | 6145 | 157 |
| TOTAL | 6435 | 447 |
Verdict: 447 mature unpriced << 5000 threshold → NOT a structural gap. Bulk of the 6435 unpriced (95%) is 2026 immature (T+90 not yet reached, normal).
Cause breakdown of 447 mature unpriced
yahooSymbol IS NULL: 0yahooSymbol IS NOT NULL: 447
100% have a resolved Yahoo symbol. Gap is on the PriceHistory fetch side, not on symbol resolution. Likely candidates: delisted tickers, Yahoo 404 on those specific symbols, or rescore script never replayed against them after a price fetch.
Decision
No urgent restart. The 447 mature gaps will be cleared by the next regular rescore pass against XNAS once their price history is verified. Earlier "89k unpriced" claim was for global unpriced including immature + non-XNAS; the XNAS-specific mature gap is small.
Follow-up (low priority, separate ticket): add 447 declarationIds to a targeted price-fetch + rescore mini-batch to drop the gap to ~0.
Initial 89k claim reconciliation
Mission spec mentioned 89k XNAS unpriced. Actual measured: 6435 total unpriced (447 mature). Gap likely stems from an earlier audit before backfill-adjclose ran (which finished today at 19:53, updating 1375 symbols). Post-adjclose state is much healthier than expected.
Proof block
- Prod 200: not deployed (audit only, no code change to prod surface)
- CI green: see commit SHA pushed at end of session
- SHA match: TBD post-push