Session Report — 17-Market SEC Merge & Enrichment Pass
Date: 2026-05-17
Wall-clock window: ~3h autonomous run
Branch / final SHA: main @ f632a2d (deployed)
1. Database state (post-merge)
| Entity |
Rows |
| Declarations |
162,615 |
| Companies |
8,591 |
| Insiders |
58,004 |
| Markets |
17 |
Declarations per market
| Market |
Decls |
| SEC (US) |
126,607 |
| AMF (FR) |
25,733 |
| AFM (NL) |
6,392 |
| FSMA (BE) |
906 |
| OSLO (NO) |
601 |
| HEL (FI) |
600 |
| CONSOB (IT) |
360 |
| BAFIN (DE) |
295 |
| SIX (CH) |
226 |
| ASX (AU) |
202 |
| DK (DK) |
182 |
| RNS (UK) |
176 |
| FI (FI) |
114 |
| CNMV (ES) |
101 |
| SEDI (CA) |
68 |
| FMA (AT) |
50 |
| IE (IE) |
2 |
SEC backlog (Form 4, 2020–2026) is the bulk of the merge — driven by the 1255× speedup applied to src/lib/ingest/merge-staging.ts.
Company enrichment coverage (post-pass)
| Field |
Coverage |
Δ this session |
logoUrl |
4,932 / 8,591 (57%) |
+1,392 |
yahooSymbol |
5,889 / 8,591 (69%) |
+86 (in progress) |
relatedCompanySlugs |
8,591 / 8,591 (100%) |
full backfill ✅ |
The populate:related pass executed during the session brought related-slug
coverage to 100% on every Company and Insider entity (previously 68%).
2. Features shipped
Pipeline / data
src/lib/ingest/merge-staging.ts — 1255× speedup (batched upsert + connection-pool aware).src/lib/internal-linking.ts — graph refresh on 8.5k Companies + 58k Insiders.scripts/recompute-backtest-multimarket.mjs — re-runs the 17-market backtest harness.scripts/fetch-logos-coverage-push.mjs — aggressive logo backfill on new entities (SEC bias).scripts/smoke-curl.sh + scripts/smoke-sample.ts — prod smoke harness (random
weighted sample of 5 companies + 5 insiders per market).scripts/mint-session.ts — dev session token helper.
Site / content
- New route:
/insiders/by-market/[market] — market-specific insider hubs (17 pages).
- Methodology refresh: Ch01 Sources, Ch02 Pipeline, Ch10 Limits, ChDisclosure updated
to reflect all 17 venues + SEC volume.
- Landing components (Hero / Markets / Features / How-It-Works / WhyUs) regenerated
for multi-market story.
/admin/sources Phase-2: SourceCard extended with lastAttempt, lastSuccess,
errorTail, trend7d, sampleRow, disabled (UI stubs + safe defaults; enricher
pass tracked separately).- New components:
SourceDebugPanel, PublicSourceHealth.
- Public
/status page for source health (visible to all users).
docs/method-review/30-backtest-final-17markets.md — full multi-market backtest report.
Quant / API
src/lib/winning-strategy.ts — multi-market scoring adjustments.src/lib/recommendation-engine.ts — market-aware features.src/lib/mcp/tools.ts — multi-market MCP tool updates.src/lib/sources-registry.ts — single source of truth for source metadata + disabled flags.
3. Performance gains
| Workload |
Before |
After |
Gain |
| SEC staging → Declaration merge |
~52 min/10k |
~2.5 s/10k |
~1255× |
| Related-slugs backfill (all) |
n/a |
end-to-end ✓ |
100% |
| Logo coverage on non-AMF |
3,540 (41%) |
4,932 (57%) |
+39% |
4. Deploy validation
✅ Prod 200 https://insiders-trades-sigma.vercel.app/api/version → 200
✅ Vercel build Ready (deploy 5xqiweysx, target=production)
✅ SHA match /api/version → "sha":"f632a2d" == HEAD f632a2d
✅ TypeScript tsc --noEmit clean
✅ Local build next build clean (same as ci.yml mirror)
✅ Smoke tests 17/17 prod URLs returned 200
(/ /companies /insiders /heatmap /recommendations
/performance /methodologie + 5 entities + 5 market hubs)
⚠️ GH Actions cannot poll (no GITHUB_TOKEN in env); Vercel green is the
authoritative gate for this stack
5. Commits this session
| SHA |
Message |
45924c6 |
feat(multi-market): SEC merge 1255× speedup, 17-market backtest, … |
f632a2d |
fix(types): align admin/sources + performance with current Declaration schema |
6. Remaining TODOs (post-handoff)
- Yahoo enrichment still in-flight (~384/6022 done when last checked) —
background process running; coverage should reach >80% within ~1 h.
- Logo enrichment still in-flight (~15% through non-AMF batch) — let it
finish, expect final logo coverage ~70-75%.
/admin/sources Phase-2 enricher — the UI accepts lastAttempt,
lastSuccess, errorTail, trend7d, sampleRow but buildCard() currently
emits empty defaults. Wire up the IngestionRun history queries to populate.- Form4 → Declaration insider mapping for SEC — 50k insider rows landed,
but role normalisation may need a second pass against the new dataset.
- IE (Ireland) has only 2 declarations — Euronext Dublin ingest needs a
separate audit (likely XML feed change or selectors stale).
- Sitemap audit post 4× entity-count explosion — re-run
npm run audit:links once enrichment passes finalise.
7. Risk register (none blocking)
- Prod is stable on
f632a2d; rollback target = 9b46046 if needed.
- Pre-push hooks caught 4 stale TypeScript errors during the session
(agent-parallel edits creating partial states). All fixed inline.
- No data loss events. Merge-staging idempotency preserved.
Generated autonomously during 3-hour unsupervised window.