How we compute the scoreboard

What we compare

Each dynasty ranking source predicts how players will perform over a multi-year dynasty horizon. We measure how well each source's preseason ranking correlates with the actual cumulative fantasy points scored over the post-prediction window.

• Dynasty General v0: Our engine output. v0 uses position rubrics (RB / WR / QB / TE) plus age curves, with KTC value as a Bayesian prior. v0 has NO signal codes loaded yet (Track A in progress); position rubrics fall back to KTC-prior + age curves only.
• FantasyPros ECR: Their published expert blend, accessed under a paid trial in May 2026 from fantasypros.com/nfl/rankings/dynasty-overall.php?year=YYYY.
• FantasyPros ADP: Their published dynasty 1QB ADP page, same access path.
• KeepTradeCut: Dynasty value rankings recovered from public Wayback Machine snapshots of `keeptradecut.com`.

Loss function

We follow VALIDATION_PLAN section 8.2. The dynasty loss is a 0.6/0.4 weighted composite of KTC 6-month value drift and cumulative-3-year PPR points error.

L_dynasty = 0.6 * KTC_6mo_value_MAE
          + 0.4 * cumulative_3yr_PPR_RMSE

In v0 we publish the cumulative-PPR-rank component (Spearman rank correlation) and a separate KTC-value-drift summary. v1 will composite them into the full weighted loss.

The cumulative window adapts to outcome data availability:

• 2022 prediction → 3-year cumulative (2022 + 2023 + 2024)
• 2023 prediction → 2-year cumulative (2023 + 2024)
• 2024 prediction → 1-year (2024 only); will become 3-year when 2025 + 2026 outcomes are ingested

Metrics published

• Spearman rank correlation between predicted dynasty rank and actual cumulative-PPR rank
• Mean absolute rank error across joined (predicted, actual) pairs
• Top-50 hit rate and Top-100 hit rate: overlap between predicted top-N and actual top-N
• KTC 6-month value drift (where data permits): how much value did each predicted top-N player gain or lose 6 months after prediction
• Sample size (n_pairs) for every row, so confidence intervals can be inferred (small n means wide CI)

What we do NOT publish

• Source player-by-player rankings
• Source-internal ordering reproductions
• Cropped excerpts of FantasyPros tier displays

We compute. We publish the computed numbers. We cite the source.

Snapshot dates

The 2024 FP ECR snapshot is dated Jan 9 2025 in their UI but the content is preseason 2024 (rookie variance signature confirms: Jayden Daniels worst=109, std_dev=24.8 indicates experts had not yet seen his rookie season). We trust the variance over the label.

Reproducibility

To reproduce a row in this scoreboard:

1. Subscribe to the source (FP requires a paid trial)
2. Capture the preseason-snapshot rankings for the target year and format
3. Score against `historical_outcomes` from the public Sleeper season-stats endpoint
4. Compute Spearman rank correlation between predicted rank and actual cumulative-PPR rank
5. Restrict to top-N predicted with n-pairs visibility

Scripts (in the public repository under web/scripts/):

• ingest-fp-csvs.ts
• ingest-ktc-historical.ts
• ingest-historical-outcomes.ts
• run-engine-historical.ts
• backtest-score-dynasty.ts
• build-scoreboard.ts

Industry prior art

Comparative-accuracy publication is standard in the fantasy industry. FantasyCalc maintains a public performance-analysis page that explicitly compares their values against FantasyPros ECR by name. Draft Sharks publishes commentary on FantasyPros accuracy. Academic sports-analytics literature regularly cites and benchmarks against industry sources.

We are not the first to publish a comparative scoreboard. We aim to be the first to publish a calibrated dynasty-loss- function-based comparative scoreboard with full reproducibility documentation.

Caveats

1. Sample sizes are small. Top-100 joined pairs per source per year run 60-90. Confidence intervals on Spearman values are roughly ±0.05 to ±0.10.
2. Engine v0 has zero signal codes loaded. Track A (`scripts/extract-historical-signals.ts`) is in progress. v1 with signals will be re-run.
3. Loss function is partial in v0. We publish the cumulative-PPR-rank component. The full §8.2 loss composites that with KTC value drift; v1 will composite.
4. Dynasty horizons differ by source. Source rankings target 1-3 year horizons, not single-season. Single-season-only Spearman scores are noisy by design.