Model card: correct the recalibration — leaderboard is best-fit only, so the like-for-like gap is +0.019; both checkpoints now measured under both packings

README.md

@@ -49,43 +49,52 @@ DecoderStack tracks 0.2–1.4% behind through step 1000 (where the upstream base
 stopped) while running ~6% faster per step (1,212 vs 1,291 ms/step). Full head-to-head in
 [`logs/baseline_report.md`](logs/baseline_report.md).
 
-⚠️ **but 0.719042 is not comparable to an upstream nanochat val bpb.** Upstream batches with
-best-fit cropping — it fills each row with largest-doc-that-fits and crops a document to
-finish the row, then attends causally across the whole row with no document isolation
-(`cu_seqlens` appears nowhere in upstream `nanochat/gpt.py`). DecoderStack packs varlen and
-isolates documents in validation as in training. Different truncation points, different
-conditioning context: the two are not scoring the same predictions.
-
-### Measured 2026-08-01: the packing gap is larger than the model gap
-
-Re-measured this checkpoint under **upstream's own row packer**, driven with this repo's
-tokenizer over the same pinned val shard and unwound into varlen batches whose `cu_seqlens`
-sit on the row boundaries — so the model predicts exactly the tokens upstream would ask
-for, with exactly the same context.
-
-| packing | val tokens | bpb |
+⚠️ **but 0.719042 is not comparable to an upstream nanochat val bpb, and reads better than
+a like-for-like number would.** Upstream batches with best-fit cropping — it fills each row
+with largest-doc-that-fits and crops a document to finish the row, then attends causally
+across the whole row with no document isolation (`cu_seqlens` appears nowhere in upstream
+`nanochat/gpt.py`). DecoderStack packs varlen and isolates documents in validation as in
+training. Different truncation points, different conditioning context: the two are not
+scoring the same predictions. Measured below.
+
+### Measured 2026-08-01: what this checkpoint scores under upstream's packing
+
+Varlen packing does not exist in upstream nanochat — it is this fork's own work, and
+**every nanochat leaderboard entry is measured with karpathy's best-fit row packer**. So a
+varlen val bpb cannot be read against the leaderboard at all. To get a comparable number,
+both checkpoints below were re-scored under upstream's row packer, driven with each
+model's own tokenizer over the same pinned val shard and unwound into varlen batches whose
+`cu_seqlens` sit on the row boundaries — the model predicts exactly the tokens upstream
+would ask for, with exactly the same context.
+
+Both models, both packings, same code, same shard, 41.9M val tokens:
+
+| model (step 5568) | best-fit rows | varlen |
 |---|---|---|
-| upstream best-fit rows | 41.9M | **0.737183** |
-| upstream best-fit rows | 10.5M | 0.735274 |
-| varlen, document-isolated | 41.9M | 0.724316 |
-| varlen, document-isolated | 10.5M | 0.720717 |
-
-(all on the legacy `token_bytes` basis, for continuity with the numbers above; the
-regenerated basis reads ~0.05% higher — see [⚠️ Tokenizer](#️-tokenizer))
-
-Three things follow, and they revise the comparison:
-
-1. **Packing alone is worth ~0.0129 bpb** — same model, same shard, same budget. That is
-   *larger* than the gap between this model and its sibling release, so bpb must never be
-   cross-compared between the two packings.
-2. **`ChrisMcCormick/nanochat-varlen-d24-2026-03-22` is ahead, not behind.** Its 0.724772
-   was measured under best-fit rows; the like-for-like number here is **0.737183**, so it
-   leads by ~0.0124 bpb. The raw headline figures (0.719042 vs 0.724772) pointed the other
-   way purely because of packing. It also trained with FP8 and a different 32k tokenizer,
-   so this is still not a controlled comparison — but it is now an honest one.
-3. **The 10.5M val budget reads low.** Upstream's default is `--eval-tokens 80*524288` =
-   41.9M; this run used 10.5M, which comes in ~0.002–0.004 bpb optimistic rather than
-   merely noisier.
+| **this checkpoint** (`d24_decoderstack`) | **0.737183** | 0.724316 |
+| `nanochat-varlen-d24-2026-03-22` | 0.737535 | 0.725014 |
+
+At the 10.5M budget this run originally used: 0.735274 (best-fit) / 0.720717 (varlen).
+All on the legacy `token_bytes` basis, matching how the leaderboard numbers were computed;
+the regenerated basis reads ~0.05% higher — see [⚠️ Tokenizer](#️-tokenizer).
+
+What this says:
+
+1. **Against the leaderboard, the honest gap is ~0.019, not ~0.006.** The nanochat record
+   is 0.71808 (Run 5) / 0.71854 (Run 4), both best-fit. Like-for-like this checkpoint is
+   **0.737183**, i.e. **+0.0191** — matching upstream's tokens moves *away* from the
+   record, not toward it. The 0.719042 headline above invited the opposite impression.
+2. **The packing penalty is universal, not model-specific**: +0.0129 here, +0.0125 for the
+   sibling release. Best-fit truncation costs ~0.0127 bpb whoever runs it.
+3. **The two checkpoints are near-identical**, this one marginally ahead under both
+   packings (−0.00035 best-fit, −0.00070 varlen). Not a controlled comparison — the
+   sibling used FP8 and a different 32k tokenizer — but the two pipelines land in the same
+   place.
+4. **The 10.5M val budget reads optimistic**, not merely noisier: ~0.002–0.004 bpb below
+   the 41.9M number. Upstream's default is `--eval-tokens 80*524288` = 41.9M.
+
+Harness check: the sibling run self-reported 0.724772 and is measured here at 0.725014
+varlen — 0.03% agreement, using the same loader that run used.
 
 Reproduce: `agent-ops/stacks/2026-08-01_1241pm_d24-val-bpb-recalibration/`.