# Burstiness research release

This package supports **Burstiness Was Measured Wrong, and Prompting Cannot Aim It**,
accepted as a poster at the NeurIPS 2026 Workshop LP4FM and published on OpenReview.
The authoritative record is [OpenReview submission 114](https://openreview.net/forum?id=1E20ig92Zi).

`publication/paper.pdf` is the byte-exact public PDF. `paper/lp4fm/` contains the exact
submitted LaTeX source recovered from the anonymous repository linked by that PDF.
Both retain anonymous submission formatting. A later camera-ready revision has not
been verified. `publication/submission-source.json` records the recovery and file hashes.

## Included

- Model-free metric implementation, normalization, scoring, and generation harness.
- All nine families' scored evaluations, metadata, multiplicity and available equivalence
  assessments, plus the audited Qwen raw-protocol score archive.
- Scored contamination and paraphrase probes and their analysis code.
- Exact accepted PDF, submitted LaTeX source, figures, bibliography and style file.
- Original `data/released/hf-datasets.zip`: 200-item Wikinews human baseline, earlier
  paired ablation corpus, and post-scrub raw generation mirrors for eight model legs.
- October 18 data-handoff notes and sanitized inventory, with private dataset revisions
  and checksums. The newly available raw files are excluded from this public package.
- Offline tests, PDF and raw-data verifiers, deterministic package builder, and notices.

The benchmark is descriptive: neither the Goodhart guard nor the coherence gate filters
the reported generations. Delta is human minus model: positive means the model is flatter.
Provider decoding differs, and missing outputs reduce paired counts on some legs.
Read each scored cell's `n_pairs` rather than assuming every cell has 200 pairs.

## Reproduction limits

Seven recovered raw legs reproduce the committed scored JSON byte-for-byte after the
canonical normalization and scoring steps. The Qwen mirror contains 2,987 rows and is
incomplete relative to the scored evaluation: its few-shot and plan statistics differ,
including few-shot B_n 0.118 versus the published 0.119. The publication's scored snapshots
are preserved. Exact known differences and a repeatable check are in
`data/released/REPRODUCTION.json` and `scripts/verify-released-data.py`.

Claude Opus 4.8 raw outputs were not mirrored in the submitted archive; its scored
results are included. The original 3,000 Opus outputs were recovered on October 18 and
uploaded privately, alongside separate new DeepSeek runs containing 600 paraphrase and
3,000 baseline outputs. These require collaborator access and are excluded from this
public package. See [the handoff notes](camera-ready-data-2026-10-18.md) and
[verified inventory](camera-ready-data-2026-10-18.json) for revisions, hashes, later-snapshot
disclosures and the retained off-topic baseline flag.

The public archive is still not a complete nine-leg raw-data reproduction. Its bytes,
the accepted PDF and the published scored snapshots remain unchanged. Sentence-segmentation
correction and rescoring are pending; the checks below verify the archived scoring state.
Research database, private notes, internal reviews, and historical manuscript drafts are
excluded. Provider access and generation costs remain separate from the released artifacts.

## Offline checks

Run from the extracted package directory with Python 3.10 or later. These checks use only
the standard library and local data, with no model generation or network call:

```sh
python3 -m unittest discover -p 'test_*.py'
python3 -m experiments.burstiness_pp --demo
python3 scripts/verify-released-data.py --jobs 4 --output reproduction-check.json
```

The raw-data verifier normalizes before scoring, uses the canonical scorer, and succeeds
only when it reproduces the documented seven-exact/Qwen-incomplete state. For a quick
single-leg check add `--family deepseek-v4-pro`. Do not use the raw mirror README's direct
scoring example to reproduce the normalized protocol: that historical example omits
normalization. The equivalent explicit sequence for one leg is:

```sh
unzip data/released/hf-datasets.zip -d restored
python3 -m experiments.ablation.normalize_generation \
  restored/hf-datasets/deepseek-v4-news-gen/deepseek-v4.jsonl restored/deepseek-normalized.jsonl
python3 -m experiments.ablation.seed_avg_paired \
  restored/hf-datasets/paper-dataset/prompts_200_human.jsonl restored/deepseek-normalized.jsonl deepseek-v4-pro
```

The PDF-number check additionally requires Poppler's `pdftotext` on PATH:

```sh
python3 scripts/verify-pdf-against-ssot.py publication/paper.pdf
```

It compares 45 Table 1 cells, 180 Appendix B cells, and 36 multiplicity-census entries.
This verifies printed results against recorded statistics; raw reproduction is the separate
check above. Optional provider clients are in `bench/requirements.txt`. Rebuilding human
article bodies from original wikitext requires `mwparserfromhell`.

## Manuscript source

```sh
bash paper/lp4fm/build.sh
python3 scripts/verify-pdf-against-ssot.py paper/lp4fm/main.pdf
```

The verified build uses Tectonic 0.15.0 and produces 20 pages whose extracted text matches
the accepted PDF after whitespace normalization. Original font and layout warnings remain;
compiler-dependent PDF bytes are not expected to match the official PDF. The separate
`burstiness-lp4fm-source-v1.1.1.zip` places `main.tex` at its root for Overleaf import and
excludes generated PDFs and logs. Figure PDFs are included; redraw requires matplotlib
and numpy. Neither archive changes the submitted author block into a camera-ready paper.

## Integrity and deterministic rebuilding

`MANIFEST.json` lists payload sizes and SHA-256 values and a content snapshot digest.
`SHA256SUMS` covers both downloadable archives. These detect accidental corruption; they
are not digital signatures.

```sh
shasum -a 256 -c SHA256SUMS
python3 scripts/build-release-package.py --output-dir rebuilt
python3 scripts/build-release-package.py --output-dir rebuilt --check
```

The builder uses an explicit allowlist, rejects symlinked inputs and common credential
signatures in allowed text, verifies recovered source/data hashes, and fixes archive
metadata. Identical allowed inputs produce identical bytes after extraction. `--check`
verifies existing outputs without writing. Unlisted files cannot enter either archive.

## Licenses

The paper is CC BY 4.0. Code is Apache-2.0. Wikinews text retains CC BY 2.5 and attribution
requirements; generated outputs retain their applicable provider/model terms and are not
blanket-relicensed as Apache-2.0. See `NOTICE`, `THIRD_PARTY_LICENSES.md`, and
`data/released/README.md` for corpus and model provenance.