Everything needed to rerun or reject the benchmark
Methods, stated so they can be checked.
Full protocols live in the repository's research/ directory; this page is the summary.
Generative model
Periodic 64×64 worlds (32² and 128² in the sample-size slice). Noise has unit variance, so the signal amplitude is the signal-to-noise ratio: 2.0, 1.0, 0.5, 0.2. Contamination amplitude 1.5 by default — the systematic is stronger than the signal.
Six synthetic nuisance templates (depth, extinction, stellar density, seeing, airmass, scan stripes), smooth and partly degenerate. Four signal families: power-law Gaussian field, log-normal clustered field, a field with a single-scale spectral feature, and sparse localised events. Six contamination families: linear, nonlinear, stripe- dominated, depth-multiplicative, localised artefacts, and half-unmodelled.
Correlation. The signal is rebuilt as s = σ(√(1−ρ²)·u + ρ·v) with u orthogonal to
every template and v a random unit direction inside their span, so exactly ρ² of its variance can be
mimicked by the templates. Verified to 10⁻⁹ in the tests.
The decomposition
Regressing the output on (signal, contamination, noise) lets least squares credit destroyed signal back through the
contamination term when the two are correlated: it reported 0.81 transfer on a world where the correction provably
destroyed 36% of the signal variance. DarkSignal splits the signal first, s = s⊥ + s∥ with
s∥ inside the true template span, and regresses the output on [s⊥, s∥, c, n].
| Metric | Meaning |
|---|---|
| clean signal transfer | coefficient on s⊥ · below 1 is erasure the method was not forced into |
| degenerate transfer | coefficient on s∥ · full template regression drives it to 0 |
| signal transfer | (1−ρ²)·clean + ρ²·degenerate · closed form 1−ρ² for linear regression |
| contamination removed | 1 − coefficient on c |
| unexplained structure | residual variance ÷ signal variance · includes contamination left with a changed shape |
| T(k) | cross-spectrum transfer by wavenumber |
Anchors asserted in the test suite: no correction gives transfer 1 and leakage 1; the oracle gives 1 and 0; linear regression gives 1 − ρ².
Correctors
No transformer: the data is a template stack on a regular grid with no sequence structure and a few thousand pixels per world. The CNN takes template maps only — given the observed map as input and target it learns the identity. The autoencoder never sees a template; it is the "just denoise it" approach, included because it is common. ICA equals PCA under linear subtraction, to 10⁻¹⁴ in every world: whitening puts it in the same subspace and least squares is rotation-invariant.
Statistics and controls
The unit of replication is the simulated world. Differences are paired within world, summarised by 95% percentile bootstrap over seeds, and tested with Wilcoxon signed-rank and Holm correction. Where a slice had only five seeds the test could not reach p < 0.05 at all, so those slices were extended (decision log D19) with the rules unchanged.
Cross-validation uses 4 folds of contiguous tiles; random-pixel folds are run alongside to measure the leak. Controls pass at signal transfer ≥ 0.95 (zero contamination; spurious templates) and contamination removed ≥ 0.80 (strong contamination).
Reproduce
| Command | What it does |
|---|---|
make setup | venv + dependencies |
make test | unit and scientific tests, including the closed-form anchors |
make simulate-smoke | 126 runs in seconds; what CI runs |
make simulate | the canonical STANDARD grid |
make real-data | download the DESI map, verify its checksum, run the case study |
make analyze · figures · paper · site | derive every table, figure, the paper and this site |
make reproduce | all of the above |