Leaderboard · release v0.1.0 · condition structured_memory

Component metrics first

Each cell is the macro-average over stories of the per-story pass rate in that dimension. Hover a cell for its 95% story-bootstrap CI. The global column is an unweighted mean of the ten dimensions and exists only for ranking; its sensitivity to weights is reported below.

Evidence status. Every model here is a procedural simulator with stated parameters (see Methods). The board demonstrates that the measurement stack separates models with known differences; it does not rank real systems.

Primary leaderboard. Coherence = memory-requirement pass rate; Adherence = stated-requirement pass rate.
Model	Identity	Clothing	Object persistence	Object state	Spatial	Environment	Causal	Chronology	Action	Adherence	Coherence	Global
sim·amnesiacsim	96.3%	97.2%	96.5%	91.9%	93.5%	99.8%	100.0%	99.7%	98.3%	98.7%	96.3%	97.5%
sim·drift-lowsim	99.7%	97.5%	96.9%	89.5%	98.1%	99.6%	93.6%	99.2%	99.2%	99.4%	98.1%	98.2%
sim·drift-midsim	96.1%	95.0%	95.4%	95.6%	97.1%	97.2%	80.0%	97.7%	98.2%	97.7%	95.1%	96.4%
sim·oraclesim	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%
sim·sloppysim	75.5%	67.2%	74.9%	74.2%	53.0%	80.3%	62.9%	84.0%	81.6%	83.3%	68.9%	76.4%

By tier (global score), first-failure position, pixel pairwise similarity, and generation cost.
Model	Easy	Medium	Hard	First failure (rel. pos.)	Pairwise figure sim.	API calls	Cost (USD)	p50 latency
sim·amnesiac	96.7%	97.8%	97.7%	0.68	0.518	1260	$0.00	0.004 s
sim·drift-low	98.4%	97.9%	98.7%	0.81	0.515	210	$0.00	0.004 s
sim·drift-mid	98.8%	96.8%	93.1%	0.60	0.506	1470	$0.00	0.004 s
sim·oracle	100.0%	100.0%	100.0%	1.00	0.521	210	$0.00	0.005 s
sim·sloppy	79.3%	78.6%	69.1%	0.24	0.372	210	$0.00	0.004 s

Paired comparisons

Story-level paired bootstrap on the global score (2,000 resamples). A comparison is significant when the 95% interval excludes zero.

A	B	Δ global	95% CI	P(Δ>0)	Win rate	Sig.
sim·oracle	sim·drift-low	1.8%	[0.8%, 3.0%]	1.000	75%	yes
sim·oracle	sim·amnesiac	2.5%	[1.4%, 3.9%]	1.000	81%	yes
sim·oracle	sim·drift-mid	3.6%	[2.2%, 5.2%]	1.000	84%	yes
sim·oracle	sim·sloppy	23.6%	[20.0%, 27.4%]	1.000	100%	yes
sim·drift-low	sim·amnesiac	0.8%	[-1.0%, 2.4%]	0.805	56%	no
sim·drift-low	sim·drift-mid	1.9%	[-0.0%, 3.5%]	0.975	68%	no
sim·drift-low	sim·sloppy	21.8%	[18.1%, 25.8%]	1.000	100%	yes
sim·amnesiac	sim·drift-mid	1.1%	[-0.8%, 2.9%]	0.875	64%	no
sim·amnesiac	sim·sloppy	21.1%	[17.6%, 24.6%]	1.000	100%	yes
sim·drift-mid	sim·sloppy	20.0%	[16.3%, 24.1%]	1.000	98%	yes

Score weighting and its sensitivity

Weights are equal (0.100 each) by construction. We perturbed the weights 1000 times with a flat Dirichlet and re-ranked. Mean Kendall τ between the equal-weight ranking and perturbed rankings: 0.891.

Model	Equal-weight rank	P(rank unchanged)
sim·oracle	1	100%
sim·drift-low	2	56%
sim·amnesiac	3	54%
sim·drift-mid	4	91%
sim·sloppy	5	100%

Prompt sensitivity

The same stories with a deterministic surface paraphrase of every scene text (synonyms only; every fact kept). A benchmark that moves a lot here is measuring wording, not coherence.

Model · condition	Plain	Paraphrase	Δ	95% CI
sim·drift-mid\|structured_memory	96.4%	96.4%	0.00%	[0.00%, 0.00%]

All conditions

Intervention arm: the same models under the other five prompting conditions. See Models for paired analysis.
Model · condition	character identity	clothing	object persistence	object state	spatial consistency	environment	causal continuity	chronology	action consistency	text image adherence	coherence	global
sim·amnesiac · full_context	98.4%	96.4%	94.1%	88.6%	96.3%	97.8%	84.3%	98.9%	97.6%	98.7%	96.0%	96.8%
sim·amnesiac · independent	60.7%	53.0%	53.3%	61.6%	42.8%	77.5%	63.6%	90.0%	61.4%	90.2%	45.1%	68.3%
sim·amnesiac · previous_image	90.5%	85.7%	86.8%	80.9%	91.5%	74.6%	65.7%	94.1%	79.5%	98.5%	78.3%	86.8%
sim·amnesiac · previous_text	70.6%	60.3%	59.0%	70.8%	50.0%	88.1%	50.0%	97.9%	69.9%	91.1%	59.3%	75.5%
sim·amnesiac · summarized_state	96.9%	96.8%	96.9%	95.2%	94.9%	98.5%	98.6%	100.0%	98.1%	98.9%	96.3%	97.8%
sim·drift-mid · full_context	94.4%	94.0%	98.0%	89.1%	90.9%	96.5%	91.4%	99.0%	94.5%	98.4%	92.8%	95.4%
sim·drift-mid · independent	83.1%	74.5%	75.2%	75.2%	83.5%	87.6%	57.1%	98.6%	86.8%	94.6%	74.6%	85.0%
sim·drift-mid · previous_image	90.6%	91.4%	93.3%	95.8%	95.3%	91.8%	95.7%	96.0%	88.9%	97.3%	88.4%	93.2%
sim·drift-mid · previous_text	88.8%	86.2%	88.3%	88.1%	86.7%	92.7%	90.0%	98.8%	91.9%	97.1%	85.2%	91.7%
sim·drift-mid · summarized_state	95.5%	92.5%	99.1%	90.5%	90.4%	95.3%	85.7%	97.4%	98.7%	97.9%	93.3%	95.6%