The benchmark has an intervention arm: the same stories under six prompting strategies. It answers RQ5 (does explicit state prompting help?) and RQ6 (do prior images help?) without a leaderboard.
Evidence status. Simulator responses to conditions are built into their parameters (facts available through context or a reference image are corrupted at a lower, stated rate). The intervention analysis therefore demonstrates that the pipeline detects such effects with paired story-level statistics; it says nothing about whether real models respond to these conditions. Hosted adapters (OpenAI Images, Replicate, Stability, Google Imagen) run under identical conditions when credentials are set.
Simulators on this release
Model
Parameters
Role
sim·oracle sim
p_memory 0.00 · p_adherence 0.00
Perfect model. Establishes the evaluator's false-positive floor.
sim·drift-low sim
p_memory 0.04 · p_adherence 0.01
Rare memory drift; drifted facts persist with probability 0.7.
Good memory, poor adherence: many prompt-stated facts are missed.
p_memory: per-scene probability that a fact available only from memory is corrupted. p_adherence: the same for a fact stated in the scene text (context-provided facts use p_adherence × a context factor: structured 1.0, summarised 1.3, full context 1.6). image_bonus multiplies p_memory for visual facts when a reference image is supplied.
Conditions
Condition
What the model receives
independent
Each scene is generated from its own text only. No memory of earlier scenes.
previous text
The previous scene's text is prepended to the current scene text.
structured memoryprimary
A compact JSON state (characters, attributes, held objects, object states, order, setting) from the end of the previous scene is prepended.
previous image
The previous generated image is passed as a reference to adapters that support image conditioning; text is the scene only.
summarized state
A one-paragraph natural-language summary of every persistent fact is prepended.
Paired story-level differences (2,000-resample bootstrap). RQ5 = structured memory vs independent; RQ6 = previous image vs independent.
Comparison
Δ coherence
95% CI
Δ identity
Δ clothing
Δ object persistence
Δ global
Sig. (coherence)
full context vs independent
18.2%
[13.5%, 23.3%]
11.3%
19.5%
22.7%
10.4%
yes
full context vs structured memory
-2.3%
[-4.7%, 0.3%]
-1.8%
-1.0%
2.5%
-1.0%
no
independent vs structured memory
-20.5%
[-25.5%, -16.0%]
-13.1%
-20.5%
-20.2%
-11.4%
yes
previous image vs independent
13.8%
[8.3%, 19.2%]
7.5%
16.9%
18.1%
8.1%
yes
previous image vs structured memory
-6.7%
[-11.4%, -3.1%]
-5.5%
-3.7%
-2.1%
-3.2%
yes
previous text vs independent
10.6%
[5.6%, 16.0%]
5.7%
11.7%
13.1%
6.7%
yes
previous text vs structured memory
-9.9%
[-12.8%, -6.8%]
-7.3%
-8.8%
-7.1%
-4.7%
yes
structured memory vs independent
20.5%
[16.0%, 25.5%]
13.1%
20.5%
20.2%
11.4%
yes
summarized state vs independent
18.7%
[14.6%, 22.8%]
12.4%
18.0%
23.8%
10.6%
yes
summarized state vs structured memory
-1.8%
[-4.5%, 0.9%]
-0.6%
-2.5%
3.6%
-0.7%
no
RQ3 — single-image quality vs narrative coherence
Text–image adherence (fraction of prompt-stated facts satisfied) is the single-image quality proxy available offline; on real models it would be joined by ImageReward/HPS-style scores. Large dots are model means; small dots are stories.
Adherence vs coherence, primary condition. The dashed line is equality.
Adherence (prompt-stated facts satisfied) is the single-image quality proxy; with n_models this small the correlation is descriptive only. Model-level Spearman ρ = 1.00 (n = 5).
Decay by tier and fitted decay rates
Independent condition: memory pass rate by scene for the mid-drift simulator, split by tier.
easy medium hard
Fitted per-scene decay rate λ (log-linear fit of memory pass rate on scene index ≥ 2), by condition.