Benchmark · research site · release v0.1.0

Beyond the Frame

Benchmarking long-horizon coherence in generative visual narratives

A single beautiful image says nothing about whether the umbrella is still broken three scenes later.

The Visual Story Benchmark (VSB) generates stories with exact, machine-checkable world state, asks an image model to draw every scene, reads the images back into structured state, and scores ten consistency dimensions separately. No single evaluator decides coherence, and no single number hides the dimensions.

40stories · 10 categories · 3 tiers

210scenes

3,865checkable requirements

640frozen runs

5models on the primary board

Evidence status. All rows on this release are procedural simulators: renderer-backed models whose fault rates are stated parameters. They validate the measurement pipeline (a perfect model scores 100%; injected faults are recovered with known recall). They are not measurements of any real image model. Hosted-provider adapters ship, are transport-tested, and run only when credentials are present. No human ratings have been collected yet; the human-rating interface and reliability pipeline are exercised with clearly labelled synthetic raters. See Judges and Methods.

01 — What is measured

Ten dimensions, two kinds of fact

Every scene carries requirements derived from a replayed state machine. A requirement is either stated in that scene's text (text–image adherence) or must be remembered from earlier scenes (coherence). Scene 1 states everything; later scenes read like a screenplay and state only what changes.

Identityskin tone, hair colour and style; presence and absence

Clothingtop, trousers, accessory

Object persistenceobjects stay, leave when they should, never resurrect; counts

Object statebroken / empty / lit / open / wilted persist

Spatialleft-to-right order of characters

Environmentlocation and weather

Causala stated cause has its effect, and the effect persists

Chronologytime of day follows the story

Actionstated pose or action is depicted

Adherenceevery fact stated in the scene text

02 — Primary leaderboard

Coherence under structured-memory prompting

Global score (unweighted mean of ten dimensions), primary condition = structured_memory. Whiskers: 95% story-bootstrap CI.

Full leaderboard with every dimension

03 — RQ1: coherence decay

How fast do memory facts get lost?

Without any memory (independent scenes) drift compounds scene after scene; with the structured-memory prompt the curve is flat because every persistent fact is restated. The gap between the two panels is the value of state prompting.

Independent condition: pass rate of memory (unstated) requirements by scene index. Bands: 95% CI over stories present at that index.

sim·amnesiac sim·drift-mid

Primary condition (structured_memory): the same curves.

sim·amnesiac sim·drift-low sim·drift-mid sim·oracle sim·sloppy

04 — RQ4: do the evaluators work?

Recovering injected faults

98.5%rule-checker recall of injected faults (sealed oracle)

74.4%precision, counting knock-on flags as false positives

0.42AUROC · pairwise figure hist

0.42AUROC · pairwise global hist

0.40AUROC · dino consecutive

Automatic-vs-human evaluation

05 — Explore

Story browser

Forty procedurally generated stories with full ground truth, side-by-side model outputs, and rule overlays.

Browse stories

Failure gallery

Every violation, labelled with an eleven-class taxonomy, searchable by model, category and scene.

Open gallery

Interventions

Six prompting conditions: independent scenes, previous text, structured memory, previous image, summarised state, full context.

Compare conditions

Rate stories

Blind, quality-controlled human evaluation interface. Export your ratings as JSON for the reliability pipeline.

Start rating