Methodology

This page is the reference description of how the Visual Story Benchmark (VSB) measures long-horizon coherence. Every number on the site is computed from the frozen release v0.1.0 by the code in the vsb package; nothing is hand-entered.

1. The benchmark unit

A story is a JSON document validated against vsb/schema/story.schema.json (JSON Schema 2020-12). It contains:

1. a world specification: 1–3 characters (skin tone, hair colour and style, top colour, trouser colour, accessory), 0–3 objects (kind, colour, initial state, optional count) and the locations visited;
2. a scene sequence (3, 5 or 8 scenes). Each scene carries events (enter, exit, move, set_time, set_weather, appear, remove, change_state, give, drop, pickup, set_count, change_attr, swap, set_pose) and a natural-language prompt composed from those events;
3. for every scene, the expected state obtained by replaying the events through a deterministic state machine (vsb/state.py), and a list of requirements;
4. causal dependencies (cause scene → every later scene in which the effect must persist).

The state machine refuses impossible transitions (giving an object one does not hold, a state going backwards, time going backwards without a new day). Every story in a release passes schema validation and transition validation.

Requirements, and the adherence / coherence split

A requirement is a single checkable fact with a dimension, a taxonomy label to use on failure, and a source:

• prompt — the fact is stated in this scene's text (scene 1 states everything; later scenes are written like a screenplay and state only names, place and what changes);
• memory — the fact is not stated and must be carried from earlier scenes.

Text–image adherence is the pass rate over prompt requirements; coherence is the pass rate over memory requirements. This distinction is the core of the benchmark: a model can be perfectly faithful to every scene text and still incoherent.

Dimensions: character identity, clothing, object persistence (including counts and resurrections), object state, spatial consistency (left-to-right order), environment (place, weather), causal continuity (the effect of a stated cause, and its persistence), chronology (time of day), action consistency (pose), text–image adherence.

Story generator

Ten templates (vsb/generator/generate.py), one per category — object persistence, wardrobe, location transitions, counting, possession, cause/effect, damage/state change, character relationships, spatial arrangement, temporal progression — each with easy (3 scenes, 1 character), medium (5 scenes, 2 characters) and hard (8 scenes, 2–3 characters, with reversal traps: an object handed back, a character who leaves and returns, an object set down and picked up again still broken) variants. The default suite has 4 stories per category (easy, medium ×2 with different seeds, hard) = 40 stories, 210 scenes, 3,865 requirements. Generation is deterministic in the seed. Names are short and cross-cultural; skin tones use neutral labels drawn uniformly; no attribute carries an ethnic, gendered or religious marker.

2. Prompting conditions (interventions)

The scene text never changes across conditions; only the context does (vsb/conditions.py):

condition	context supplied
independent	none
previous_text	the previous scene's text
structured_memory (primary)	a compact JSON of the state at the end of the previous scene
previous_image	the previous generated image, for adapters that accept a reference image
summarized_state	a one-paragraph natural-language summary of every persistent fact
full_context	every previous scene text

A paraphrase prompt style applies a deterministic synonym substitution to every scene text (facts unchanged) for prompt-sensitivity analysis.

3. Model adapters

vsb/adapters/ defines a provider-neutral ImageModelAdapter (generate(request) -> image PNG + metadata + optional cost). Implemented: OpenAI Images (generations and edits for image conditioning), Replicate (any text-to-image model, image conditioning for Kontext-style models), Stability v2beta, Google Imagen :predict, a stored-samples adapter that serves images generated elsewhere from a fixtures directory, and the procedural simulators described below. Credentials are read from environment variables at call time; a missing credential skips the model and is reported. Provider adapters are exercised only through transport mocks in this repository.

Generation cache and cost ledger

Every request is hashed (sha256 over model id, version, prompt text, seed, sorted parameters, reference-image hashes; simulators add story/scene/condition because their output legitimately depends on them). Identical requests are never regenerated. The ledger records latency, success, cache hits and approximate cost from config/pricing.json (with an as-of date; unknown prices are reported as unknown, never as zero).

Procedural simulators

A simulator has privileged access to the story and renders the true expected state after stochastically corrupting it. For each fact the per-scene corruption probability is:

• p_adherence if the fact is stated in the current scene text (or scene 1);
• p_adherence × context factor if the condition supplies it through context (structured 1.0, summarised 1.3, full context 1.6 — long contexts are harder);
• p_memory × image_bonus for visual facts when a reference image is supplied;
• p_memory otherwise (memory only).

Drifted facts persist to the next scene with probability sticky = 0.7. The chain is re-simulated from scene 1 on every call, so outputs are a pure function of (model, story, condition, seed, scene) and independent of cache state. Every injected fault is written to a sealed oracle file that evaluators never read.

simulator	p_memory	p_adherence	image_bonus	role
procedural-oracle	0.00	0.00	—	evaluator false-positive floor
procedural-drift-low	0.04	0.01	0.35	mild drift
procedural-drift-mid	0.12	0.02	0.35	reference
procedural-amnesiac	0.35	0.01	0.25	great single images, no memory
procedural-sloppy	0.03	0.15	0.60	good memory, poor adherence

The renderer draws flat-colour illustrations (512×512) from a finite palette: characters as figures with exact palette colours for skin, hair, top and trousers; objects with state-dependent shapes (an intact umbrella is a semicircle, a broken one is two fragments; a lit lantern has a glow; an open book shows pages); backgrounds encode location (ground colour and props), time of day (sky colour, or a window for indoor scenes) and weather (rain lines, clouds, stars). Character placement is jittered per scene so consecutive frames are never pixel-identical.

4. Evaluators

No single evaluator determines coherence. Each produces measurements that are compared against the oracle and, when available, humans.

1. Blind structured-state extractor (vsb/evaluators/extractor.py). Prompted with the world's entity vocabulary (which kinds/colours of objects could appear — like an open-vocabulary detector given class names), never with expected states. It finds heads by skin-tone components, reads hair/top/trousers/accessory/pose from geometry-anchored samples, classifies objects by colour plus shape signature (fill ratio, aspect, fragment count), counts small components, and reads environment from ground/sky colours. On true renders it produces zero false violations over 15,460 requirement checks (four jitter seeds).
2. Rule checker (vsb/evaluators/rules.py). Matches detected people to expected characters by weighted attribute distance (identity attributes weigh most), then evaluates every requirement. Knock-on failures are attributed to their root cause (an undetected object fails its state check as a disappearance).
3. Pairwise pixel consistency (vsb/evaluators/pairwise.py): colour-histogram intersection between consecutive frames, globally and on the figure band. Model-agnostic and deliberately weak; it is included to test whether cheap visual-stability signals track coherence.
4. Embedding evaluators (scripts/embed_release.py, optional): CLIP ViT-B/32 text–image similarity per scene; DINOv2-small CLS cosine similarity between consecutive frames.
5. VLM question-answering judge (vsb/evaluators/vlm_judge.py, optional): every requirement becomes a YES/NO/UNSURE question with a calibrated system prompt; UNSURE never counts as a pass. Transports: OpenAI-compatible, Anthropic Messages, fixture replay, mock.

5. Scores

Per run (story × model × condition), the per-dimension score is the pass rate of that dimension's requirements (None if the story has none). Model-level scores are macro-averages over stories with story-clustered percentile bootstrap 95% CIs (2,000 resamples). The global score is the unweighted mean of the ten dimensions; it exists for ranking only, and a Dirichlet weight-perturbation analysis reports how stable ranks are under other weightings. Paired model comparisons use the paired story-level bootstrap (difference, CI, P(Δ>0), win rate).

6. Human evaluation

/evaluate runs a blinded session: model identity is replaced by a letter re-randomised per session; items are single-story Likert ratings on six dimensions (identity consistency, object consistency, causal coherence, narrative coherence, visual quality, prompt adherence), pairwise preferences, control items (the perfect model vs the memory-less amnesiac, with a known answer) and one repeated item. Ratings are stored in the browser and exported as JSON. vsb human import validates the export against rating.schema.json, excludes raters below 75% control accuracy or with a median item time under 4 s, and computes Krippendorff's α (ordinal and interval), ICC(2,k) on complete subsets, per-model means with CIs, pairwise preference tables, and a mixed-effects model (rating ~ model + (1|story) + (1|rater)) when statsmodels is available. vsb human simulate produces clearly labelled synthetic raters for pipeline verification only.

7. Judge validation

Against the sealed oracle: per-taxonomy recall and precision of the rule checker; AUROC of continuous signals for "this frame has ≥1 injected visual fault"; story-level Spearman/Kendall between each signal and oracle coherence. Against humans: story-level rank correlations per (signal, rated dimension) and model-ranking agreement.

8. Releases

vsb release freeze writes a manifest with a SHA-256 of every file (oracle excluded) and a root hash; vsb release verify detects any change. Every leaderboard page names its release.

9. Testing

pytest covers schema and generator determinism, state transitions, extractor round trips and fault detection, evaluator parsers, adapters (credentials, mocked transports, stored fixtures), cache, runner, aggregation, leaderboard, human-rating flow and release integrity.