IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation

Shaohan Yu^1,2 Jie Qin² Wende Tan³ Haiwen Diao⁴

Limeng Qiao² Yueming Lyu¹ Chenyang Si^1,†

¹PRLab, Nanjing University ²Meituan ³Imperial College London ⁴S-Lab, Nanyang Technological University

Explore IntReal-100K

High-quality interleaved data curated from a million-scale pool.

Planning

Wikivoyage · Wikitravel

Loading samples…

IntReal Evaluation Suite

Coarse-to-fine diagnosis with expert models.

Holistic

Generate the full interleaved response.

Generated image of garlic and butter in a pan — Assess the instructions and both generated images as one complete response.

Generated image of butter being brushed onto bread slices — Assess the instructions and both generated images as one complete response.

Fine-grained

Fix real context. Generate only the next image.

Ground-truth image of a baguette and bread slices — Assess how faithfully the new image follows the fixed text and visual context.

Next image generated from the fixed real-world context — Assess how faithfully the new image follows the fixed text and visual context.

Seven metrics

Seven evaluation metrics and their expert models or evaluators, grouped by setting.
Metric	Expert model / evaluator
Holistic
Text Richness	LLM · Gemma4
Text Logicality	LLM · Gemma4
Image Aesthetic	IQA · RALI
Image Diversity	DINOv3
Quantity Compliance	Exact image count
Fine-grained
Context Fidelity	VLM · Gemma4
Subject Consistency^†	DINOv3

^† Subject Consistency is evaluated on Tutorial and Story.

IntReal Experimental Results

Model strengths, the open–closed gap, and the effects of SFT and RL.

NanoBanana2 leads strongly on text, even against the best open-source score for each metric.

Metric	Best OSS	NanoBanana2

Best OSS is a per-metric envelope across evaluated checkpoints, not a single model.

SFT improves text at a visual cost. Image-generation RL partially recovers visual quality and reaches the best mean among evaluated OSS models.

U1-8B-MoT

Metric	Base	+ SFT	+ SFT + RL

IntReal Metric Reliability

Complementary dimensions, human agreement, and consistent evaluator rankings.

Complementary metrics

max |ρ| = 0.42

Seven evaluation dimensions

Different metrics capture different strengths.

Limited redundancy in sample-level correlations, averaged over five models.

Human agreement

83.3%

Average agreement with expert judgments

Automatic scores align with human preferences.

750 expert pairwise comparisons across five subjective metrics: TR, TL, ID, CF, and SC.

Evaluator robustness

ρ > 0.90

Gemma4 vs. Qwen3.6-35B-A3B

Model rankings stay consistent across evaluators.

Eight model variants. Rank correlations: TR 0.905 · TL 0.929 · CF 0.976.