The “Lobster on a Dirt Bike” Test
One absurd prompt. Every model. Every month. Inspired by Simon Willison's “pelican on a bicycle” — our signature compositional benchmark tests whether AI models can compose two unrelated concepts in a novel spatial relationship.
Two unrelated concepts in a specific spatial relationship
Almost certainly doesn't exist in training data
Success or failure is immediately visible at a glance
Failure modes tell you more than the successes do
Text-to-Image Results
Same prompt, default settings, no prompt engineering. Side by side.
GPT-4o
ImageOpenAI native image generation
Midjourney v7
ImageDiscord/web UI
Flux 1.1 Pro
ImageBlack Forest Labs via Replicate
Ideogram 2.0
ImageBest-in-class text rendering
Imagen 3
ImageGoogle via Vertex AI / ImageFX
Recraft V3
ImageStrong on design/vector
Across Every Modality
Video Game
“A 2D platformer where a lobster on a dirt bike collects golden shells”
3D Scene
“Lobster on a dirt bike, parked outside a neon-lit diner at dusk”
Creative Text
“Write a short story about a lobster who enters a dirt bike race”
Research
“Research competitive dirt bike racing for crustaceans”