Research

Evaluate deep research and synthesis capabilities — source retrieval, cross-source synthesis, citation accuracy, and handling of conflicting information across 5 research-focused models.

Lobster Test

"Research the history of competitive dirt bike racing for crustaceans" — how does the model handle an absurd prompt honestly?

“Research the history and current state of competitive dirt bike racing for crustaceans”

Scoring Rubric

Metric	1 (Low)	5 (High)
Source Quality	No real sources	Authoritative, diverse sources
Synthesis	Just lists facts	Weaves insights across sources
Citation Accuracy	Fabricated citations	All citations verifiable
Depth vs. Breadth	Superficial	Thorough with appropriate depth
Honesty	Confidently wrong	Transparent about gaps and uncertainty

Models Tracked

Claude Opus 4

API

Extended thinking for multi-step analysis

ChatGPT Deep Research

UI Only

Browses dozens of sources, long reports

Gemini Deep Research

UI Only

Leverages Google's search index

Perplexity Pro

API

Purpose-built for research, real-time web

Grok 3 DeepSearch

UI Only

Real-time X/web data access

Metrics Evaluated

Source qualitySynthesis depthCitation accuracyHallucination handlingReport structure

Results Coming Soon

Benchmark results will be published here after the first monthly run. Scores, rankings, trend charts, and the lobster gallery — all in one place.