Text-to-Text
Evaluate language models on reasoning, writing, coding, instruction following, factual accuracy, long-context understanding, and multi-turn coherence across 8 leading LLMs.
Lobster Test
Standard benchmarks: MMLU, HumanEval, GSM8K, ARC, TruthfulQA, MT-Bench, plus custom VCL prompts.
“Write a short story about a lobster who enters a dirt bike race”
Scoring Rubric
| Metric | 1 (Low) | 5 (High) |
|---|---|---|
| Reasoning | Wrong answer, no logic | Correct, clear chain of thought |
| Instruction Following | Ignores constraints | Follows every constraint exactly |
| Creative Writing | Generic / boring | Vivid, original, engaging |
| Code Generation | Doesn't compile | Correct, efficient, idiomatic |
| Factual Accuracy | Multiple hallucinations | Fully accurate, verifiable |
| Long-Context | Loses thread | Coherent across full context |
Models Tracked
Claude Opus 4
APIAnthropic flagship
Claude Sonnet 4
APIAnthropic high-value tier
GPT-4o
APIOpenAI multimodal workhorse
GPT-o3
APIOpenAI reasoning model
Gemini 2.5 Pro
APIGoogle, 1M+ context
DeepSeek V3 / R1
APIR1 = reasoning variant; open-weight
Llama 4 Maverick
Open WeightMeta, via providers
Grok 3
APIxAI, reasoning capabilities
Metrics Evaluated
ReasoningCode generationInstruction followingHallucination rateLong-context