Ethan Mollick argues that despite measurable progress in AI, standard benchmarks fail to capture what really matters: performance on YOUR specific tasks with YOUR judgment criteria. He prescribes "interviewing" AI models like job candidates rather than relying on generic test scores.
Problems with standard benchmarks
Benchmarks like MMLU-Pro ask obscure questions ("average cranial capacity of Homo erectus?", "title of Cheap Trick's 1979 live album?") whose real measurement value is unclear. The tests are not calibrated (the difficulty of going from 84% to 85% versus 40% to 41% is unknown), contain errors, and top scores can be unreachable. Worse: published answer keys allow their incorporation into training (accidentally or deliberately). Collectively, all benchmarks (AIME, GPQA, MMLU, SWE-bench, ARC-AGI, METR) trend "up and to the right," measuring an underlying capability factor correlated with real-world impact. But their focus on math, science, reasoning, and code leaves gaps in writing, business advice, and empathy. "What you actually care about is which model would be best for YOUR needs."
"Vibes-based" benchmarking: an individual approach
Practitioners develop idiosyncratic tests: Simon Willison asks for a pelican on a bike, Mollick an otter on a plane, a spaceship control panel in JavaScript, difficult poems, video games. A writing exercise reveals patterns: Claude 4.5 Sonnet is solid at writing, Gemini 2.5 Pro (currently the weakest) can't keep a word count, GPT-5 Thinking is an exuberant but sometimes incoherent stylist, Kimi K2 Thinking produces interesting turns of phrase but a story that makes no sense. Vibes give a "feel" for models, but remain idiosyncratic: answers vary each time and one relies on impressions rather than real measurements.
Real-world benchmarking: the GDPval method
OpenAI's GDPval paper demonstrates a rigorous approach: (1) experts with 14 years of experience on average create complex, realistic projects representing 4 to 7 hours of human work, (2) several AI models and human experts perform the tasks, (3) a third group of experts grades blind, spending more than an hour per question. The study reveals AI's strengths (software development, financial advising: it beats humans) and its weaknesses (pharmacists, industrial engineers, real estate agents beat AI). Differences between models emerge: ChatGPT is a better sales manager, Claude a better financial advisor. The shape of the "Jagged Frontier" takes form.
GuacaDrone reveals model personality
Mollick pitches a dubious idea, "guacamole delivery by drone," and asks models to rate its viability from 1 to 10, ten times each. Grok is enthusiastic, Copilot excited, GPT-5 and Claude skeptical. Mollick himself would rate it 2 or lower. "Systematically rating ideas 3-4 points higher or lower is systematically steering you in a different direction." Depending on the company, one may want an AI that embraces or avoids risk: it is necessary to understand how AI "thinks" about critical questions.
Prescription for organizations
Vibes are enough for individuals. Organizations deploying at scale need systematic testing: AI on real work and real judgments, realistic scenarios, run multiple times, evaluated by experts, with head-to-head comparison on the tasks that matter. "The difference between 'the model scored 85% on MMLU' and 'the model is more accurate at financial analysis but more conservative on risk.'" To be redone several times a year as new models are released.
Final analogy: "You wouldn't hire a VP based solely on their SAT scores. Don't choose the AI that will advise on thousands of decisions based on its knowledge of the average cranial capacity of Homo erectus."