Professor at Wharton studying how people actually use AI, and author of Co-Intelligence. Writes One Useful Thing, where the claims are dated and testable because the thing they describe keeps changing under them.

• Our readingkorrents.com

  Google currently has no leading frontier AI model and no agentic coding tool comparable to Codex or Claude Code

  AnthropicGooglebenchmarks

  Their words

  Google, which led on benchmarks not that long ago, has fallen behind where it now counts: it has no leading frontier model and it has nothing close to Codex and Code.

  oneusefulthing.org

Founder of HumanLayer; wrote 12-Factor Agents, on how to build AI agents that hold up in production.

• Our readingkorrents.com

  Models will not learn to write maintainable code from today’s benchmarks, because the cost of bad architecture cannot be measured by running the unit tests — it arrives three to six months later.

  benchmarks

  Their words

  the problem with training models on maintainability is like the cost function of bad architecture and bad program design can't be evaluated by running the unit test because it hits you 3 to 6 months later

  Context engineering with Dex Horthyyoutube.com

Mathematician and maker of the 3Blue1Brown YouTube channel, which explains mathematics through animation. Creator of the open-source Manim animation library and founder of the Summer of Math Exposition.

• Our readingkorrents.com

  AI learning to generate good conjectures will never show up as a benchmark being knocked down; it will show up as a shift in how mathematicians talk about the tools.

  benchmarks

  Their words

  I think the way that you'd measure conjecture generating ability is going to be more subjective on like that tone shift where um it'll be mathematicians saying they're not just using it to like solve their problems, but as they step back and decide what their research field should even be that a conversation with such and such model like was genuinely helpful for that.

  youtube.com

AI researcher at OpenAI; built the poker systems Libratus and Pluribus and the Diplomacy agent Cicero, and works on reasoning models.

From one piece Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown 5 beliefs, in the piece's order there

• Our readingkorrents.com

  Running a model five times and keeping the best answer buys a higher benchmark score without buying a better model.

  benchmarks

  Their words

  Um so if you say okay well we're going to instead of just running this model once we're going to run it five times and take the best of the five responses or like ask a judge which one it thinks is best then you can get much higher scores than that model. And so it's really easy to make something that looks a lot better on paper but is actually not better once you control for the amount of test time compute.

  youtube.com

• Our readingkorrents.com

  Running a model until its performance plateaus is no longer a usable evaluation rule, because a well-scaffolded model keeps improving for weeks.

  benchmarks

  Their words

  But what we're seeing today with the modern models is that 5.5 and other models can think for if you scaffold them reasonably well, can think for weeks even um before having performance plateau on some of these benchmarks. And so, the point at which they plateau is simply too far out to reasonably test.

  youtube.com

• Our readingkorrents.com

  A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.

  benchmarks

  Their words

  my claim is the proper way to evaluate the models now is you either have some kind of budget for the benchmark whether it's tokens or cost or time or whatever or you plot the performance as a function of the amount of test time compute that's going into the model

  youtube.com

Music critic of The New Yorker since 1996 and author of The Rest Is Noise, Listen to This and Wagnerism.

• Our readingkorrents.com

  Benjamin Britten is the benchmark among English-language composers for setting the work of intensely musical writers like Shakespeare and Keats to music.

  benchmarks

  Their words

  In the English-language arena, Britten sets the standard for handling writers of inborn musical power-the likes of Shakespeare, Donne, Blake, Keats, Hopkins.

  Liner notes to What Belongs to Youtherestisnoise.com

Founding member of OpenAI and former director of AI at Tesla; creator of nanoGPT and the term "vibe coding".

• Lovedrcmnd.app

  Claude Fable 5

  Their words

  This is a super exciting release - Claude Fable 5 is the same underlying model as Mythos but with added safeguards. The benchmarks are great and it's SOTA on everything by a margin but I'll add that *qualitatively* also, this is a major-version-bump-deserving step change forward

  x.com

Developer, author and YouTuber covering Raspberry Pi, homelab hardware and Ansible automation. Author of Ansible for DevOps.

• Likedrcmnd.app

  MacBook Neo

  Their words

  I had already put both laptops through my benchmark gauntlet, which revealed one theme: the Mac is faster (in most cases), more efficient, quieter, built better, has a much nicer display, and costs much less.

  jeffgeerling.com

Writes Strange Loop Canon, on the intersection of technology, economics and business — "building models of the world to understand these problem areas and to define them better".

• Our readingkorrents.com

  Designing a good AI benchmark has become a task only the most capable people can do well.

  benchmarks

  Their words

  Creating benchmarks is now a job relegated to the smartest and best of us.

  Introducing BenchBenchstrangeloopcanon.com

Co-founder and CEO of Every. Writes the Chain of Thought column about working with AI tools and hosts the podcast AI & I.

• Our readingkorrents.com

  Benchmarks only rise on problems somebody has already framed and scored, so saturating them does not mean senior engineers have been replaced.

  benchmarks

  Their words

  And so I think it's it's really important uh when when we think about benchmark progress to think about it from that perspective, which is benchmarks rise on problems that we've framed that we can articulate, that we can score. And there's a lot of work that's human work that uh it it can't be scored until you write it down

  youtube.com

Co-founder of NVIDIA and, as of 2026, its chief executive; the company designs the GPUs most large AI models are trained on.

• Our readingkorrents.com

  No platform in the world has a better performance-per-total-cost-of-ownership ratio than Nvidia's, and nobody will run the public benchmark to show otherwise.

  benchmarks

  Their words

  Nvidia's computing stack is the best performance per TCO in the world, bar none. Nobody can demonstrate to me that any single platform in the world today has better performance TCO ratio. Not one company.

  Jensen Huang – Will Nvidia’s moat persist?youtube.com

Uber's first Chief Technology Officer, from 2013 to 2020; later CTO of Coupang and of Faire.

• Our readingkorrents.com

  A company's staff-engineer bar should be set against the best companies in the industry rather than against its own history, which is what makes title inflation a real cost.

  inflationGooglebenchmarks

  Their words

  we have a very clear definition and expectation of what it is at the staff engineer level because we benchmark ourselves to all the great company out there Google, Facebook and all that

  Scaling Uber with Thuan Pham (Uber's first CTO)youtube.com

Professor of economics at George Mason University and Bartley J. Madden Chair at the Mercatus Center. Co-writes the blog Marginal Revolution and co-authors the textbook Modern Principles of Economics with Tyler Cowen.

• Our readingkorrents.com

  Under binding ceilings, cost-minimizing suppliers drive allocations to corners rather than Harberger or random benchmarks, producing larger losses and discontinuous welfare jumps.

  benchmarks

  Their words

  We show that equilibrium generically occurs at neither the Harberger nor Glaeser-Luttmer benchmark. Cost-minimizing suppliers drive allocations to vertices, not interiors. Corners are not an assumption but an outcome about what cost-minimizing suppliers choose. The correct benchmark is corners, not random, and corners generate qualitatively different welfare properties: losses far larger than either efficient or random distributions, and discontinuous jumps from small parameter perturbations.

  arxiv.org

Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.

• Our readingkorrents.com

  The right bar for an LLM evaluator is human-level performance, not perfect accuracy.

  LLMsbenchmarks

  Their words

  The benchmark is human performance, not perfection. We sometimes get requirements for 90%+ accuracy.

  Product Evals in Three Simple Stepseugeneyan.com

Computer performance engineer working on datacenter performance at OpenAI, previously Netflix and Sun/Joyent. Author of Systems Performance and BPF Performance Tools, and creator of flame graphs.

• Our readingkorrents.com

  Fair hardware benchmarks must use the best software and tuning for each hardware option, not hardware specs alone.

  benchmarks

  Their words

  The most accurate and realistic evaluation for HW involves selecting the best software and then tuning it, and doing this for all HW options.

  Third Stage Engineeringbrendangregg.com

Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.

• Our readingkorrents.com

  Long-context evaluation benchmarks are likely contaminated by model training data, making them unreliable as a sole evaluation method

  benchmarks

  Their words

  Since these datasets are likely already part of model training data, we shouldn't rely solely on them to evaluate our Q&A system.

  eugeneyan.com

Programmer and streamer; was an engineer at Netflix, now teaches and argues about editors, Vim and how people learn to program.

• Our readingkorrents.com

  A speed difference between languages that share the LLVM backend measures the benchmark author, not the languages.

  benchmarks

  Their words

  It's just that Zig, C, C++, all those languages that were being tested, they're all LLVM backends, right? That's the one that actually turns the thing into the executable part. And if there's a variation in speed, it just means in one language you didn't quite express what you are supposed to correctly.

  youtube.com

Software engineer who writes the blog Made of Bugs about performance, debugging and understanding computer systems. Previously worked at Anthropic on interpretability, at Stripe on Sorbet, and at Ksplice.

• Our readingkorrents.com

  Impressive-looking benchmark improvements often result from comparing against a poorly-tuned baseline rather than a genuine advance.

  benchmarks

  Their words

  It's easy to get impressive-looking results if you're comparing against a poorly-tuned baseline, and that observation turns out to explain a surprising fraction of supposed improvements.

  blog.nelhage.com

Growth advisor and operator who has led growth at Miro, Amplitude, Dropbox and SurveyMonkey.

• Our readingkorrents.com

  Industry benchmarks are a dangerous target because every company defines the underlying metric differently.

  benchmarks

  Their words

  So, taking also any benchmark that is derived from competition and saying this is where we should be is also so dangerous because it might not even be applicable depending on how you define the metric.

  youtube.com

Research scientist at Meta in Berlin working on multilingual models and evaluation; led the multilingual team at Cohere and was a research scientist at Google DeepMind before that.

From one piece The Evolving Landscape of LLM Evaluation 2 beliefs, in the piece's order there

• Our readingkorrents.com

  Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.

  benchmarks

  Their words

  GPT models perform much better on coding problems released before their pre-training data cut-off.

  The Evolving Landscape of LLM Evaluationruder.io

• Our readingkorrents.com

  Reliance on public benchmark results for evaluating AI models will decrease over time.

  benchmarks

  Their words

  The time when benchmarks lasted multiple decades has passed. Going forward, we will rely less on public benchmark results.

  The Evolving Landscape of LLM Evaluationruder.io

Co-founder, chairman and chief investment officer of Ritholtz Wealth Management. Hosts the Masters in Business podcast for Bloomberg and wrote How NOT to Invest.

• Recommendsaffiliate linkrcmnd.app

  Trillion Dollar Coach: The Leadership Playbook of Silicon Valley's Bill Campbell

  Their words

  Eric Schmidt, Jonathan Rosenberg and Alan Eagle, Trillion Dollar Coach: The Leadership Playbook of Silicon Valley’s Bill Campbell: I understood Bill Campbell was a behind-the-scenes guy in Silicon Valley, but I had no idea just how influential he was. Bill Gurley of Benchmark noted “I would argue that Bill has had a bigger impact on Silicon Valley than any other single person simply because his reach was so amazingly wide.” That’s a story I want to read.

  ritholtz.com