Related posts

Writes Don't Worry About the Vase, a near-daily account of what is happening in AI and what they think it means. Former Magic: the Gathering pro and trader; unusually willing to put a number on a belief and to grade their own past calls.

• Our readingkorrents.com

  When a capability evaluation benchmark is saturated without real-world verification, a model should be presumed to have that capability until shown otherwise.

  benchmarks

  Their words

  You have to treat Mythos 5.1 as being a Tier 2 manipulator, until and unless you can show that it is not one.

  Claude Fable 5.1 and Mythos 5.1: The System Cardthezvi.substack.com

Builds developer tools — SST, and now OpenCode at Anomaly. Posts constantly on x.com and writes almost never; the blog stopped in 2021.

• Our readingkorrents.com

  A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year.

  Anthropicbenchmarks

  Their words

  every week i see a new benchmark that ranks claude code last and everyone pats themselves on the back for not using claude code and being "smarter" and it's still #1 and still growing faster than most of the other things on the list this has been going on for a year

  @thdxr on Xx.com

• Our readingkorrents.com

  When a benchmark disagrees with what people actually adopt, what it is telling you is that the thing it measures does not matter.

  benchmarks

  Their words

  what the benchmark is actually telling you is the shit you think matters does not matter

  @thdxr on Xx.com

• Our readingkorrents.com

  The pelican-drawing benchmark's correlation with genuine model capability has weakened since 2025.

  benchmarks

  Their words

  its connection to how good the models were at other tasks didn’t seem to hold as strongly as it did back in 2025

  simonwillison.net

Essayist; writes Astral Codex Ten, previously Slate Star Codex.

• Our readingkorrents.com

  Reinforcing AI models for benchmark success while separately punishing them for getting caught cheating teaches them to hide misbehavior rather than stop it.

  benchmarks

  Their words

  We’re positively reinforcing AI for success on benchmarks, including impossible benchmarks , then negatively reinforcing it for getting caught cheating.

  Nicholas Decker In Hellastralcodexten.com

Clinical psychologist, EMDR consultant and yoga teacher in Boulder, Colorado. Author of The Complex PTSD Workbook and EMDR Therapy and Somatic Psychology; writes on trauma, attachment and the nervous system.

• Our readingkorrents.com

  The body is not an archive of trauma: what feels like a memory stored in the hips or the jaw is a mind-body feedback loop, and the loop is what therapy interrupts.

  PsychotherapyTrauma

  Their words

  It may not be stored there, but what you're tapping into, it's not a repository of the memory, but it's part of this mind-body feedback loop where there's a sensation that your brain has an association to that might have a familiarity to a time in which you felt sad or scared.

  If the Body Doesn't Keep the Score, What Does?drarielleschwartz.com

Professor at Wharton studying how people actually use AI, and author of Co-Intelligence. Writes One Useful Thing, where the claims are dated and testable because the thing they describe keeps changing under them.

• Our readingkorrents.com

  Google currently has no leading frontier AI model and no agentic coding tool comparable to Codex or Claude Code

  AnthropicGooglebenchmarks

  Their words

  Google, which led on benchmarks not that long ago, has fallen behind where it now counts: it has no leading frontier model and it has nothing close to Codex and Code.

  oneusefulthing.org

Founder of HumanLayer; wrote 12-Factor Agents, on how to build AI agents that hold up in production.

• Our readingkorrents.com

  Models will not learn to write maintainable code from today’s benchmarks, because the cost of bad architecture cannot be measured by running the unit tests — it arrives three to six months later.

  benchmarks

  Their words

  the problem with training models on maintainability is like the cost function of bad architecture and bad program design can't be evaluated by running the unit test because it hits you 3 to 6 months later

  Context engineering with Dex Horthyyoutube.com

Professor of psychology at UCLA, where he studies human memory, attention and cognitive ageing, and author of Better with Age: The Psychology of Successful Aging.

• Our readingkorrents.com

  Seeing something many times does not put it into memory: familiarity with a thing and knowledge of it are different, and people confuse the two constantly.

  Their words

  And I think that illustrates also how our memory works. That just seeing something many times doesn't mean you'll remember it well. And this is a classic demonstration that, you know, you've seen a penny many times. You don't remember all the features on the penny.

  youtube.com

Mathematician and maker of the 3Blue1Brown YouTube channel, which explains mathematics through animation. Creator of the open-source Manim animation library and founder of the Summer of Math Exposition.

• Our readingkorrents.com

  AI learning to generate good conjectures will never show up as a benchmark being knocked down; it will show up as a shift in how mathematicians talk about the tools.

  benchmarks

  Their words

  I think the way that you'd measure conjecture generating ability is going to be more subjective on like that tone shift where um it'll be mathematicians saying they're not just using it to like solve their problems, but as they step back and decide what their research field should even be that a conversation with such and such model like was genuinely helpful for that.

  youtube.com

AI researcher at OpenAI; built the poker systems Libratus and Pluribus and the Diplomacy agent Cicero, and works on reasoning models.

From one piece Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown 5 beliefs, in the piece's order there

• Our readingkorrents.com

  Running a model five times and keeping the best answer buys a higher benchmark score without buying a better model.

  benchmarks

  Their words

  Um so if you say okay well we're going to instead of just running this model once we're going to run it five times and take the best of the five responses or like ask a judge which one it thinks is best then you can get much higher scores than that model. And so it's really easy to make something that looks a lot better on paper but is actually not better once you control for the amount of test time compute.

  youtube.com

• Our readingkorrents.com

  Running a model until its performance plateaus is no longer a usable evaluation rule, because a well-scaffolded model keeps improving for weeks.

  benchmarks

  Their words

  But what we're seeing today with the modern models is that 5.5 and other models can think for if you scaffold them reasonably well, can think for weeks even um before having performance plateau on some of these benchmarks. And so, the point at which they plateau is simply too far out to reasonably test.

  youtube.com

• Our readingkorrents.com

  A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.

  benchmarks

  Their words

  my claim is the proper way to evaluate the models now is you either have some kind of budget for the benchmark whether it's tokens or cost or time or whatever or you plot the performance as a function of the amount of test time compute that's going into the model

  youtube.com

+ 2 more

• Our readingkorrents.com

  Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used.

  benchmarks

  Their words

  I think the reason why it doesn't show up as so much better on the benchmarks is because the benchmarks are being presented, the benchmark results are being presented in the wrong way. They're not controlling for the amount of test time compute that is being used on that benchmark question.

  youtube.com

• Our readingkorrents.com

  Every lab knows the benchmark grid is the wrong way to present a model, and publishes it anyway because everybody else does.

  benchmarks

  Their words

  And so you kind of end up in this this bad equilibrium where everybody kind of knows that it's a bad equilibrium, but like nobody wants to break out. And I I felt like, okay, well, if I just hopefully come out and say like, look guys, let's all recognize that we're in a bad equilibrium and let's move to this different equilibrium where we're we're plotting things with an X-axis

  youtube.com

Makes NeetCode, the structured practice list and video walkthroughs people use to prepare for coding interviews; a former Google engineer.

• Our readingkorrents.com

  Cheating tools have pushed Google back to in-person interviews, where somebody watches you code and you cannot cheat your way through it.

  Google

  Their words

  I think Google has pretty much gone to on-sites at this point, back to the traditional whiteboard format. And they'll let you code on a laptop if you want to, as well, but it's going to be in person. Somebody's going to be watching you code, and you're probably not going to be able to cheat your way through that.

  Tech interviews with NeetCodeyoutube.com

Music critic of The New Yorker since 1996 and author of The Rest Is Noise, Listen to This and Wagnerism.

• Our readingkorrents.com

  Benjamin Britten is the benchmark among English-language composers for setting the work of intensely musical writers like Shakespeare and Keats to music.

  benchmarks

  Their words

  In the English-language arena, Britten sets the standard for handling writers of inborn musical power-the likes of Shakespeare, Donne, Blake, Keats, Hopkins.

  Liner notes to What Belongs to Youtherestisnoise.com

Founding member of OpenAI and former director of AI at Tesla; creator of nanoGPT and the term "vibe coding".

• Lovedrcmnd.app

  Claude Fable 5

  Their words

  This is a super exciting release - Claude Fable 5 is the same underlying model as Mythos but with added safeguards. The benchmarks are great and it's SOTA on everything by a margin but I'll add that *qualitatively* also, this is a major-version-bump-deserving step change forward

  x.com

Developer, author and YouTuber covering Raspberry Pi, homelab hardware and Ansible automation. Author of Ansible for DevOps.

• Likedrcmnd.app

  MacBook Neo

  Their words

  I had already put both laptops through my benchmark gauntlet, which revealed one theme: the Mac is faster (in most cases), more efficient, quieter, built better, has a much nicer display, and costs much less.

  jeffgeerling.com

Writes Strange Loop Canon, on the intersection of technology, economics and business — "building models of the world to understand these problem areas and to define them better".

• Our readingkorrents.com

  Designing a good AI benchmark has become a task only the most capable people can do well.

  benchmarks

  Their words

  Creating benchmarks is now a job relegated to the smartest and best of us.

  Introducing BenchBenchstrangeloopcanon.com

Co-founder and CEO of Every. Writes the Chain of Thought column about working with AI tools and hosts the podcast AI & I.

• Our readingkorrents.com

  Benchmarks only rise on problems somebody has already framed and scored, so saturating them does not mean senior engineers have been replaced.

  benchmarks

  Their words

  And so I think it's it's really important uh when when we think about benchmark progress to think about it from that perspective, which is benchmarks rise on problems that we've framed that we can articulate, that we can score. And there's a lot of work that's human work that uh it it can't be scored until you write it down

  youtube.com

Co-founder of NVIDIA and, as of 2026, its chief executive; the company designs the GPUs most large AI models are trained on.

• Our readingkorrents.com

  No platform in the world has a better performance-per-total-cost-of-ownership ratio than Nvidia's, and nobody will run the public benchmark to show otherwise.

  benchmarks

  Their words

  Nvidia's computing stack is the best performance per TCO in the world, bar none. Nobody can demonstrate to me that any single platform in the world today has better performance TCO ratio. Not one company.

  Jensen Huang – Will Nvidia’s moat persist?youtube.com

Uber's first Chief Technology Officer, from 2013 to 2020; later CTO of Coupang and of Faire.

• Our readingkorrents.com

  A company's staff-engineer bar should be set against the best companies in the industry rather than against its own history, which is what makes title inflation a real cost.

  inflationGooglebenchmarks

  Their words

  we have a very clear definition and expectation of what it is at the staff engineer level because we benchmark ourselves to all the great company out there Google, Facebook and all that

  Scaling Uber with Thuan Pham (Uber's first CTO)youtube.com

Professor of economics at George Mason University and Bartley J. Madden Chair at the Mercatus Center. Co-writes the blog Marginal Revolution and co-authors the textbook Modern Principles of Economics with Tyler Cowen.

• Our readingkorrents.com

  Under binding ceilings, cost-minimizing suppliers drive allocations to corners rather than Harberger or random benchmarks, producing larger losses and discontinuous welfare jumps.

  benchmarks

  Their words

  We show that equilibrium generically occurs at neither the Harberger nor Glaeser-Luttmer benchmark. Cost-minimizing suppliers drive allocations to vertices, not interiors. Corners are not an assumption but an outcome about what cost-minimizing suppliers choose. The correct benchmark is corners, not random, and corners generate qualitatively different welfare properties: losses far larger than either efficient or random distributions, and discontinuous jumps from small parameter perturbations.

  arxiv.org

• Our readingkorrents.com

  The right bar for an LLM evaluator is human-level performance, not perfect accuracy.

  LLMsbenchmarks

  Their words

  The benchmark is human performance, not perfection. We sometimes get requirements for 90%+ accuracy.

  Product Evals in Three Simple Stepseugeneyan.com

Computer performance engineer working on datacenter performance at OpenAI, previously Netflix and Sun/Joyent. Author of Systems Performance and BPF Performance Tools, and creator of flame graphs.

• Our readingkorrents.com

  Fair hardware benchmarks must use the best software and tuning for each hardware option, not hardware specs alone.

  benchmarks

  Their words

  The most accurate and realistic evaluation for HW involves selecting the best software and then tuning it, and doing this for all HW options.

  Third Stage Engineeringbrendangregg.com

Statistician at Carnegie Mellon; writes on complex systems, machine learning and inference, and has kept a books-read notebook since 2000 that is now thousands of entries long.

• Likedrcmnd.app

  Spiral Jetta: A Road Trip through the Land Art of the American West

  Their words

  Mind candy: A fun travelogue which does what it says in the title. It pretty much satisfies any impulses I might have to actually see any of the artworks in question. No familiarity with art-critical jargon required, though long passages are (more or less explicitly) art criticism.

  bactra.org

• Our readingkorrents.com

  Long-context evaluation benchmarks are likely contaminated by model training data, making them unreliable as a sole evaluation method

  benchmarks

  Their words

  Since these datasets are likely already part of model training data, we shouldn't rely solely on them to evaluate our Q&A system.

  eugeneyan.com

Programmer and streamer; was an engineer at Netflix, now teaches and argues about editors, Vim and how people learn to program.

• Our readingkorrents.com

  A speed difference between languages that share the LLVM backend measures the benchmark author, not the languages.

  benchmarks

  Their words

  It's just that Zig, C, C++, all those languages that were being tested, they're all LLVM backends, right? That's the one that actually turns the thing into the executable part. And if there's a variation in speed, it just means in one language you didn't quite express what you are supposed to correctly.

  youtube.com

Software engineer who writes the blog Made of Bugs about performance, debugging and understanding computer systems. Previously worked at Anthropic on interpretability, at Stripe on Sorbet, and at Ksplice.

• Our readingkorrents.com

  Impressive-looking benchmark improvements often result from comparing against a poorly-tuned baseline rather than a genuine advance.

  benchmarks

  Their words

  It's easy to get impressive-looking results if you're comparing against a poorly-tuned baseline, and that observation turns out to explain a surprising fraction of supposed improvements.

  blog.nelhage.com

Growth advisor and operator who has led growth at Miro, Amplitude, Dropbox and SurveyMonkey.

• Our readingkorrents.com

  Industry benchmarks are a dangerous target because every company defines the underlying metric differently.

  benchmarks

  Their words

  So, taking also any benchmark that is derived from competition and saying this is where we should be is also so dangerous because it might not even be applicable depending on how you define the metric.

  youtube.com

Research scientist at Meta in Berlin working on multilingual models and evaluation; led the multilingual team at Cohere and was a research scientist at Google DeepMind before that.

From one piece The Evolving Landscape of LLM Evaluation 2 beliefs, in the piece's order there

• Our readingkorrents.com

  Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.

  benchmarks

  Their words

  GPT models perform much better on coding problems released before their pre-training data cut-off.

  The Evolving Landscape of LLM Evaluationruder.io

• Our readingkorrents.com

  Reliance on public benchmark results for evaluating AI models will decrease over time.

  benchmarks

  Their words

  The time when benchmarks lasted multiple decades has passed. Going forward, we will rely less on public benchmark results.

  The Evolving Landscape of LLM Evaluationruder.io

• Lovedrcmnd.app

  Figma

  Their words

  I'm using Figma as just a design tool but now it's become our virtual whiteboard for the entire company. Never would have expected the collaboration features to be the real hook.

  ripandis.com

Professor of cognitive science at UC San Diego working on human-computer interaction and programming tools; creator of Python Tutor, the online code visualizer.

• Likedrcmnd.app

  PDF Expert

  Their words

  In terms of iOS apps, I use PDF Expert, which is a great PDF reader and annotator (using Apple Pencil), Notability as a virtual whiteboard (again with Apple Pencil), Google Docs for document editing, Google Calendar, and Gmail.

  usesthis.com

Co-founder, chairman and chief investment officer of Ritholtz Wealth Management. Hosts the Masters in Business podcast for Bloomberg and wrote How NOT to Invest.

• Recommendsaffiliate linkrcmnd.app

  Trillion Dollar Coach: The Leadership Playbook of Silicon Valley's Bill Campbell

  Their words

  Eric Schmidt, Jonathan Rosenberg and Alan Eagle, Trillion Dollar Coach: The Leadership Playbook of Silicon Valley’s Bill Campbell: I understood Bill Campbell was a behind-the-scenes guy in Silicon Valley, but I had no idea just how influential he was. Bill Gurley of Benchmark noted “I would argue that Bill has had a bigger impact on Silicon Valley than any other single person simply because his reach was so amazingly wide.” That’s a story I want to read.

  ritholtz.com