Writes Strange Loop Canon, on the intersection of technology, economics and business — "building models of the world to understand these problem areas and to define them better".

• Our readingkorrents.com

  Designing a good AI benchmark has become a task only the most capable people can do well.

  benchmarks

  Their words

  Creating benchmarks is now a job relegated to the smartest and best of us.

  Introducing BenchBenchstrangeloopcanon.com

Co-founder and CEO of Every. Writes the Chain of Thought column about working with AI tools and hosts the podcast AI & I.

• Our readingkorrents.com

  Benchmarks only rise on problems somebody has already framed and scored, so saturating them does not mean senior engineers have been replaced.

  benchmarks

  Their words

  And so I think it's it's really important uh when when we think about benchmark progress to think about it from that perspective, which is benchmarks rise on problems that we've framed that we can articulate, that we can score. And there's a lot of work that's human work that uh it it can't be scored until you write it down

  youtube.com

Co-founder of NVIDIA and, as of 2026, its chief executive; the company designs the GPUs most large AI models are trained on.

• Our readingkorrents.com

  No platform in the world has a better performance-per-total-cost-of-ownership ratio than Nvidia's, and nobody will run the public benchmark to show otherwise.

  benchmarks

  Their words

  Nvidia's computing stack is the best performance per TCO in the world, bar none. Nobody can demonstrate to me that any single platform in the world today has better performance TCO ratio. Not one company.

  Jensen Huang – Will Nvidia’s moat persist?youtube.com

Uber's first Chief Technology Officer, from 2013 to 2020; later CTO of Coupang and of Faire.

• Our readingkorrents.com

  A company's staff-engineer bar should be set against the best companies in the industry rather than against its own history, which is what makes title inflation a real cost.

  inflationGooglebenchmarks

  Their words

  we have a very clear definition and expectation of what it is at the staff engineer level because we benchmark ourselves to all the great company out there Google, Facebook and all that

  Scaling Uber with Thuan Pham (Uber's first CTO)youtube.com

Professor of economics at George Mason University and Bartley J. Madden Chair at the Mercatus Center. Co-writes the blog Marginal Revolution and co-authors the textbook Modern Principles of Economics with Tyler Cowen.

• Our readingkorrents.com

  Under binding ceilings, cost-minimizing suppliers drive allocations to corners rather than Harberger or random benchmarks, producing larger losses and discontinuous welfare jumps.

  benchmarks

  Their words

  We show that equilibrium generically occurs at neither the Harberger nor Glaeser-Luttmer benchmark. Cost-minimizing suppliers drive allocations to vertices, not interiors. Corners are not an assumption but an outcome about what cost-minimizing suppliers choose. The correct benchmark is corners, not random, and corners generate qualitatively different welfare properties: losses far larger than either efficient or random distributions, and discontinuous jumps from small parameter perturbations.

  arxiv.org

Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.

• Our readingkorrents.com

  The right bar for an LLM evaluator is human-level performance, not perfect accuracy.

  LLMsbenchmarks

  Their words

  The benchmark is human performance, not perfection. We sometimes get requirements for 90%+ accuracy.

  Product Evals in Three Simple Stepseugeneyan.com

Computer performance engineer working on datacenter performance at OpenAI, previously Netflix and Sun/Joyent. Author of Systems Performance and BPF Performance Tools, and creator of flame graphs.

• Our readingkorrents.com

  Fair hardware benchmarks must use the best software and tuning for each hardware option, not hardware specs alone.

  benchmarks

  Their words

  The most accurate and realistic evaluation for HW involves selecting the best software and then tuning it, and doing this for all HW options.

  Third Stage Engineeringbrendangregg.com

Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.

• Our readingkorrents.com

  Long-context evaluation benchmarks are likely contaminated by model training data, making them unreliable as a sole evaluation method

  benchmarks

  Their words

  Since these datasets are likely already part of model training data, we shouldn't rely solely on them to evaluate our Q&A system.

  eugeneyan.com

Programmer and streamer; was an engineer at Netflix, now teaches and argues about editors, Vim and how people learn to program.

• Our readingkorrents.com

  A speed difference between languages that share the LLVM backend measures the benchmark author, not the languages.

  benchmarks

  Their words

  It's just that Zig, C, C++, all those languages that were being tested, they're all LLVM backends, right? That's the one that actually turns the thing into the executable part. And if there's a variation in speed, it just means in one language you didn't quite express what you are supposed to correctly.

  youtube.com

Software engineer who writes the blog Made of Bugs about performance, debugging and understanding computer systems. Previously worked at Anthropic on interpretability, at Stripe on Sorbet, and at Ksplice.

• Our readingkorrents.com

  Impressive-looking benchmark improvements often result from comparing against a poorly-tuned baseline rather than a genuine advance.

  benchmarks

  Their words

  It's easy to get impressive-looking results if you're comparing against a poorly-tuned baseline, and that observation turns out to explain a surprising fraction of supposed improvements.

  blog.nelhage.com

Growth advisor and operator who has led growth at Miro, Amplitude, Dropbox and SurveyMonkey.

• Our readingkorrents.com

  Industry benchmarks are a dangerous target because every company defines the underlying metric differently.

  benchmarks

  Their words

  So, taking also any benchmark that is derived from competition and saying this is where we should be is also so dangerous because it might not even be applicable depending on how you define the metric.

  youtube.com

Research scientist at Meta in Berlin working on multilingual models and evaluation; led the multilingual team at Cohere and was a research scientist at Google DeepMind before that.

From one piece The Evolving Landscape of LLM Evaluation 2 beliefs, in the piece's order there

• Our readingkorrents.com

  Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.

  benchmarks

  Their words

  GPT models perform much better on coding problems released before their pre-training data cut-off.

  The Evolving Landscape of LLM Evaluationruder.io

• Our readingkorrents.com

  Reliance on public benchmark results for evaluating AI models will decrease over time.

  benchmarks

  Their words

  The time when benchmarks lasted multiple decades has passed. Going forward, we will rely less on public benchmark results.

  The Evolving Landscape of LLM Evaluationruder.io

Co-founder, chairman and chief investment officer of Ritholtz Wealth Management. Hosts the Masters in Business podcast for Bloomberg and wrote How NOT to Invest.

• Recommendsaffiliate linkrcmnd.app

  Trillion Dollar Coach: The Leadership Playbook of Silicon Valley's Bill Campbell

  Their words

  Eric Schmidt, Jonathan Rosenberg and Alan Eagle, Trillion Dollar Coach: The Leadership Playbook of Silicon Valley’s Bill Campbell: I understood Bill Campbell was a behind-the-scenes guy in Silicon Valley, but I had no idea just how influential he was. Bill Gurley of Benchmark noted “I would argue that Bill has had a bigger impact on Silicon Valley than any other single person simply because his reach was so amazingly wide.” That’s a story I want to read.

  ritholtz.com