Search

Said and published

• Scott Alexander Korrent 1 Sept Reinforcing AI models for benchmark success while separately punishing them for getting caught cheating teaches them to hide misbehavior rather than stop it. We’re positively reinforcing AI for success on benchmarks, including impossible benchmarks , then negatively reinforcing it for getting caught cheating.
• Dax Raad x.com 7 Sept there are many things that make an agent more pleasant to use that make them worse on benchmarks
• Daniel Lemire GitHub 22 Aug gobitmapbenchmark — A Go project to benchmark various bitmap implementations (this is not a library!)
• Daniel Lemire GitHub 22 Aug simdip — IP parsing benchmarks
• Kun Chen x.com 4 Sept one thing the recent model releases have consistently told us is that public benchmarks are completely useless now they reflect how well the models are trained to solve benchmark-like problems, but are far from how we…
• Dan Luu Bluesky 4 Jul Agentic test processes, LLM benchmarks, and other notes on agentic coding from Galapagos Island: danluu.com/ai-coding/
• Dan Luu Mastodon 3 Jul Agentic test processes, LLM benchmarks, and other notes on agentic coding from Galapagos Island: https://danluu.com/ai-coding/
• Mitchell Hashimoto x.com 25 Aug OSC parsing in libghostty was our last non-SIMD optimized VT stream path. OSCs are usually small so it just never showed up on benchmarks. But now that we are potentially passing megabytes of data (Kitty clipboard), it…
• Daniel Lemire GitHub 11 Aug simdsearch — Benchmarks and reference kernels for SIMD substring search.
• Matt Birchler YouTube 30 Mar How useful are MacBook Neo benchmarks?
• Sebastian Raschka x.com 15 Jul Interesting surprise drop from Thinky! The Inkling model looks pretty solid on benchmarks, and it has some little surprises in its architecture: - Small conv layers in several places - An RMSNorm for the embeddings…
• Ethan Mollick Korrent 23 Jul Google currently has no leading frontier AI model and no agentic coding tool comparable to Codex or Claude Code Google, which led on benchmarks not that long ago, has fallen behind where it now counts: it has no leading frontier model and it has nothing close to Codex and Code.
• Dan Shipper Korrent 24 May Benchmarks only rise on problems somebody has already framed and scored, so saturating them does not mean senior engineers have been replaced. …about benchmark progress to think about it from that perspective, which is benchmarks rise on problems that we've framed that we can articulate, that we can score. And there's a lot of work that's human work that uh…
• Eugene Yan x.com 25 Jun How do we eval if a model can find and exploit vulnerabilities? We discuss some benchmarks and the common pattern: • A sandboxed target within Docker containers • Inputs: code only (0-day), with patch (1-day scenario) •…
• Russ Cox GitHub 27 May 2025 omap — Ordered map
• Sebastian Raschka Newsletter 5 Oct 2025 Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)
• Eugene Yan Site 22 Jun 2025 Evaluating Long-Context Question & Answer Systems
• Sebastian Raschka Newsletter 30 Dec 2025 The State Of LLMs 2025: Progress, Problems, and Predictions
• Eugene Yan Bluesky 25 Jun 2025 …to eval on • How to build llm-evaluators • How to build eval datasets • Benchmarks: narratives, technical docs, multi-docs…
• Andrej Karpathy x.com 9 Jun This is a super exciting release - Claude Fable 5 is the same underlying model as Mythos but with added safeguards. The benchmarks are great and it's SOTA on everything by a margin but I'll add that *qualitatively*…