Related posts
distilled sessions evals
The the words it uses that this site has seen
least often elsewhere. Posts are matched on those words alone —
nothing here is a summary of this one.
Sources
All Writing Blog GitHub x.com Newsletter Korrents Mastodon Bluesky Site Recommends
Top people
Hamel Husain
Eugene Yan
Dan Luu
Peter Steinberger
Kun Chen
Teresa Torres
Mitchell Hashimoto
Simon Willison
Boris Cherny
Tobias Lütke
Sebastian Raschka
Tommy Wood
Showing Profile →
Show everything
Hiding Show them again
Show them again
Further back ↓
Hiding Show them again
19 September
Co-creator of Django and creator of Datasette; writes daily at simonwillison.net.
Former L8 engineer at Meta, Microsoft and Atlassian, now writing and building solo on agentic engineering. Writes Kun's Field Notes and posts a lot about AI coding agents on X.
Founded PSPDFKit in 2011 and ran it for a decade. Came back from a break to work on AI agents — the OpenClaw project, and OpenAI, joined in February 2026. Writes at steipete.me.
The coolest part: roboclaw runs our team server, is live on Discord, talks with gpt-live and knows about all the sessions it's juggling. We can ask it about context to current and past sessions right while we have meetings. Related
Love having @vhbrzezowski hijacking my sessions and desloping them before the PR lands. Also hot tip: Open the home sidebar and ask your claw to reorganize your sessions to keep your sessions organized. Related
Just landed a new feature, in the latest OC you can now ask your agent “Run this [web app] in crabbox and show me [vnc / a portal]” This was already working for cloud sessions, but often you start local and then later figure out you need a box to test, now it just works. Related
18 September
Machine learning engineer and independent AI consultant. He writes about LLM evaluation, tooling and applied ML at hamel.dev, and previously worked on machine learning at GitHub.
If you are a Data Scientist, you have never had more alpha than right now. Data Scientists email me all the time. It is no mistake that I'm focused on evals as a former DS. I talk more about this here: Related
17 September
Creator of Claude Code at Anthropic.
Projects have changed not only how I interact with Claude but how I code. I stopped managing sessions. I just send thoughts as they come, Claude splits them into threads, and the project remembers how I work. It's where I do a ton of my coding now. [screenshot: my actual prompts for claude code cli yesterday] Quoting @ClaudeDevs Today we're rolling out Projects in Claude Code on desktop and web. A project is one conversation with Claude. It splits the work into threads itself, runs them as parallel cloud sessions, passes context between them, and keeps going when you leave. In beta for select users. Anthropic Related
16 September
Former options market maker who writes Moontower, a newsletter on trading, volatility, decision-making and how to think about risk outside markets too.
Could use some input. So I want to do a zoom less for this family investing intro seminar but sitting there going through slides sounds terrible The sessions were in a classroom setting with lots of back and forth between everyone. There's definitely an arc but interaction... Related
Product discovery coach; wrote Continuous Discovery Habits and writes Product Talk.
15 September
Machine learning engineer and independent AI consultant. He writes about LLM evaluation, tooling and applied ML at hamel.dev, and previously worked on machine learning at GitHub.
Just updated our AI evals FAQ with 5 new questions, 48 questions & answers total! New FAQs just added: - Do I need a reference answer or rubric before annotating data? - How can I do evals when traces contain sensitive data? - How do you review a trace that is really large? - How much context should I give a LLM judge? - What should I do when my “gold” eval dataset becomes stale? It's all here: hamel.dev
LLMs Related
CEO of Vercel; creator of Next.js and Socket.IO. Writes at rauchg.com.
One of the coolest things about 𝚏𝚡 is that it auto-upgrades, and 𝚌𝚝𝚛𝚕+𝚐 restarts + resumes the chat. Also, that 0.0.10 is much, much faster for longer sessions 😊 Related
14 September
Co-founder of Superlogical, started in 2026 to build server-side terminal infrastructure; creator of Ghostty. Co-founded HashiCorp and created Vagrant and Terraform before that.
New week new demo! Today I want to showcase the CLI to control the @superlogical multiplexer. The CLI is an extremely powerful tool for automation and hooking into other tools like editors, agentic coding tools, and more. And, it's super fun! Everything you can do in the GUI, you can do via the CLI. Actually, you can do more in the CLI. You can create sessions, run commands, make splits, move things around, change focus, simulate any input device (key/mouse/etc.), wait on scripts, and more. There are a couple unique parts of Rex I'm showing off here too: our streaming event system that any cl… Related
13 September
Founded PSPDFKit in 2011 and ran it for a decade. Came back from a break to work on AI agents — the OpenClaw project, and OpenAI, joined in February 2026. Writes at steipete.me.
Fixed so many little perf issues that didn't matter much for single users... before we made it good for teams. OC stems ~80 sessions here rn now and doesn't sweat. Related
Moroccan senior frontend developer writing at smakosh.com; builds side projects and client work under Smakosh LLC.
New on @llmgateway: Adaptive Cache-Aware Provider Selection Routing learns cache-hit rates and output-to-input proportions from recent project/model usage to compare providers for large prompts and sessions. Available automatically across plans, with workload defaults before enough history exists and explicit overrides on Enterprise. Learn more llmgateway.io
Related
10 September
Former L8 engineer at Meta, Microsoft and Atlassian, now writing and building solo on agentic engineering. Writes Kun's Field Notes and posts a lot about AI coding agents on X.
more and more people are discovering the great combo of herdr + firstmate. it's such a good feeling that i wish more folks could discover sooner letting a single agent babysit all your sessions is so clearly the next level experience we're all heading towards Quoting @arthurcd btw, thank you @kunchenguid. FirstMate is very impressive! Related
9 September
Software engineer in Chicago; writes the cassidoo.co blog and a weekly developer newsletter.
I'm really excited about @fireworksAI_HQ new event: Forge. AI teams are owning the system: models, data, evals, infrastructure, and policies. So they created an event around elevating that. For builders, model shapers, systems engineers, and leaders building production AI. It's free on Nov 3 at Pier 27 in SF, application only. fandf.co
Related
Former L8 engineer at Meta, Microsoft and Atlassian, now writing and building solo on agentic engineering. Writes Kun's Field Notes and posts a lot about AI coding agents on X.
8 September
Co-founder of Superlogical, started in 2026 to build server-side terminal infrastructure; creator of Ghostty. Co-founded HashiCorp and created Vagrant and Terraform before that.
A quick look into remote persistent sessions with Superlogical! Not just remote sessions, but some functionality like remote directory lists, too. Superlogical can actually be a full SSH replacement (I don't run SSH on my servers anymore), and I talk about that briefly, too. Related
6 September
Co-founder and CEO of Shopify; long-time Linux user and Omarchy contributor.
Stats: 1572 sessions, longest 9603 turns, 0 amulets kinda disappointed in the X crowd right now 😁 Quoting @tobi I always liked nethack. This 42 year old marvel is one of the deepest adventure games ever made. It also looks like crap, at least out of the box. GPT6 Astra made a headless version for me. You can now use its brain as a library or as mcp or via javascript library. Meet Related
4 September
Machine learning engineer and independent AI consultant. He writes about LLM evaluation, tooling and applied ML at hamel.dev, and previously worked on machine learning at GitHub.
. @isaac_flath and I are going to see if the slop prompt is good and do some evals. (also first time trying to stream haha). Related
I love how Teresa is merging product discovery with evals. I think its a powerful combination! I started learning about product discovery b/c of Teresa and I think its a great skill for any engineer as it helps you focus on what to build. Highly recommend checking out her work. Quoting @ttorres AI evals have been the "it" skill for product teams for over a year. I've even called evals a new discovery habit. But I still meet product teams who only have a vague idea of what evals are. And it's not their fault. Most of the writing on this topic is intended for engineers or just isn't specifi… Related
3 September
Co-founder and CEO of Linear; previously principal designer at Airbnb, where he led the Design Language System, and founding designer at Coinbase.
Thanks @jasonlk! Lot to do still but: 1) @linear is used by pretty much by all major labs/ai companies out there 2) our MCP traffic grows 50% MoM 3) we have the most agent integrations and active usage than any platform like this (3rd party, in-house agents) 4) our AI revenue (loops, coding sessions) has grown about 500% per quarter 5) we are closing several public companies/enterprises each quarter to help them coordinate their agent work Quoting @HarryStebbings The Bull Case for Linear Being a $100BN Company: "Linear is the clear winner. They have built an agentic product first that allows us to build 100x more software, and that means 100x more features than ever before. Humans cannot keep up with it, and humans still have to work with agents. If every h… MCP Related
2 September
Co-founder of Superlogical, started in 2026 to build server-side terminal infrastructure; creator of Ghostty. Co-founded HashiCorp and created Vagrant and Terraform before that.
The Superlogical server is built to run anywhere and everywhere and serve a level of scale (terminals+clients) unmatched w/ prior solutions. So let's start talking about that, starting with memory usage and comparisons to other popular multiplexers. I want to start by talking high level: why does terminal/client scale matter? This matters because we're seeing unprecedented terminal usage and client attachment mainly due to AI. They like to spawn more sessions than we've ever seen, and they are all their own clients. So we have to optimize it. Even if you don't use AI/agents, this just makes t… Related
Product discovery coach; wrote Continuous Discovery Habits and writes Product Talk.
1 September
Co-founder and CEO of Shopify; long-time Linux user and Omarchy contributor.
Technical staff at METR, where she works on threat modelling and risk assessment for loss-of-control risks from advanced AI.
Their words
But from their perspective, they've just been trained for millions of subjective years to do as well as they possibly can on these evals. In many cases, the only way in which they've been able to perform well on that training is explicitly by cheating, right?
Show the whole quote
youtube.com
31 August
Machine learning engineer and independent AI consultant. He writes about LLM evaluation, tooling and applied ML at hamel.dev, and previously worked on machine learning at GitHub.
It's fun posting about evals b/c I can reply like this to bots Related
28 August
Investor and writer. Previously a partner at Andreessen Horowitz and a product leader at Twitter, Facebook, Snap and Microsoft. Writes essays and memos at sriramk.com.
really impressed with the quality of work from @RyanGreenblatt , @METR_Evals and @OpenAI to investigate the HuggingFace incident. Both reports are recommended reading. OpenAI HuggingFace Related
26 August
American software engineer; worked on code-server at Coder and teaches TypeScript and open-source contribution. Publishes at joeprevite.com.
today is the day you stop doomscrolling stay with me - this will take 5mins 1. open the AppStore 2. download "ScreenZen" (free) 3. configure limits (e.g. 4 15-min sessions) 4. start with 1 social media app 5. a few days later, add another 6. repeat until you've added all apps too busy right now? add a calendar event with this tweet and do it this weekend trust me, you'll thank me next month Related
20 August
Writer and developer. He runs the Latent Space newsletter and podcast about AI engineering, and writes essays on software and careers at swyx.io.
by total views, matt’s now the top @aidotengineer speaker in our brief history, and arguably one of the best skills experts in the world — his /grill-me has reached all echelons up to @satyanadella (with some variations…) /wayfinder is @mattpocockuk’s /grill-me for /grill-me, when you are just navigating the fog of war and don’t yet know what you dont yet know, and need to orchestrate research and other grill sessions to get there. super glad to have @ricmac help launch our new Skills coverage with his course launch this week! 👇exclusive interview, quick read Quoting @latentspacepod We chat to @mattpocockuk about his /wayfinder skill, which he designed for the "fog of war" — when you need to figure out a project but the end state isn’t entirely clear. This is the first in a series of skills we'll be exploring in the coming weeks. Related
16 August
Machine learning engineer and independent AI consultant. He writes about LLM evaluation, tooling and applied ML at hamel.dev, and previously worked on machine learning at GitHub.
11 August
AI research engineer working on large language models. He writes the Ahead of AI newsletter and is the author of Build a Large Language Model (From Scratch).
Whoa, Meta released a new open-weight LLM yesterday, something that hasn't happened since the good old Llama days. Their Meta Muse Glimmer model is a 30B multimodal reasoning model with a Gemma-like architecture design. (“Glimmer” is probably a wordplay on “Spark,” the more likely capable model from which Glimmer was distilled. Muse Spark is only available through Meta’s Model API, though.) Architecture-wise, here are some of the main points: 1. "Only" a 131k context window, compared to Qwen3.6 and Gemma 4, which support 2x that natively; it's reasonable, but maybe on the shorter end in the a… LLMs Related
10 August
Founded PSPDFKit in 2011 and ran it for a decade. Came back from a break to work on AI agents — the OpenClaw project, and OpenAI, joined in February 2026. Writes at steipete.me.
27 July
Creator of Claude Code at Anthropic.
25 July
Programmer and writer on computer architecture, performance, and software reliability. He has worked on CPU design at Centaur Technology and on software at Google and Microsoft, and writes long-form technical essays at danluu.com.
In another variant of https://danluu.com/learn-what/, I caught up with a former colleague who worked on automated theorem proving. It turns out he's had an interesting career doing all sorts of interesting stuff using the skills he developed by spending a decade writing/using theorem provers. At one point, he said, "if you use X like a theorem prover, it works really well", which surprised me to hear, but of course this is a highly generalizable skill just like compilers or benchmarking/evals. Related
24 July
Programmer and writer on computer architecture, performance, and software reliability. He has worked on CPU design at Centaur Technology and on software at Google and Microsoft, and writes long-form technical essays at danluu.com.
x.com Exercises in evals and benchmarking: DeepSWE / Senior SWE-Bench, performance math, and cold weather tires Related
Bluesky Exercises in benchmarking and evals, part 7: performance napkin math, DeepSWE / Senior SWE-Bench, and winter tires danluu.com
Related
Mastodon Exercises in benchmarking and evals: performance math, winter tires, and DeepSWE / Senior SWE-Bench, danluu.com
Related
20 July
Physician and neuroscientist at the University of Washington; researches brain health, metabolism and athletic performance.
From one piece
Accelerate Learning & Increase Cognitive Capacity | Dr. Tommy Wood
2 beliefs, in the piece's order there
Their words
They just published a secondary analysis of the active trial that showed that um those who did processing speed training with uh booster sessions at one and three years had a significantly decreased risk of dementia 20 years later.
Show the whole quote
youtube.com
Their words
We kind of expect that we can go to work and we can work really hard for eight hours straight during the day, but like the brain doesn't really work like that.
Show the whole quote
youtube.com
1 July
Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.
I’m at @aiDotEngineer and hanging out around the music corner on the 2nd floor from 1415 - 1515! Come by to chat about https://t.co/pdd8bk66Jz, https://t.co/eJSa2MISCW, https://t.co/51rtuo4PDO, evals, agents, memory, how to work effectively with claude code, claude tag, etc! Anthropic Related
29 June
Machine learning engineer and independent AI consultant. He writes about LLM evaluation, tooling and applied ML at hamel.dev, and previously worked on machine learning at GitHub.
26 June
AI research engineer working on large language models. He writes the Ahead of AI newsletter and is the author of Build a Large Language Model (From Scratch).
21 June
Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.
9 June
Founding member of OpenAI and former director of AI at Tesla; creator of nanoGPT and the term "vibe coding".
8 June
Programmer and writer on computer architecture, performance, and software reliability. He has worked on CPU design at Centaur Technology and on software at Google and Microsoft, and writes long-form technical essays at danluu.com.
x.com Exercises in benchmarking, evals, and experimental design, part 6: Related
Mastodon Exercises in benchmarking, evals, and experimental design, part 6: patreon.com
Related
4 June
Programmer known for LiveJournal, memcached, OpenID and gearman; worked on the Go programming language at Google and later joined Tailscale.
26 May
Programmer; wrote Extreme Programming Explained and Test-Driven Development, signed the Agile Manifesto, and writes Tidy First?.
13 May
Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.
Mythos evals from XBOW and UK AISI: • UK AISI: Mythos completed a 32-step network attack (est. at ~20 hrs for experts) in 6/10 tries. First model to solve their e2e cyber ranges! • XBOW: "token-for-token, unprecedented precision" Read more: https://t.co/hD6G6DvALg, xbow.com
Related
Software engineer and engineering leader at Canonical, working on Ubuntu; writes about his homelab, Nix and tooling at jnsgr.uk.
4 May
Co-founder of Modem and a founding engineer at Sentry, where he went on to be VP of Engineering. Co-author of Third-party JavaScript, and co-host of the State of Agentic Coding podcast with Armin Ronacher.
In 3 weeks at the next Devtools Toronto: traces.com - share agent coding sessions polarity.so - sandboxes for agent testing superuser.app - agents in your team chat tidra.ai - AI code maintenance at scale docsalot.dev - docs agents can read codepathfinder.dev - trace vulns across your codebase Related
2 March
Machine learning engineer and independent AI consultant. He writes about LLM evaluation, tooling and applied ML at hamel.dev, and previously worked on machine learning at GitHub.
Evals Skills for Coding Agents Today, Shreya Shankar and I are publishing evals skills, a set of skills for AI product evals1. Eval tools often get in the way. They nudge you toward generic off-the-shelf metrics and fully automated evals before you’v…
coding agents Related
12 February
Managing partner at Ritholtz Wealth Management. Writes the blog The Irrelevant Investor and co-hosts the Animal Spirits podcast with Ben Carlson.
Wild market. We haven't seen anything like this since the dotcom bubble burst. Over the last 8 sessions, 115 stocks in the S&P 500 have decline 7% or more in a single day. The average drawdown when that happens is 34%. Right now we're 1.5% below the all-time high. Related
23 January
Software developer and entrepreneur; co-founder of Heroku, author of The Twelve-Factor App, and a researcher at Ink & Switch.
25 November 2025
Co-founder and chief scientist of Safe Superintelligence Inc., and previously co-founder and chief scientist of OpenAI.
Their words
And one of the one thing you could do, and I think that's something that is done inadvertently, is that people take inspiration from the evals.
Show the whole quote
youtube.com
23 November 2025
Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.
1 October 2025
Machine learning engineer and independent AI consultant. He writes about LLM evaluation, tooling and applied ML at hamel.dev, and previously worked on machine learning at GitHub.
Selecting The Right AI Evals Tool Over the past year, I’ve focused heavily on AI Evals, both in my consulting work and teaching. A question I get constantly is, “What’s the best tool for evals?”. I’ve always resisted answering directly for two reasons.…
Related
25 June 2025
Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.
Wrote an intro to evals for long-context Q&A systems: • How it differs from basic Q&A • What dimensions & metrics to eval on • How to build llm-evaluators • How to build eval datasets • Benchmarks: narratives, technical docs, multi-docs eugeneyan.com
LLMs benchmarks Related
22 June 2025
Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.
LLMs
Their words
This is why model-based evaluation is increasingly popular-it offers more reliable and nuanced evals than traditional metrics.
Show the whole quote
eugeneyan.com
28 May 2025
Machine learning engineer and independent AI consultant. He writes about LLM evaluation, tooling and applied ML at hamel.dev, and previously worked on machine learning at GitHub.
30 April 2025
Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.
@hamel.bsky.social & @sh-reya.bsky.social are two of the world's best on evals. They've built evals for 35+ AI apps & helped teams ship confidently. Now they'll teach everything they know on building evals that work. Enrollment closes in 4 days. Secret 35% discount code: maven.com
Related
23 April 2025
Research psychologist, instrumental in founding the field of naturalistic decision making; developed the recognition-primed decision model, the PreMortem method and the ShadowBox training method, and wrote Sources of Power and Seeing What Others Don't.
Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.
Product evals are misunderstood. Many teams think that adding another tool, metric, or llm-as-judge will solve all their problems and save their product. But that just dodges the hard truth and avoids the real work. Here's how to fix your process instead. eugeneyan.com
LLMs Related
16 April 2025
Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.
@hamel.bsky.social & his wisdom on evals, error analysis, looking at your data is what we need. Here are his 10 Don'ts: • Don't skip error analysis • Don't skip looking at your data • Don't gatekeep who can write prompts • Don't let zero users be a roadblock • Don't be blindsided by criteria drift Related
9 April 2025
Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.
If you were building a Q&A feature (or chatbot) based on very long documents (like books), what evals would you focus on? Related
3 February 2025
Machine-learning researcher on open language models; writes the Interconnects newsletter and the RLHF Book, after leading post-training at Ai2.
Their words
And the important thing to say is that no matter how you want the model to behave, these RLHF and preference-tuning techniques also improve performance. So, on things like math evals and code evals, there is something innate to these, what is called contrastive loss functions.
Show the whole quote
youtube.com
29 October 2024
Machine learning engineer and independent AI consultant. He writes about LLM evaluation, tooling and applied ML at hamel.dev, and previously worked on machine learning at GitHub.
Their words
You might be skeptical of using synthetic data. After all, it’s not real data, so how can it be a good proxy? In my experience, it works surprisingly well. Some of my favorite AI products, like Hex use synthetic data to power their evals
Show the whole quote
hamel.dev
21 June 2024
Co-creator of Django and creator of Datasette; writes daily at simonwillison.net.
Their words
Your AI Product Needs Evals by Hamel Husain remains my favourite piece of writing on how to go about putting these together.
Show the whole quote
simonwillison.net
21 July 2022
Executive coach for product leaders. Previously a partner at GV (Google Ventures) and a product manager at Google. Writes at Bring the Donuts, including a standing essential-reading list for product managers.
CEOs and Product Leaders: A Better Relationship A relationship built on communication, candor, and trust - In my coaching sessions with CPOs and other senior product leaders, the relationship with a CEO is a frequent topic. Sometimes there’s been a colossal dust-up;…
Related
Nothing matches. Show everything
A korrent is a belief a person has stated in their own words: one
sentence stating the claim, backed by a quote and a source, kept at
korrents.com .
Under a name here, the quoted block is what they actually said.
The korrent beneath it is the claim those words support, in
korrents' wording — tap it to see the record, its source, and who
else holds it.
Nobody here wrote their own korrents. They are compiled from public
statements, and a person can change their mind, which is recorded too.
Got it
About the English under a post
Some people here publish in a language other than English. Where they
do, this site shows a machine translation beneath the post, in
this typeface — the site's own, not theirs.
The post itself is never changed, moved or hidden: what is set in the
serif above is exactly what the person published, and it is what to
quote them on. A translation can be wrong in ways that matter,
especially about tone.
Only the post's own words are translated. A quoted post, a linked
article and a belief on korrents.com
are left in their original language.
Got it