Hamel Husain
Machine learning engineer and independent AI consultant. He writes about LLM evaluation, tooling and applied ML at hamel.dev, and previously worked on machine learning at GitHub.
Hamel Husain did not write this page. ppll compiled it from their own words, each linked to the source. Is this you? · Report a mistake
Latest Site
What they believe
-
Evaluation and error analysis will remain necessary practices even if AI models become near-perfect.
Even with perfect models, you still need to verify they're solving the right problem. The need for systematic error analysis, domain-specific testing, and monitoring will still be important.
hamel.devSaid 18 Sept 2026This on korrents
-
An artifact that is hard for its builder to verify is usually also hard for its users to verify.
Artifacts that are hard for you to verify are often hard for users too.
hamel.devSaid 29 Jun 2026This on korrents
The bold line is korrents’ wording; the quote is theirs. These 2: what they wrote before what they said aloud, one per source, newest first.
Feed
As its own page →Hiding
20 October
18 October
12 October
11 October
10 October
9 October
8 October
7 October
6 October
5 October
4 October
3 October
2 October
30 September
18 September
From one piece AI Evals: Everything You Need to Know 2 beliefs · hamel.dev
-
korrents.com
People cannot fully specify what they want from an AI system without first observing how it behaves.Their words
Additionally, a LLM cannot read your mind, and research shows that people need to observe the LLM's behavior in order to properly externalize their requirements.
-
korrents.com
Evaluation and error analysis will remain necessary practices even if AI models become near-perfect.Their words
Even with perfect models, you still need to verify they're solving the right problem. The need for systematic error analysis, domain-specific testing, and monitoring will still be important.
16 August
12 August
13 July
1 July
29 June
From one piece “It’s Hard to Eval” Is a Product Smell 3 beliefs · hamel.dev
-
Their words
Before AI, verification often happened incidentally during the process of creating work product. With AI, verification is the bottleneck.
-
korrents.com
Designing a product for ease of verification should happen before building evals for it.Their words
More importantly, designing your product for ease of verification should come before building evals.
-
korrents.com
An artifact that is hard for its builder to verify is usually also hard for its users to verify.Their words
Artifacts that are hard for you to verify are often hard for users too.
11 May
11 April
27 March
26 March
15 March
9 March
2 March
27 February
18 January
From one piece Why I Stopped Using nbdev 2 beliefs · hamel.dev
-
korrents.com
Careful step-by-step thinking is a property of the developer, not something a tool or framework can enforceTheir words
Notebooks don't force you to write clean code. AI tools don't force you to think carefully. Discipline comes from the developer, not the environment.
-
korrents.com
Good documentation is produced by effort, not by tooling that keeps code and docs togetherTheir words
This reinforced my belief that good documentation comes from effort, not tooling.
-
Usesrcmnd.app
AmpTheir words
My daily drivers now include Amp, Cursor, and Claude Code. I still enjoy notebooks, but only for data analysis, machine learning, or other exploratory workflows where the iterative, visual nature of notebooks shines.
-
Usesrcmnd.app
CursorTheir words
My daily drivers now include Amp, Cursor, and Claude Code. I still enjoy notebooks, but only for data analysis, machine learning, or other exploratory workflows where the iterative, visual nature of notebooks shines.
-
Usesrcmnd.app
Claude CodeTheir words
My daily drivers now include Amp, Cursor, and Claude Code. I still enjoy notebooks, but only for data analysis, machine learning, or other exploratory workflows where the iterative, visual nature of notebooks shines.
+ 2 more
-
Mixed ontheir ownrcmnd.app
nbdevTheir words
Because I have invested thousands of hours into nbdev, it's difficult to admit there are better tools for the outcomes I want.
-
Likedrcmnd.app
Next.jsTheir words
I now use different languages for different tasks. For web development, I prefer the Next.js stack.
12 November 2025
31 October 2025
1 October 2025
-
Mixed onrcmnd.app
LangsmithTheir words
The overall workflow is intuitive, especially for those new to formal evaluation processes. The UI guides you through creating datasets, running experiments, and annotating results.
-
Mixed onrcmnd.app
BraintrustTheir words
The panel had a generally positive view of Braintrust, highlighting its clean UI and structured approach to evaluations. The tool’s emphasis on human-in-the-loop workflows was a significant strength.
-
Likedrcmnd.app
Arize PhoenixTheir words
The panel had a generally positive view of Phoenix, with one panelist calling it one of his “favorite open source eval tools.” The tool is positioned as a developer-first, notebook-centric platform.
+ 1 more
-
Usesrcmnd.app
Jupyter notebooksTheir words
As for me personally, I tend to use these tools as a backend data store and use Jupyter notebooks as well as my own custom built annotation interfaces for most of my needs.
24 March 2025
30 November 2024
29 October 2024
29 July 2024
Nothing matches.
Beliefs
Korrents What they believe 7 beliefs — each backed by an exact quote.
Each is a — compiled by korrents.com, not by them: the one-line wordings are korrents', the quotes are theirs.
Recent
Evaluation and error analysis will remain necessary practices even if AI models become near-perfect.
Even with perfect models, you still need to verify they're solving the right problem. The need for systematic error analysis, domain-specific testing, and monitoring will still be important.
AI Evals: Everything You Need to Know Said 18 Sept 2026
People cannot fully specify what they want from an AI system without first observing how it behaves.
Additionally, a LLM cannot read your mind, and research shows that people need to observe the LLM's behavior in order to properly externalize their requirements.
AI Evals: Everything You Need to Know Said 18 Sept 2026
An artifact that is hard for its builder to verify is usually also hard for its users to verify.
Artifacts that are hard for you to verify are often hard for users too.
“It’s Hard to Eval” Is a Product Smell Said 29 Jun 2026
Show 4 more
Designing a product for ease of verification should happen before building evals for it.
More importantly, designing your product for ease of verification should come before building evals.
“It’s Hard to Eval” Is a Product Smell Said 29 Jun 2026
With AI, verification has become the bottleneck in producing work, whereas before it happened incidentally during creation.
Before AI, verification often happened incidentally during the process of creating work product. With AI, verification is the bottleneck.
“It’s Hard to Eval” Is a Product Smell Said 29 Jun 2026
Good documentation is produced by effort, not by tooling that keeps code and docs together
This reinforced my belief that good documentation comes from effort, not tooling.
Why I Stopped Using nbdev Said 18 Jan 2026
Careful step-by-step thinking is a property of the developer, not something a tool or framework can enforce
Notebooks don't force you to write clean code. AI tools don't force you to think carefully. Discipline comes from the developer, not the environment.
Why I Stopped Using nbdev Said 18 Jan 2026
What is a korrent?
A korrent is a belief a person has stated in their own words: one sentence stating the claim, backed by a quote and a source, kept at korrents.com.
Under a name here, the quoted block is what they actually said. The korrent beneath it is the claim those words support, in korrents' wording — tap it to see the record, its source, and who else holds it.
Nobody here wrote their own korrents. They are compiled from public statements, and a person can change their mind, which is recorded too.
Recommends rcmnd
rcmnd What they recommend 10 things — each backed by an exact quote.
Compiled by rcmnd.app, not by them: the quotes are theirs, and each links to the post it was copied from.
Recent
Liked Next.js
I now use different languages for different tasks. For web development, I prefer the Next.js stack.
hamel.dev 18 Jan 2026
Mixed on nbdev
Because I have invested thousands of hours into nbdev, it's difficult to admit there are better tools for the outcomes I want.
hamel.dev 18 Jan 2026
their own
Uses Claude Code
My daily drivers now include Amp, Cursor, and Claude Code. I still enjoy notebooks, but only for data analysis, machine learning, or other exploratory workflows where the iterative, visual nature of notebooks shines.
hamel.dev 18 Jan 2026
Show 7 more
Uses Cursor
My daily drivers now include Amp, Cursor, and Claude Code. I still enjoy notebooks, but only for data analysis, machine learning, or other exploratory workflows where the iterative, visual nature of notebooks shines.
hamel.dev 18 Jan 2026
Uses Amp
My daily drivers now include Amp, Cursor, and Claude Code. I still enjoy notebooks, but only for data analysis, machine learning, or other exploratory workflows where the iterative, visual nature of notebooks shines.
hamel.dev 18 Jan 2026
Uses Jupyter notebooks
As for me personally, I tend to use these tools as a backend data store and use Jupyter notebooks as well as my own custom built annotation interfaces for most of my needs.
hamel.dev 1 Oct 2025
Liked Arize Phoenix
The panel had a generally positive view of Phoenix, with one panelist calling it one of his “favorite open source eval tools.” The tool is positioned as a developer-first, notebook-centric platform.
hamel.dev 1 Oct 2025
Mixed on Braintrust
The panel had a generally positive view of Braintrust, highlighting its clean UI and structured approach to evaluations. The tool’s emphasis on human-in-the-loop workflows was a significant strength.
hamel.dev 1 Oct 2025
Mixed on Langsmith
The overall workflow is intuitive, especially for those new to formal evaluation processes. The UI guides you through creating datasets, running experiments, and annotating results.
hamel.dev 1 Oct 2025
Loved Hex
You might be skeptical of using synthetic data. After all, it’s not real data, so how can it be a good proxy? In my experience, it works surprisingly well. Some of my favorite AI products, like Hex use synthetic data to power their evals
hamel.dev 29 Oct 2024
Where they publish
Site hamel.dev His blog: posts on LLM evaluation, AI tooling and applied machine learning.
Recent
- How To Use Decision Models for Evals 20 Oct 2026 Decision models have exploded onto the scene recently, starting with TypeSafe’s Jev, followed by Perplexity’s Decider, Liquid’s d1, Cloudflare’s Clef, and OpenAI’s Decisions API1. These models promise fast, inexpensive…
- Claude’s new auto eval tool 30 Sept 2026 Anthropic released new eval tooling for Claude Code. Their claude-api plugin now includes a new build_eval and hill-climb command that helps you build evals, check the graders, and improve your application against them.…
- AI Evals: Everything You Need to Know 18 Sept 2026 This document curates the most common questions Shreya and I received while teaching 5,000+ engineers and PMs AI Evals. Warning: These are sharp opinions about what works in most cases. They are not universal truths. Us…
Show 10 more
- AI Product Engineering Notes 12 Aug 2026 Notes from 13 sessions on evals, context, and systems. 9.5 hours of talks distilled into about 20 minutes of reading.
- “It’s Hard to Eval” Is a Product Smell 29 Jun 2026 For the past 3 years, AI evals have been my professional focus.1 The most common objection I hear to evals is “our product is hard to eval”. This objection is a product smell. Artifacts that are hard for you to verify a…
- The Revenge of the Data Scientist 26 Mar 2026 Is the heyday of the data scientist over? The Harvard Business Review once called it “The Sexiest Job of the 21st Century.”1 In tech, data scientist roles were often among the best paid.2 The job also demanded an unusua…
- Evals Skills for Coding Agents 2 Mar 2026 Today, Shreya Shankar and I are publishing evals skills, a set of skills for AI product evals1. Eval tools often get in the way. They nudge you toward generic off-the-shelf metrics and fully automated evals before you’v…
- Why I Stopped Using nbdev 18 Jan 2026 Programmers love to proclaim they’ve found the best tool. Paul Graham called Lisp his “secret weapon.” DHH described Ruby as “a magical glove that just fit my brain perfectly.” Pieter Levels ships million-dollar product…
- Selecting The Right AI Evals Tool 1 Oct 2025 Over the past year, I’ve focused heavily on AI Evals, both in my consulting work and teaching. A question I get constantly is, “What’s the best tool for evals?”. I’ve always resisted answering directly for two reasons.…
- A Field Guide to Rapidly Improving AI Products 24 Mar 2025 Most AI teams focus on the wrong things. Here’s a common scene from my consulting work: AI TEAM Here’s our agent architecture – we’ve got RAG here, a router there, and we’re using this new framework for… ME [Holding up…
- Building an Audience Through Technical Writing: Strategies and Mistakes 30 Nov 2024 People often find me through my writing on AI and tech. This creates an interesting pattern. Nearly every week, vendors reach out asking me to write about their products. While I appreciate their interest and love learn…
- Using LLM-as-a-Judge For Evaluation: A Complete Guide 29 Oct 2024 Earlier this year, I wrote Your AI product needs evals. Many of you asked, “How do I get started with LLM-as-a-judge?” This guide shares what I’ve learned after helping over 30 companies set up their evaluation systems.…
- An Open Course on LLMs, Led by Practitioners 29 Jul 2024 Today, we are releasing Mastering LLMs, a set of workshops and talks from practitioners on topics like evals, retrieval-augmented-generation (RAG), fine-tuning and more. This course is unique because it is: Taught by 25…
Link verified 4 Sept 2026. Recent items update automatically from the channel.
x.com @HamelHusain Day-to-day posting about LLM evals and tooling.
Recent
- 👀 18 Oct 2026
- I’ve known Ajay for a while and he’s doubled down on making tools that help you look at your data for evals Promising new entrant into the eval tools space 12 Oct 2026
- Marketers: The next wave of founders will be marketers, distribution is the new moat. PMs: Taste is the moat, and its easier than ever to build the wrong thing. Now is the age of the PM. Eng: You need systems thinking and technical chops to solve hard problems and wield agents effectively at scale. Engineers have more leverage than ever. I suspect good people in each discipline will remain valuable and there isn't one role poised to become unicorns en masse. I do think its good to be skilled at more than 1 discipline, but I am not sure any one of these disciplines has a leg up over the others… 11 Oct 2026
Show 17 more
- Excalidraw for slides is the 🐐 MCP works great & has animations etc. You can create nice visuals that don't feel like over-produced slop 11 Oct 2026
- Recently had the chance to interview @benhylak on how he is thinking about simulations in our AI Evals Course 😁 Simulations are incredibly useful for evals when you do them well. Ben discusses what he learned from implementing simulations for evals at several companies. Some fun topics came up, my favorite being "simulation awareness" - when models know they are in a simulation. Below is a chapter summary: Why input/output tests miss agent failures Replay production traces to see what a change breaks Recreate the data and tools around the agent How agents detect simulations and change their… 10 Oct 2026 youtube.com
- Q: How do I make the case for investing in evaluations to my team? A: Review real user interactions and show your team the problems you find. Fix them, then show what improved. 9 Oct 2026
- Q: Can I use the same model for both the main task and evaluation? A: Usually, yes. Test whether the judge agrees with human labels on a held-out test set. 8 Oct 2026
- Q: Should I use "ready-to-use" evaluation metrics? A: No. It's a bad idea in most situations. 7 Oct 2026
- Saw this job posting for an "anti-slop" professional writer but the post itself written with a slop cannon. This is peak AI psychosis, someone is so enamored with their own writing they can't even see it 7 Oct 2026
- Q: Are similarity metrics useful for evaluating LLM outputs? A: Similar wording does not tell you whether an answer works for your application. Check specific failures. Similarity metrics can help with retrieval and output diversity. 6 Oct 2026
- Q: How often should I run my evals? A: Weigh the cost and how saturated the eval is against the business value of catching the error. Read more: https://t.co/qkszc9HgJ0 Read all the FAQs here: 5 Oct 2026
- You have to respect Schmidhuber’s commitment to the bit 🤣 he was just trolling us all along (which is probably the right way to use this site tbh) 4 Oct 2026
- I cannot understand this demo video at all. Sadly, it’s visual slop 😭 Not trying to disrespect anyone maybe the feature is cool it’s just what happened to demos humans can understand 3 Oct 2026
- Last open lesson of the year! How to crack the AI Evals Interview! AI Evals are now a part of an increasing number of AI job postings, and you're now more likely to encounter them in interviews. I've helped design quite a few evals interviews, so I know what kinds of traps exist, along with nuanced things companies tend to look for. I'll discuss this, as well as how to structure a take home or report to help you think through the problem clearly. Tuesday Oct 6, 8:30am PT. This one assumes some basic evals knowledge. Read this first if you're new to the topic to get the most out of the session… 3 Oct 2026 maven.com
- When I see this it confuses me. ssh into a computer for god sakes, or use codex remote feature w/ mac mini. Once you have been stuck in "i need to keep my laptop open" more than 3 times, its crazy to keep this insanity going. 2 Oct 2026
- WTF is SF Tech Week? Isn’t every week tech week there LMAO 2 Oct 2026
- Q: Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)? A: Binary labels force clearer thinking and more consistent labeling, and involves significantly lower complexity to operationalize. 2 Oct 2026
- Are there any humans here 👋? 95% of replies are bots 1 Oct 2026
- Q: Should I build automated evaluators for every failure mode I find? A: No. Different types of evaluators have different costs (code vs. LLM judge), which you need to weigh before building an eval. 1 Oct 2026
- People keep asking about this new Auto Eval plugin for Claude this so I wrote a longer post reviewing it. Would love to hear what other people think! If you used it how did it go? Full writeup here: 1 Oct 2026
Link verified 4 Sept 2026. Recent items update automatically from the channel.
GitHub github.com/hamelsmu His code, including work on nbdev and fastpages.
Recent
- evals-skills — Skills for AI Evals to compliment the course: AI Evals For Engineers & PMs 16 Aug 2026
- reverse-eng-site-skill — A reusable agent skill for discovering and validating website internal APIs through instrumented browser traffic and direct HTTP replay. 13 Jul 2026
- substack-stats — Interactive dashboard for public Substack newsletter revenue modeling 1 Jul 2026
Show 9 more
- stfu — Find the app or browser tab making noise and make it STFU. 11 May 2026
- website-to-api — Claude Code plugin for reverse-engineering website APIs using Chrome browser automation 11 Apr 2026
- hamelsmu — GitHub Profile README 27 Mar 2026
- claude-review-loop — Claude Code plugin: automated code review loop with Codex 15 Mar 2026
- hamelnb 9 Mar 2026
- research-council 27 Feb 2026
- design-research — Claude Code skill for comprehensive website design research using browser-automation agent teams 27 Feb 2026
- air-demo 12 Nov 2025
- text-editor-tutorial 31 Oct 2025
Link verified 4 Sept 2026. Recent items update automatically from the channel.
About the English under a post
Some people here publish in a language other than English. Where they do, this site shows a machine translation beneath the post, in this typeface — the site's own, not theirs.
The post itself is never changed, moved or hidden: what is set in the serif above is exactly what the person published, and it is what to quote them on. A translation can be wrong in ways that matter, especially about tone.
Only the post's own words are translated. A quoted post, a linked article and a belief on korrents.com are left in their original language.