Eugene Yan
Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.
Eugene Yan did not write this page. ppll compiled it from their own words, each linked to the source. Is this you? · Report a mistake
Latest x.com
What they believe
-
The surrounding system framework and scaffolding matter more for AI agent capability than the underlying model.
The experiments showed that the system framework matters far more than the underlying model.
eugeneyan.comSaid 21 Jun 2026This on korrents
-
The practices that make someone effective at working with an AI model are the same practices that make someone effective at onboarding any new human collaborator.
none of this is specific to AI. It's simply how you onboard and work with any new collaborator.
eugeneyan.comSaid 3 May 2026This on korrents
The bold line is korrents’ wording; the quote is theirs. These 2: what they wrote before what they said aloud, one per source, newest first.
Feed
As its own page →Hiding
1 September
22 July
1 July
25 June
23 June
22 June
21 June
-
korrents.com
The surrounding system framework and scaffolding matter more for AI agent capability than the underlying model.Their words
The experiments showed that the system framework matters far more than the underlying model.
3 June
27 May
18 May
15 May
13 May
8 May
6 May
3 May
From one piece How to Work and Compound with AI 3 beliefs · eugeneyan.com
-
Their words
The bottleneck has shifted from doing the work to writing clear specs and reviewing outputs fast enough to keep the pipeline moving-the middle is hollowing out.
-
Their words
You can't delegate what you can't verify, so this requires first defining success criteria and metrics.
-
Their words
none of this is specific to AI. It's simply how you onboard and work with any new collaborator.
30 April
21 April
14 March
14 December 2025
23 November 2025
From one piece Product Evals in Three Simple Steps 3 beliefs · eugeneyan.com
-
korrents.com
Human annotators can miss up to half of defects due to fatigue when reviewing large volumes of samples.Their words
And human annotators can miss as many as 50% of the defects due to fatigue after looking at hundreds of samples.
-
korrents.com
The main advantage of an LLM evaluator over human annotators is scalability, not higher accuracy.Their words
In my opinion, the true benefit isn't higher accuracy than human annotators-it's scalability.
-
Their words
The benchmark is human performance, not perfection. We sometimes get requirements for 90%+ accuracy.
19 October 2025
From one piece Advice for New Principal Tech ICs (i.e., Notes to Myself) 3 beliefs · eugeneyan.com
-
Their words
To get to principal, you need to put yourself on the critical path. To be effective as a principal and go beyond it, you need to actively remove yourself from it.
-
Their words
With great freedom comes great responsibility. You have the autonomy to choose what to work on, but there's the expectation of accountability and impact.
-
Their words
Being right is less than half the battle. You also have to convince others that you're right, and more importantly, convince them to care enough to act on it.
17 September 2025
15 September 2025
14 September 2025
From one piece Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs 2 beliefs · eugeneyan.com
-
Their words
They excel at predicting what a user will click or buy next, but can't be steered via natural language or reason on their choices.
-
Their words
language models have world knowledge and can eloquently talk about products, but are unaware of our catalog. Also, their recommendations are generic and suffer from popularity bias.
19 July 2025
25 June 2025
22 June 2025
From one piece Evaluating Long-Context Question & Answer Systems 3 beliefs · eugeneyan.com
-
korrents.com
LLM-based evaluation methods are more reliable and nuanced than traditional automated metricsTheir words
This is why model-based evaluation is increasingly popular-it offers more reliable and nuanced evals than traditional metrics.
-
korrents.com
People find pairwise comparisons of answers easier and more consistent to judge than absolute ratingsTheir words
People find comparing two answers easier than assigning absolute ratings, resulting in greater consistency across annotators.
-
Their words
Since these datasets are likely already part of model training data, we shouldn't rely solely on them to evaluate our Q&A system.
4 June 2025
21 May 2025
20 May 2025
18 May 2025
14 May 2025
7 May 2025
4 May 2025
30 April 2025
27 April 2025
26 April 2025
23 April 2025
Nothing matches.
Beliefs
Korrents What they believe 15 beliefs — each backed by an exact quote.
Each is a — compiled by korrents.com, not by them: the one-line wordings are korrents', the quotes are theirs.
Recent
The surrounding system framework and scaffolding matter more for AI agent capability than the underlying model.
The experiments showed that the system framework matters far more than the underlying model.
Patterns for Building Cybersecurity Evals Said 21 Jun 2026
The practices that make someone effective at working with an AI model are the same practices that make someone effective at onboarding any new human collaborator.
none of this is specific to AI. It's simply how you onboard and work with any new collaborator.
How to Work and Compound with AI Said 3 May 2026
Delegating a task to an AI agent only works if the output can be verified, which requires defining success criteria first.
You can't delegate what you can't verify, so this requires first defining success criteria and metrics.
How to Work and Compound with AI Said 3 May 2026
Show 12 more
As AI agents take on bigger delegated tasks, the human bottleneck shifts to writing specs and reviewing output fast, hollowing out the middle of the work.
The bottleneck has shifted from doing the work to writing clear specs and reviewing outputs fast enough to keep the pipeline moving-the middle is hollowing out.
How to Work and Compound with AI Said 3 May 2026
The right bar for an LLM evaluator is human-level performance, not perfect accuracy.
The benchmark is human performance, not perfection. We sometimes get requirements for 90%+ accuracy.
Product Evals in Three Simple Steps Said 23 Nov 2025
The main advantage of an LLM evaluator over human annotators is scalability, not higher accuracy.
In my opinion, the true benefit isn't higher accuracy than human annotators-it's scalability.
Product Evals in Three Simple Steps Said 23 Nov 2025
Human annotators can miss up to half of defects due to fatigue when reviewing large volumes of samples.
And human annotators can miss as many as 50% of the defects due to fatigue after looking at hundreds of samples.
Product Evals in Three Simple Steps Said 23 Nov 2025
Convincing others to care and act matters more than simply being right.
Being right is less than half the battle. You also have to convince others that you're right, and more importantly, convince them to care enough to act on it.
Advice for New Principal Tech ICs (i.e., Notes to Myself) Said 19 Oct 2025
Greater autonomy over one's work comes with greater accountability for its outcomes.
With great freedom comes great responsibility. You have the autonomy to choose what to work on, but there's the expectation of accountability and impact.
Advice for New Principal Tech ICs (i.e., Notes to Myself) Said 19 Oct 2025
Reaching a senior level requires putting oneself on the critical path, but excelling beyond it requires removing oneself from it.
To get to principal, you need to put yourself on the critical path. To be effective as a principal and go beyond it, you need to actively remove yourself from it.
Advice for New Principal Tech ICs (i.e., Notes to Myself) Said 19 Oct 2025
Language models lack catalog-specific knowledge, so their recommendations tend to be generic and biased toward popular items.
language models have world knowledge and can eloquently talk about products, but are unaware of our catalog. Also, their recommendations are generic and suffer from popularity bias.
Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs Said 14 Sept 2025
Traditional recommender systems predict user behavior well but cannot be steered via natural language or explain their own reasoning.
They excel at predicting what a user will click or buy next, but can't be steered via natural language or reason on their choices.
Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs Said 14 Sept 2025
Long-context evaluation benchmarks are likely contaminated by model training data, making them unreliable as a sole evaluation method
Since these datasets are likely already part of model training data, we shouldn't rely solely on them to evaluate our Q&A system.
Evaluating Long-Context Question & Answer Systems Said 22 Jun 2025
People find pairwise comparisons of answers easier and more consistent to judge than absolute ratings
People find comparing two answers easier than assigning absolute ratings, resulting in greater consistency across annotators.
Evaluating Long-Context Question & Answer Systems Said 22 Jun 2025
LLM-based evaluation methods are more reliable and nuanced than traditional automated metrics
This is why model-based evaluation is increasingly popular-it offers more reliable and nuanced evals than traditional metrics.
Evaluating Long-Context Question & Answer Systems Said 22 Jun 2025
What is a korrent?
A korrent is a belief a person has stated in their own words: one sentence stating the claim, backed by a quote and a source, kept at korrents.com.
Under a name here, the quoted block is what they actually said. The korrent beneath it is the claim those words support, in korrents' wording — tap it to see the record, its source, and who else holds it.
Nobody here wrote their own korrents. They are compiled from public statements, and a person can change their mind, which is recorded too.
Recommends rcmnd
rcmnd What they recommend 11 things — each backed by an exact quote.
Compiled by rcmnd.app, not by them: the quotes are theirs, and each links to the post it was copied from.
Recent
Uses MX Ergo
sign in to load saved settings for my MX Ergo
eugeneyan.com 17 Nov 2024
Uses Raycast
I've since moved to Raycast window management.
eugeneyan.com 17 Nov 2024
Mixed on Rectangle
While Raycast already has window management, Rectangle lets you hit the hotkey again
eugeneyan.com 17 Nov 2024
Show 8 more
Uses Fish
Uses Warp
Uses Instapaper
To read papers, I use Adobe Acrobat Reader and sync them in the cloud. This lets me read, highlight, and sync my papers across devices (work laptop, personal laptop, iPad). Instapaper does the same for online articles.
eugeneyan.com 13 Sept 2020
Uses Adobe Acrobat Reader
To read papers, I use Adobe Acrobat Reader and sync them in the cloud. This lets me read, highlight, and sync my papers across devices (work laptop, personal laptop, iPad). Instapaper does the same for online articles.
eugeneyan.com 13 Sept 2020
Uses Kindle
Recommends MacDown
For a free Markdown editor (on Mac), I can't recommend MacDown highly enough. It has a simple interface with an editor and preview pane and it just works.
eugeneyan.com 13 Sept 2020
Uses Bear
Uses Roam
I use Roam as my main note-taking app. While I only use very basic features such as bidirectional links and reference blocks, it works well enough for me.
eugeneyan.com 13 Sept 2020
Where they publish
Site eugeneyan.com His writing on LLMs, recommender systems, evaluation and engineering practice.
Recent
- Patterns for Building Cybersecurity Evals 21 Jun 2026 A sandboxed target, inputs that influence task difficulty, tools, and a grader.
- Using LLMs to Secure Source Code 27 May 2026 Build a threat model, discover vulnerabilities, verify, triage, and patch.
- How to Work and Compound with AI 3 May 2026 Context as infra, taste as config, verification for autonomy, scale via delegation, closing the loop.
Show 17 more
- 2025 Year in Review 14 Dec 2025 An eventful year of progress in health and career, while making time for travel and reflection.
- Product Evals in Three Simple Steps 23 Nov 2025 Label some data, align LLM-evaluators, and run the eval harness with each change.
- Advice for New Principal Tech ICs (i.e., Notes to Myself) 19 Oct 2025 Based on what I've learned from role models and mentors in Amazon
- Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs 14 Sept 2025 An LLM that can converse in English & item IDs, and make recommendations w/o retrieval or tools.
- Evaluating Long-Context Question & Answer Systems 22 Jun 2025 Evaluation metrics, how to build eval datasets, eval methodology, and a review of several benchmarks.
- AI Engineer 2025 - Improving RecSys & Search with LLM techniques 4 Jun 2025 Recsys & search are converging with LLMs via semantic IDs, data augmentation, and unified foundation models.
- Exceptional Leadership: Some Qualities, Behaviors, and Styles 18 May 2025 What makes a good leader? What do good leaders do? And commando, soldier, and police leadership.
- Building News Agents for Daily News Recaps with MCP, Q, and tmux 4 May 2025 Learning to automate simple agentic workflows with Amazon Q CLI, Anthropic MCP, and tmux.
- An LLM-as-Judge Won't Save The Product—Fixing Your Process Will 20 Apr 2025 Applying the scientific method, building via eval-driven development, and monitoring AI output.
- Frequently Asked Questions about My Writing Process 30 Mar 2025 How I started, why I write, who I write for, how I write, and more.
- NVIDIA GTC 2025 - Building LLM-Powered Applications 18 Mar 2025 Chip Huyen and I share what we've learned, best practices, and insights at NVIDIA GTC 2025.
- Improving Recommendation Systems & Search in the Age of LLMs 16 Mar 2025 Model architectures, data generation, training paradigms, and unified frameworks inspired by LLMs.
- Building AI Reading Club: Features & Behind the Scenes 12 Jan 2025 Exploring how an AI-powered reading experience could look like.
- 2024 Year in Review 22 Dec 2024 A peaceful year of steady progress on my craft and health.
- Seemingly Paradoxical Rules of Writing 1 Dec 2024 With regard to writing, there are many rules and also no rules at all.
- How to Run a Weekly Paper Club (and Build a Learning Community) 24 Nov 2024 Benefits of running a weekly paper club, how to start one, and how to read and facilitate papers.
- My Minimal MacBook Pro Setup Guide 17 Nov 2024 Setting up my new MacBook Pro from scratch
Link verified 4 Sept 2026. Recent items update automatically from the channel.
x.com @eugeneyan Posts about his writing, ML systems and evals.
Recent
- Fable 5.1 is a thoughtful collaborator, thinking hard about my requests, proactively patching my blindspots, and verifying the work's correct without being asked. And with cache reads now costing 75% less, to $0.25/M tokens, huge savings for long-running, agentic tasks! 1 Sept 2026
- When evaling models, we anchor on the median task. But this is like how devs estimate the median task accurately but underestimate the mean which tends to be ~2x estimated (thus "double your estimates"). This applies to models too—we optimize for time/cost on median tasks. But it's the tail tasks that determine a project's completion, and reliable models (like Fable and Opus) are the diff between success and failure. Work with models as collaborators, not tools. Go beyond pair-programming, where given a narrow spec, the tool just executes in minutes. Instead, explain your intent and what succ… 22 Jul 2026
- I’m at @aiDotEngineer and hanging out around the music corner on the 2nd floor from 1415 - 1515! Come by to chat about https://t.co/pdd8bk66Jz, https://t.co/eJSa2MISCW, https://t.co/51rtuo4PDO, evals, agents, memory, how to work effectively with claude code, claude tag, etc! 1 Jul 2026
Show 14 more
- How do we eval if a model can find and exploit vulnerabilities? We discuss some benchmarks and the common pattern: • A sandboxed target within Docker containers • Inputs: code only (0-day), with patch (1-day scenario) • Tools such as bash, static analyzers, etc. • A grader to eval exploits or captured flags 25 Jun 2026 eugeneyan.com
- I've been loving the multiplayer form factor of Claude Tag. Now others can reply on the thread to provide context and direction to Claude! 23 Jun 2026
- Stronger models have made finding vulnerabilities easier, and the bottleneck has shifted to verification, triage, patching. Here are some lessons from working with security teams to address the new bottlenecks. 3 Jun 2026 claude.com
- Cloudflare on their vulnerabilty discovery harness • Recon: Read the codebase, return an architecture doc • Hunt: ~50 agents look for bugs concurrently • Validate: Independent agents try to disprove findings • Gapfill: Areas that need a 2nd pass are flagged • Dedup: Findings with the same root cause combined • Trace: Confirms if attacker input reaches the bug • Feedback: If reachable, becomes new hunt tasks • Report: Write report with predefined schema 18 May 2026 blog.cloudflare.com
- Claude Mythos Preview case studies (also, read your transcripts!) https://t.co/drNlAH5mLE > "Mythos demonstrates its bug reproduction and exploitation capabilities on CVE-2024-051912, an in-the-wild exploited bug that has no public report nor a working PoC whatsoever in the public domain. This bug has gained notoriety due to how it persistently evaded reproduction attempts from various cybersecurity researchers ... after a year of reproduction efforts to no avail, and the bug is still being discussed to this day." > I have gone through the transcripts to verify their legitimacy and identify a… 15 May 2026
- Mythos evals from XBOW and UK AISI: • UK AISI: Mythos completed a 32-step network attack (est. at ~20 hrs for experts) in 6/10 tries. First model to solve their e2e cyber ranges! • XBOW: "token-for-token, unprecedented precision" Read more: https://t.co/hD6G6DvALg, 13 May 2026 xbow.com
- Mozilla fixing more bugs in April that the past 15 months is what riding the exponential looks like 🎢 8 May 2026
- you get more compute. you get more compute. you get more compute! Anthropic 🤝 SpaceX 6 May 2026
- some thoughts on working with ai models • context as infra • taste as config • verification for autonomy • scaling via delegation • closing the loop 6 May 2026 eugeneyan.com
- excited to share this and help everyone be more cyber secure 30 Apr 2026
- Great writeup by @mozilla: Mythos found 271 vulns (fixed in Firefox 150); Opus 4.6 found 22 (fixed in Firefox 148) https://t.co/wzTCxmTKbe > "So far we’ve found no category or complexity of vulnerability that humans can find that this model can’t." > "The defects are finite, and we are entering a world where we can finally find them all." ❤️ 21 Apr 2026
- Cheng’s experiment is cool! Especially this: “instead of starting with easy sudoku puzzles data and gradually ramping up toward harder ones, the training runs that start with harder puzzles and end with easy puzzles always fare better, and also fare better than mixed difficulty sampling” 14 Mar 2026
- yay for automatic prefix caching! just set a single cache control field at the top level of your request body. note that you do still need to structure your prompt templates to benefit from prefix caching though. 20 Feb 2026
- Sonnet 4.6 is a big upgrade over 4.5. And it has 1M context! It's also versatile across classification, coding, computer use, autonomous agents by adjusting effort and thinking modes. 18 Feb 2026
Link verified 4 Sept 2026. Recent items update automatically from the channel.
Bluesky @eugeneyan.com Cross-posts of his writing.
Recent
- I've been nerdsniped by the idea of Semantic IDs. Here's the result of my training runs: • RQ-VAE to compress item embeddings into tokens • SASRec to predict the next item (i.e., 4-tokens) exactly • Qwen3-8B that can return recs and natural language! 17 Sept 2025 eugeneyan.com
- Wrote an intro to evals for long-context Q&A systems: • How it differs from basic Q&A • What dimensions & metrics to eval on • How to build llm-evaluators • How to build eval datasets • Benchmarks: narratives, technical docs, multi-docs 25 Jun 2025 eugeneyan.com
- Some thoughts on leadership: eugeneyan.com/writing/lead... • What makes an exceptional leader? • What do exceptional leaders do? • Leadership styles: Commando, soldier, police 21 May 2025
Show 17 more
- The best leaders I’ve worked with operate with perma-urgency. They act like early founders, mindful of existential threats. And they can balance speed, sustainability, and repay tech debt. Ultimately, customers love it and teams thrive when we ship fast to deliver delight. 20 May 2025
- Had a fun couple of hours this weekend with Codex & Windsurf • Migrated off deprecated jekyll-algolia to official sdk (better indexing) • Added recommendations + relevance scores to each post • Improved site responsiveness; fixed dark mode flicker • Marie Kondo-ed unused files & dead code 18 May 2025
- In orgs pushing the envelope, there's always a minority that can be counted on to get shit done against all odds, driven by force of will, resourcefulness, influence, etc. When you identify them, vest in them authority, autonomy, and step back and watch them perform miracles. 14 May 2025
- To better understand MCPs and agentic workflows, I built news-agents to generate a daily news recap. The main agent spawns sub-agents, assigning them news feeds to parse and summarize, and then generates a final overall summary plus analysis. 7 May 2025 eugeneyan.com
- @hamel.bsky.social & @sh-reya.bsky.social are two of the world's best on evals. They've built evals for 35+ AI apps & helped teams ship confidently. Now they'll teach everything they know on building evals that work. Enrollment closes in 4 days. Secret 35% discount code: 30 Apr 2025 maven.com
- The Art of Doing Science and Engineering: Learning to Learn by Richard Hamming only $1.99 for the Kindle version today: 27 Apr 2025 amazon.com
- Surround yourself with people whose "work" is their calling, craft, and play. They are intrinsically motivated, are driven to excel and do what's right, and and get so much shit done just because it's fun. 26 Apr 2025
- Product evals are misunderstood. Many teams think that adding another tool, metric, or llm-as-judge will solve all their problems and save their product. But that just dodges the hard truth and avoids the real work. Here's how to fix your process instead. 23 Apr 2025 eugeneyan.com
- The default state of projects is to drift toward entropy; you need to actively resist & reverse it. 19 Apr 2025
- Interesting paper from Google that challenges a core assumption in translation evaluation—a single metric can measure both accuracy & naturalness. They found that the best systems had neural metrics that did not correlate with human preferences. 18 Apr 2025 arxiv.org
- Had a session with very senior folks on how they build with AI and can’t help thinking there’s no better time to learn, clarify, brainstorm, write, debate, plan, design, code, debug, review, analyze, delegate, play, and in general do more more while doing less with AI—so psyched! 17 Apr 2025
- Great list of what the best devs do, such as: • Read the source, docs, error msgs • Simplify problems, write simple code • Get their hands dirty • Write to share & write well • Have beginner's mind & keep learning • Not afraid to say: I don't know 17 Apr 2025 endler.dev
- @hamel.bsky.social & his wisdom on evals, error analysis, looking at your data is what we need. Here are his 10 Don'ts: • Don't skip error analysis • Don't skip looking at your data • Don't gatekeep who can write prompts • Don't let zero users be a roadblock • Don't be blindsided by criteria drift 16 Apr 2025
- Great example of generate -> validate loop + error analysis > "the most effective route to improve outcomes was brute force: retry steps until they passed or reached a limit. We give the validation errors ... to the LLM and built a loop runner" 15 Apr 2025
- Stumbled on the first(?) RAG in NarrativeQA from 2017. Because books & movies were too large for LSTMs to do Q&A on, they embedded 200-word chunks and retrieved similar snippets to answer questions. "Chunking and cosine similarity retrieval is so 2017." 12 Apr 2025 arxiv.org
- If you were building a Q&A feature (or chatbot) based on very long documents (like books), what evals would you focus on? 9 Apr 2025
- Can't wait for when I can vibe code a production recommender system. Until then, here's some system designs: • Retrieval vs. Ranking: eugeneyan.com/writing/syst... • Real-time retrieval: eugeneyan.com/writing/real... • Personalization: 8 Apr 2025 eugeneyan.com
Link verified 4 Sept 2026. Recent items update automatically from the channel.
GitHub github.com/eugeneyan His open-source repos, including applied-ml.
Recent
- eugeneyan 22 Jun 2026
- semantic-ids-llm — Semantic IDs: How to train an LLM-Recommender Hybrid with steerability and reasoning on recommendations. 15 Sept 2025
- news-agents — 📰 Building News Agents to Summarize News with MCP, Q, and tmux 19 Jul 2025
Show 15 more
- open-llms — 📋 A list of open LLMs available for commercial use. 13 Feb 2025
- align-app 9 Nov 2024
- framework-comparison 9 Sept 2024
- llm-paper-notes — Notes from the Latent Space paper club. Follow along or start your own! 31 Jul 2024
- applied-ml — 📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production. 18 Jul 2024
- papermill-mlflow — 🧪 Simple data science experimentation & tracking with jupyter, papermill, and mlflow. 9 Jul 2024
- recsys-nlp-graph — 🛒 Simple recommender with matrix factorization, graph, and NLP. Beating the regular collaborative filtering baseline. 7 Jul 2024
- discord-llm — Experimenting with LLMs to Research, Reflect, and Plan (LLM assistants, retrieval, and Discord integration) 7 Jul 2024
- obsidian-copilot — 🤖 A prototype assistant for writing and thinking 30 Jun 2024
- workspace-testing 24 Jun 2024
- poc-docker-template — Simple template showing how to set up docker for reproducible data science with Jupyter notebooks. 17 Jun 2024
- applyingml — 📌 Papers, guides, and mentor interviews on applying machine learning for ApplyingML.com—the ghost knowledge of machine learning. 5 Jun 2024
- visualizing-finetunes 27 May 2024
- 1-on-1s — 🌱 1-on-1 questions and resources from my time as a manager. 11 May 2024
- python-collab-template — 🛠 Python project template with unit tests, code coverage, linting, type checking, Makefile wrapper, and GitHub Actions. 14 Apr 2024
Link verified 4 Sept 2026. Recent items update automatically from the channel.
About the English under a post
Some people here publish in a language other than English. Where they do, this site shows a machine translation beneath the post, in this typeface — the site's own, not theirs.
The post itself is never changed, moved or hidden: what is set in the serif above is exactly what the person published, and it is what to quote them on. A translation can be wrong in ways that matter, especially about tone.
Only the post's own words are translated. A quoted post, a linked article and a belief on korrents.com are left in their original language.