Eugene Yan

• Our readingkorrents.com

  The surrounding system framework and scaffolding matter more for AI agent capability than the underlying model.

  Their words

  The experiments showed that the system framework matters far more than the underlying model.

  Patterns for Building Cybersecurity Evalseugeneyan.com

From one piece How to Work and Compound with AI 3 beliefs, in the piece's order there

• Our readingkorrents.com

  As AI agents take on bigger delegated tasks, the human bottleneck shifts to writing specs and reviewing output fast, hollowing out the middle of the work.

  AI agents

  Their words

  The bottleneck has shifted from doing the work to writing clear specs and reviewing outputs fast enough to keep the pipeline moving-the middle is hollowing out.

  How to Work and Compound with AIeugeneyan.com

• Our readingkorrents.com

  Delegating a task to an AI agent only works if the output can be verified, which requires defining success criteria first.

  Their words

  You can't delegate what you can't verify, so this requires first defining success criteria and metrics.

  How to Work and Compound with AIeugeneyan.com

• Our readingkorrents.com

  The practices that make someone effective at working with an AI model are the same practices that make someone effective at onboarding any new human collaborator.

  Their words

  none of this is specific to AI. It's simply how you onboard and work with any new collaborator.

  How to Work and Compound with AIeugeneyan.com

From one piece Product Evals in Three Simple Steps 3 beliefs, in the piece's order there

• Our readingkorrents.com

  Human annotators can miss up to half of defects due to fatigue when reviewing large volumes of samples.

  Their words

  And human annotators can miss as many as 50% of the defects due to fatigue after looking at hundreds of samples.

  Product Evals in Three Simple Stepseugeneyan.com

• Our readingkorrents.com

  The main advantage of an LLM evaluator over human annotators is scalability, not higher accuracy.

  LLMs

  Their words

  In my opinion, the true benefit isn't higher accuracy than human annotators-it's scalability.

  Product Evals in Three Simple Stepseugeneyan.com

• Our readingkorrents.com

  The right bar for an LLM evaluator is human-level performance, not perfect accuracy.

  LLMsbenchmarks

  Their words

  The benchmark is human performance, not perfection. We sometimes get requirements for 90%+ accuracy.

  Product Evals in Three Simple Stepseugeneyan.com

From one piece Advice for New Principal Tech ICs (i.e., Notes to Myself) 3 beliefs, in the piece's order there

• Our readingkorrents.com

  Reaching a senior level requires putting oneself on the critical path, but excelling beyond it requires removing oneself from it.

  Their words

  To get to principal, you need to put yourself on the critical path. To be effective as a principal and go beyond it, you need to actively remove yourself from it.

  eugeneyan.com

• Our readingkorrents.com

  Greater autonomy over one's work comes with greater accountability for its outcomes.

  Their words

  With great freedom comes great responsibility. You have the autonomy to choose what to work on, but there's the expectation of accountability and impact.

  eugeneyan.com

• Our readingkorrents.com

  Convincing others to care and act matters more than simply being right.

  Their words

  Being right is less than half the battle. You also have to convince others that you're right, and more importantly, convince them to care enough to act on it.

  eugeneyan.com

From one piece Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs 2 beliefs, in the piece's order there

• Our readingkorrents.com

  Traditional recommender systems predict user behavior well but cannot be steered via natural language or explain their own reasoning.

  Their words

  They excel at predicting what a user will click or buy next, but can't be steered via natural language or reason on their choices.

  eugeneyan.com

• Our readingkorrents.com

  Language models lack catalog-specific knowledge, so their recommendations tend to be generic and biased toward popular items.

  LLMs

  Their words

  language models have world knowledge and can eloquently talk about products, but are unaware of our catalog. Also, their recommendations are generic and suffer from popularity bias.

  eugeneyan.com

From one piece Evaluating Long-Context Question & Answer Systems 3 beliefs, in the piece's order there

• Our readingkorrents.com

  LLM-based evaluation methods are more reliable and nuanced than traditional automated metrics

  LLMs

  Their words

  This is why model-based evaluation is increasingly popular-it offers more reliable and nuanced evals than traditional metrics.

  eugeneyan.com

• Our readingkorrents.com

  People find pairwise comparisons of answers easier and more consistent to judge than absolute ratings

  Their words

  People find comparing two answers easier than assigning absolute ratings, resulting in greater consistency across annotators.

  eugeneyan.com

• Our readingkorrents.com

  Long-context evaluation benchmarks are likely contaminated by model training data, making them unreliable as a sole evaluation method

  benchmarks

  Their words

  Since these datasets are likely already part of model training data, we shouldn't rely solely on them to evaluate our Q&A system.

  eugeneyan.com

• Usesaffiliate linkrcmnd.app

  Warp

  Their words

  Trying Warp instead of my usual iTerm

  eugeneyan.com

• Usesrcmnd.app

  Fish

  Their words

  Trying Fish instead of my usual Oh My Zsh

  eugeneyan.com

• Mixed onrcmnd.app

  Rectangle

  Their words

  While Raycast already has window management, Rectangle lets you hit the hotkey again

  eugeneyan.com

+ 2 more

• Usesrcmnd.app

  Raycast

  Their words

  I've since moved to Raycast window management.

  eugeneyan.com

• Usesrcmnd.app

  MX Ergo

  Their words

  sign in to load saved settings for my MX Ergo

  eugeneyan.com