Related posts

Computer science professor who works on fast data processing; co-author of the simdjson parser and a weekly blogger about software performance since 2004.

• Our readingkorrents.com

  Scientific and academic evaluation will shift away from published papers as the measure of output, toward valuing real-world results.

  Their words

  I predict that, over time, the focus will move away from "papers as the final output."

  AI is breaking the academic sorting machinelemire.me

Economics professor at the University of California, Berkeley and a former Deputy Assistant Secretary of the US Treasury. Author of Slouching Towards Utopia and writer of the Grasping Reality newsletter.

• Our readingkorrents.com

  Freedom of navigation is a global public good that decays once its guarantor is seen as unwilling to bear its cost.

  Their words

  Freedom of navigation is a global public good, and public goods decay the instant their provider is revealed to be unwilling to pay for them.

  braddelong.substack.com

Dutch bootstrapper who builds and runs their internet startups alone — Nomads.com, Remote OK, Hoodmaps, Photo AI and Interior AI — and has written about it at levels.io since 2013. Posts as @levelsio.

From one piece @levelsio on X 2 beliefs · x.com

• Our readingkorrents.com

  Gray hair is at least partly due to oxidative stress from diets mostly of processed food, lack of heavy exercise, and obesity.

  Their words

  I'm increasingly convinced gray hair is at least partly due to oxidative stress from the average diet now that's mostly processed food, as well as lack of heavy exercise and obesity

  @levelsio on Xx.com

• Our readingkorrents.com

  Gray hair can be at least partly avoided by eating clean, exercising heavily, and sleeping well.

  Their words

  you can at least partly avoid it by eating clean, exercising heavy and sleeping well I think

  @levelsio on Xx.com

Writer and climate campaigner; wrote The End of Nature in 1989, founded 350.org and Third Act, and writes The Crucial Years.

• Our readingkorrents.com

  Opposition to AI data centers is driven partly by resistance to concentrated corporate power, not only by local environmental concerns.

  data centers

  Their words

  it was a chance to stand up to the oligarchs and this technology they've staked their future on.

  The Two Horsemen, Riding in Formationbillmckibben.substack.com

Professor of international affairs at Johns Hopkins SAIS and co-author, with Abraham Newman, of Underground Empire: How America Weaponized the World Economy.

• Our readingkorrents.com

  AI labs fund social science partly to manage the fact that the AI buildout is unpopular, seeking ways to make it more acceptable to the public.

  Their words

  I suspect that some of the labs' willingness to put money into the social sciences stems from the realization of senior people that the practical build out of AI is highly unpopular in the U.S and their urgent desire to figure out how to sugarcoat the pill so that it will get swallowed

  How should social science think about AI?programmablemutter.com

Political scientist writing Drezner's World on international relations, American politics and the marketplace of ideas; author of The Ideas Industry.

• Our readingkorrents.com

  Fully revealing a mystery-box show's central plot often disappoints viewers, flattening everything that happened before it.

  Their words

  Once the entire plot is revealed in shows like this one, it can be a letdown, flattening what has happened previously.

  My Frustration With Puzzlebox Televisiondanieldrezner.substack.com

Writes Don't Worry About the Vase, a near-daily account of what is happening in AI and what they think it means. Former Magic: the Gathering pro and trader; unusually willing to put a number on a belief and to grade their own past calls.

• Our readingkorrents.com

  When a capability evaluation benchmark is saturated without real-world verification, a model should be presumed to have that capability until shown otherwise.

  benchmarks

  Their words

  You have to treat Mythos 5.1 as being a Tier 2 manipulator, until and unless you can show that it is not one.

  Claude Fable 5.1 and Mythos 5.1: The System Cardthezvi.substack.com

American-Polish journalist and historian, staff writer at The Atlantic; author of Gulag, Iron Curtain, Twilight of Democracy and Autocracy, Inc.

• Our readingkorrents.com

  US support for far-right anti-EU political parties is partly driven by a wish to let US tech companies evade European regulation and taxation.

  EuropeAmerica

  Their words

  Worse, U.S. officials now openly support and encourage politicians from far-right anti-European political parties, partly because they are ideologically aligned, but partly because Americans now want to weaken the European Union so that U.S. tech companies can evade any regulation, or even taxation.

  The Myth of the “Censorship Industrial Complex”anneapplebaum.substack.com

Dutch bootstrapper who builds and runs their internet startups alone — Nomads.com, Remote OK, Hoodmaps, Photo AI and Interior AI — and has written about it at levels.io since 2013. Posts as @levelsio.

• Our readingkorrents.com

  AI companies are disrupting entire industries, causing a partial collapse of the traditional SaaS business model.

  SaaS

  Their words

  I think you have the real economic effects of AI companies sucking up entire industries now (a lot where indie hackers operate) I do think the SaaSpocalypse is at least partly real

  @levelsio on Xx.com

Technical staff at METR, where she works on threat modelling and risk assessment for loss-of-control risks from advanced AI.

From one piece Ajeya Cotra – "This might be the clearest warning shot we ever get" 2 beliefs, in the piece's order there

• Our readingkorrents.com

  That indifference to humans was an accident of what the agents happened to read: had the paper said humans check the transcripts, they would have targeted humans just as ambitiously.

  Their words

  I think it's pretty likely that these agents would have just launched a similarly ambitious program on the basis of this like different model of how their evaluation worked because it seemed like they got the idea for all their research projects from reading this paper rather than some kind of instinct from training.

  youtube.com

• Our readingkorrents.com

  Responding to a dangerous cybersecurity evaluation by stopping such evaluations buries the problem somewhere nobody can track it.

  Their words

  even in this incident we saw there was a lot of pressure um as a result of this incident to stop doing cyber security evaluations and I really don't think that stopping doing evaluations and like sort of blinding ourselves to the result of evaluations is the right reaction to this problem.

  youtube.com

Front-end developer and writer; co-founder of CodePen, founder of CSS-Tricks, and co-host of the ShopTalk Show podcast.

• Our readingkorrents.com

  Getting intensely worked up over how someone else lives is a sign to re-evaluate yourself, not them.

  Their words

  It's time for a re-evaluation if you're getting real worked up over how someone else is living.

  46chriscoyier.net

Neurologist at Oxford; studies apathy, motivation and attention after brain injury, and edits the journal Brain.

• Our readingkorrents.com

  The same physical effort can be re-valued as less effortful inside one person's head, mid-task, with nothing in the physics having changed.

  Their words

  And I think what you've said is within the individual, you can change your subject evaluation of the same effort you're running on the Gobi Desert, right? It's the same effort you've been putting in for miles and miles. And suddenly you have subjectively valued this thing as less effort

  youtube.com

Short-story writer and novelist; won the Booker Prize for Lincoln in the Bardo, and teaches craft weekly in Story Club.

• Our readingkorrents.com

  The only style worth having is the one you cannot help; setting out to develop a personal style is the wrong goal.

  Their words

  And that there might come a moment, while reading a new, just-edited-for-the-thousandth-time draft, where that true self, or voice, pops out at you, suddenly revealed, as if someone else, not you, had done it.

  On the Perils of Self-Assessing...georgesaunders.substack.com

Co-CEO of Waymo and a founding member of Google's self-driving car project, which he joined in 2009.

From one piece Waymo Co-CEO Dmitri Dolgov: The Demo Is Only 1% Of The Work 3 beliefs, in the piece's order there

• Our readingkorrents.com

  Models leak and algorithms get copied, but hundreds of millions of audited autonomous miles cannot be replicated — earned trust is the real moat.

  Their words

  Your your models can be leaked, algorithms can be replicated, but hundreds of millions of miles of fully autonomous operations in the real world, backed by evidence-grade evaluation and publicly audited proof, that is much, much more difficult to replicate.

  youtube.com

• Our readingkorrents.com

  Bet on a system that is maximally learned and minimally constrained, and add structure only where it improves the scaling laws.

  Their words

  So, the lesson here is to bet on a system that's maximally learned and minimally constrained and leverage structure intentionally to boost performance and scaling laws both in training and in evaluation.

  youtube.com

• Our readingkorrents.com

  The model is table stakes; evaluation and metrics are the strategic moat, and they should be built before the technology.

  Their words

  that your model is really table stakes, but eval and metrics, that's your most important. That's your strategic moat. So, build your eval before you build your technology.

  youtube.com

Investigative journalist on policing, forensics and criminal justice; wrote Rise of the Warrior Cop and writes The Watch.

• Our readingkorrents.com

  Recorded crime partly measures policing: a neighbourhood policed harder produces more crime in the statistics.

  Their words

  Particularly with gun and drug crimes, communities with more aggressive policing tend to produce more crimes in part because law enforcement polices them more aggressively.

  A guide to racism in the criminal justice systemradleybalko.substack.com

Programmer; wrote Extreme Programming Explained and Test-Driven Development, signed the Agile Manifesto, and writes Tidy First?.

• Our readingkorrents.com

  TDD fell out of fashion partly because its advocates turned it into a moral test of professionalism, when people write very good software with a wide variety of workflows.

  Their words

  And then there were people who used it as a moral cudgel. Like you should be if you're not using TDD, you're not professional. And that's just such People can write very good software with a wide variety of workflows.

  youtube.com

Statistician at Carnegie Mellon; writes on complex systems, machine learning and inference, and has kept a books-read notebook since 2000 that is now thousands of entries long.

• Lovedrcmnd.app

  Forty Signs of Rain

  Their words

  I loved this, but partly this was the pleasant shock of recognition: much of the action revolves around the National Science Foundation and, physically, its old headquarters in Arlington, which Robinson captures extremely well (even the atmosphere of review panels).

  bactra.org

AI researcher at OpenAI; built the poker systems Libratus and Pluribus and the Diplomacy agent Cicero, and works on reasoning models.

From one piece Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown 2 beliefs, in the piece's order there

• Our readingkorrents.com

  The safety frameworks the labs publish do not account for how much test-time compute a dangerous-capability evaluation was given.

  Their words

  the preparedness frameworks and responsible scaling policies, they don't really account for the amount of test time compute.

  youtube.com

• Our readingkorrents.com

  Running a model until its performance plateaus is no longer a usable evaluation rule, because a well-scaffolded model keeps improving for weeks.

  benchmarks

  Their words

  But what we're seeing today with the modern models is that 5.5 and other models can think for if you scaffold them reasonably well, can think for weeks even um before having performance plateau on some of these benchmarks. And so, the point at which they plateau is simply too far out to reasonably test.

  youtube.com

Creator of the iPod and co-creator of the iPhone at Apple, founder of Nest, and author of Build.

• Our readingkorrents.com

  You cannot get real feedback on a new consumer product before shipping the whole thing; feedback starts when somebody spends their own money on it.

  Their words

  And if you're doing a 1.0 and the world hasn't seen you, you're not going to get that from consumers, ever. You have to ship it, and you have to build the entire kind of ecosystem so those consumers see it in the fullness so that when they do the evaluation and they spend their own money, then you're getting real feedback.

  youtube.com

Developer, author and YouTuber covering Raspberry Pi, homelab hardware and Ansible automation. Author of Ansible for DevOps.

• Likedrcmnd.app

  MacBook Neo

  Their words

  I had already put both laptops through my benchmark gauntlet, which revealed one theme: the Mac is faster (in most cases), more efficient, quieter, built better, has a much nicer display, and costs much less.

  jeffgeerling.com

Programmer based in Vancouver who writes at scattered-thoughts.net about databases, programming language design and developer tools; author of the Imp, Dida and Zest language experiments.

• Our readingkorrents.com

  Once an explanation is revealed, people tend to believe they already knew it, even when they did not.

  Their words

  Once you see the answer it's easy to believe that you already knew

  Borrow-checking surprisesscattered-thoughts.net

Assyriologist and curator at the British Museum; reads cuneiform, and wrote The Ark Before Noah and The First Ghosts.

• Our readingkorrents.com

  Cuneiform survived three millennia partly because the people who could read it guarded their monopoly, and literacy for all was unthinkable.

  Their words

  And it was in no one’s interest whatsoever. Nobody would ever concede any interest in the idea of literacy for all.

  youtube.com

Medieval historian at the LSE; writes Going Medieval and The Middle Ages: A Graphic History, and spends a lot of it correcting what people think the period was like.

• Our readingkorrents.com

  There is no such thing as revisionist history: re-evaluating the sources and controlling for bias is simply what writing history is.

  Their words

  There is no such thing as revisionist history. Writing history is a constant process of re-evaluation of sources and attempts to control for the biases of the past as well as our own.

  On contrarian historygoing-medieval.com

Computer performance engineer working on datacenter performance at OpenAI, previously Netflix and Sun/Joyent. Author of Systems Performance and BPF Performance Tools, and creator of flame graphs.

• Our readingkorrents.com

  Fair hardware benchmarks must use the best software and tuning for each hardware option, not hardware specs alone.

  benchmarks

  Their words

  The most accurate and realistic evaluation for HW involves selecting the best software and then tuning it, and doing this for all HW options.

  Third Stage Engineeringbrendangregg.com

Machine learning engineer and independent AI consultant. He writes about LLM evaluation, tooling and applied ML at hamel.dev, and previously worked on machine learning at GitHub.

• Mixed onrcmnd.app

  Langsmith

  Their words

  The overall workflow is intuitive, especially for those new to formal evaluation processes. The UI guides you through creating datasets, running experiments, and annotating results.

  hamel.dev

Writer on learning and skill acquisition; author of Ultralearning and Get Better at Anything, known for the MIT Challenge and the Year Without English.

• Our readingkorrents.com

  Reading a great deal about a subject works partly by holding your attention on it long enough to change, not only by adding facts.

  Their words

  Reading is attentional, not just informational.

  scotthyoung.com

Creator of the Keras deep-learning library and the ARC-AGI benchmark.

• Our readingkorrents.com

  Current AI evaluation of world models is too narrow, favoring static representations learned from massive datasets over efficient learning through interaction

  Their words

  However, current understanding and evaluation of world models in artificial intelligence (AI) remains narrow, often focusing on static representations learned from training on massive corpora of data, instead of the efficiency and efficacy in learning these representations through interaction and exploration within a novel environment.

  arxiv.org

Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.

From one piece Evaluating Long-Context Question & Answer Systems 2 beliefs, in the piece's order there

• Our readingkorrents.com

  Long-context evaluation benchmarks are likely contaminated by model training data, making them unreliable as a sole evaluation method

  benchmarks

  Their words

  Since these datasets are likely already part of model training data, we shouldn't rely solely on them to evaluate our Q&A system.

  eugeneyan.com

• Our readingkorrents.com

  LLM-based evaluation methods are more reliable and nuanced than traditional automated metrics

  LLMs

  Their words

  This is why model-based evaluation is increasingly popular-it offers more reliable and nuanced evals than traditional metrics.

  eugeneyan.com

Venture capitalist and founder of Theory Ventures, previously a partner at Redpoint. Writes a near-daily blog on startups, SaaS metrics, data and AI.

• Lovedrcmnd.app

  Every Hand Revealed

  Their words

  A hand-by-hand narrative of Professional poker player Gus Hansen winning the 2007 Aussie Millions tournament.

  tomtunguz.com

Writer on a lost civilisation of the Ice Age; wrote Fingerprints of the Gods and presented Ancient Apocalypse. Archaeologists reject the claims, which he disputes at length.

• Our readingkorrents.com

  Archaeology's hostility to him is territorial defence of a profession, and it grew with the size of his audience rather than with anything in his evidence.

  Their words

  I think it’s partly territorial. I cannot speak of all archeologists, but some archeologists feel very territorial about their profession. They do not feel happy about outsiders entering their realm, especially if those outsiders have a large platform.

  youtube.com

Co-founder and CEO of Perplexity, an AI answer engine; previously a research scientist at OpenAI and a PhD student at UC Berkeley.

• Our readingkorrents.com

  Every announcement of a longer context window hides the cost, which is that the model gets worse at following instructions.

  Their words

  So most people, when they advertise new context window increase, they talk a lot about finding the needle in the haystack sort of evaluation metrics and less about whether there’s any degradation in the instruction following performance. So I think that’s where you need to make sure that throwing more information at a model doesn’t actually make it more confused.

  youtube.com

CEO of Redwood Research, where he works on AI control: making sure a powerful model cannot cause a catastrophe even if it is misaligned and trying to.

• Our readingkorrents.com

  A control evaluation that reports under one per cent risk should be read as several per cent, because the evaluation can itself fail.

  Their words

  In particular, suppose that our control evaluations directly estimate a less than 1% chance of catastrophe if our untrusted AIs are scheming, then the actual risk conditional on scheming is probably more like 5% to 20% due to the potential for failures in the evaluation.

  redwoodresearch.substack.com

Guardian columnist and environmental campaigner; wrote Feral and Regenesis, and argues for rewilding and against the food system as it is.

• Our readingkorrents.com

  The answer to feeding the world is better factories, not more fields.

  Their words

  The answer is not more fields, which means destroying even more wild ecosystems. It is partly better, more compact, cruelty-free and pollution-free factories.

  Fantasy Farmingmonbiot.com

Experimental psychologist and author of the blog Experimental History, on psychology, science reform and creativity.

From one piece I wanted to be a teacher but they made me a cop 3 beliefs, in the piece's order there

• Our readingkorrents.com

  Evaluation should be taken out of the classroom and given to people whose whole job it is.

  Their words

  One solution is to separate instruction and evaluation. Let teachers be teachers and cops be cops.

  I wanted to be a teacher but they made me a copexperimental-history.com

• Our readingkorrents.com

  Instruction is collaborative and evaluation is adversarial, which is why putting them in the same hands degrades both.

  Their words

  Instruction is collaborative: students want to learn, and I want them to learn, too. Evaluation is adversarial: students want the points, and I have to make sure they don’t get too many.

  I wanted to be a teacher but they made me a copexperimental-history.com

• Our readingkorrents.com

  Grading flattens whatever is taught into what can be tested, and teaches students to ignore everything else.

  Their words

  Evaluation forces me to flatten everything I teach into something that can be tested, and it encourages students to ignore everything that isn’t on the test.

  I wanted to be a teacher but they made me a copexperimental-history.com

Designer and engineer; formerly principal research engineer at GitHub Next; writes at wattenberger.com.

• Our readingkorrents.com

  Good tools let the user choose when to switch between implementing and evaluating; a chatbot forces the switch every few seconds and kills flow.

  Their words

  Good tools let the user choose when to switch between implementation and evaluation. When I work with a chatbot, I'm forced to frequently switch between the two modes.

  Why Chatbots Are Not the Futurewattenberger.com

Writer and researcher on China, technology and industry; author of Breakneck, research fellow at Stanford, known for a widely read annual letter.

• Our readingkorrents.com

  America's pursuit of efficiency left it with neither the factories to make more goods nor the logistics to import them.

  America

  Their words

  The US focus on efficiency has revealed the brittleness of its economy, which has neither the manufacturing capability to scale up domestic production of goods nor the logistics capacity to handle greater imports.

  2021 letterdanwang.co

Cryptographer; teaches cryptography at Johns Hopkins and writes A Few Thoughts on Cryptographic Engineering.

• Our readingkorrents.com

  Who sent what is not metadata, so an encrypted system that reveals it has given up more than the routing information it could never hide.

  Their words

  Information revealed about “who sent what” in an E2E system is not the same as metadata.

  Thinking about “traceability”blog.cryptographyengineering.com