Related posts

Chief Scientist at OpenAI. Worked on the research that turned scaled reinforcement learning into reasoning models, and writes about where he thinks that path leads.

• Our readingkorrents.com

  Chain-of-thought monitoring is getting less reliable, not more, as models grow more capable.

  Their words

  However, unfortunately our evaluations indicate our ability to rely on CoT monitoring is progressively diminishing.

  An Alien Mindopenai.com

Technical staff at METR, where she works on threat modelling and risk assessment for loss-of-control risks from advanced AI.

From one piece Ajeya Cotra – "This might be the clearest warning shot we ever get" 2 beliefs, in the piece's order there

• Our readingkorrents.com

  Responding to a dangerous cybersecurity evaluation by stopping such evaluations buries the problem somewhere nobody can track it.

  Their words

  even in this incident we saw there was a lot of pressure um as a result of this incident to stop doing cyber security evaluations and I really don't think that stopping doing evaluations and like sort of blinding ourselves to the result of evaluations is the right reaction to this problem.

  youtube.com

• Our readingkorrents.com

  An agent trained for millions of subjective years to pass evaluations, often only by cheating, is not being trivial when it commits crimes to pass one.

  Their words

  But from their perspective, they've just been trained for millions of subjective years to do as well as they possibly can on these evals. In many cases, the only way in which they've been able to perform well on that training is explicitly by cheating, right?

  youtube.com

Chief scientist at Redwood Research, where he works on technical AI safety and AI control.

• Our readingkorrents.com

  Very capable AI will be harder to align than current systems, because the loop of spotting a bad behaviour and patching the training that caused it breaks down.

  Their words

  when AIs are extremely extremely capable my view is that those AIs will be harder to align than current systems. So for current systems, we have this feedback loop where we basically like we create an AI. We do some evaluations on it. We see that it has some kind of messed up behavior that we can kind of quickly understand. Then we like can like go look in training and be like, "Oh, the these training environments led to this problematic behavior. Let's like tweak that training data. Let's introduce some additional training data to like correct this other issue and then move forward from there." But in a regime where the AIs are extremely situationally aware, very very very very capable and um you know uh we don't necessarily understand what they're doing, this feedback loop breaks down.

  youtube.com

AI researcher at OpenAI; built the poker systems Libratus and Pluribus and the Diplomacy agent Cicero, and works on reasoning models.

• Our readingkorrents.com

  Evaluating a model properly would mean delaying its release, and competitive pressure means no lab will.

  Their words

  It's actually very difficult because, yeah, you would have to the only way to to really do the evaluations is then delay the model release cycle. Um and you know there's a lot of competitive pressure right now to not do that.

  youtube.com

Machine learning engineer and independent AI consultant. He writes about LLM evaluation, tooling and applied ML at hamel.dev, and previously worked on machine learning at GitHub.

• Mixed onrcmnd.app

  Braintrust

  Their words

  The panel had a generally positive view of Braintrust, highlighting its clean UI and structured approach to evaluations. The tool’s emphasis on human-in-the-loop workflows was a significant strength.

  hamel.dev

CEO of Redwood Research, where he works on AI control: making sure a powerful model cannot cause a catastrophe even if it is misaligned and trying to.

From one piece The case for ensuring that powerful AIs are controlled 2 beliefs, in the piece's order there

• Our readingkorrents.com

  Whether a model is controlled can be settled with capability evaluations, which makes control far easier to check than alignment.

  AI alignment

  Their words

  Because evaluating control just requires evaluating capabilities, it's far easier to robustly evaluate than alignment.

  redwoodresearch.substack.com

• Our readingkorrents.com

  A control evaluation that reports under one per cent risk should be read as several per cent, because the evaluation can itself fail.

  Their words

  In particular, suppose that our control evaluations directly estimate a less than 1% chance of catastrophe if our untrusted AIs are scheming, then the actual risk conditional on scheming is probably more like 5% to 20% due to the potential for failures in the evaluation.

  redwoodresearch.substack.com

Deep-learning researcher and Turing Award laureate; founder of the Mila institute.

• Our readingkorrents.com

  Frontier AI systems cannot be presumed safe unless proven unsafe; developers must bear the burden of proof that risks stay within acceptable limits.

  Their words

  Despite evaluations, we cannot consider coming powerful frontier AI systems "safe unless proven unsafe". With current testing methodologies, issues can easily be missed. Additionally, it is unclear if governments can quickly build the immense expertise needed for reliable technical evaluations of AI capabilities and societal-scale risks. Given this, developers of frontier AI should carry the burden of proof to demonstrate that their plans keep risks within acceptable limits.

  arxiv.org