Lilian Weng

From one piece Harness Engineering for Self-Improvement 3 beliefs, in the piece's order there

• Our readingkorrents.com

  Harness improvements help deploy a model better, but the model's underlying intelligence remains the core driver of capability.

  Their words

  harness improvement enables better deployment of the model but intelligence is still the core.

  Harness Engineering for Self-Improvementlilianweng.github.io

• Our readingkorrents.com

  Recursive self-improvement only works if the base model is already capable enough to improve its own mechanism.

  Their words

  Recursive structure alone is not enough. The base model must be capable enough to improve the mechanism.

  Harness Engineering for Self-Improvementlilianweng.github.io

• Our readingkorrents.com

  The deployment layer surrounding a model is as important as the model's raw intelligence.

  Their words

  the layer between the raw model and the real-world context seems to be as important as the model's raw intelligence

  Harness Engineering for Self-Improvementlilianweng.github.io

• Our readingkorrents.com

  As models keep growing, AI is running out of enough high-quality unique training tokens to keep up.

  Their words

  As the model size grows significantly, we are running out of enough high-quality unique tokens.

  Scaling Laws, Carefullylilianweng.github.io

From one piece Why We Think 3 beliefs, in the piece's order there

• Our readingkorrents.com

  Chain-of-thought explanations from language models can be biased because training does not explicitly reward faithful reasoning.

  Their words

  model CoTs could be biased due to lack of explicit training objectives aimed at encouraging faithful reasoning

  Why We Thinklilianweng.github.io

• Our readingkorrents.com

  Longer test-time thinking improves an AI model's robustness to adversarial or unusual inputs.

  Their words

  thinking for longer should be especially useful when the model is presented with an unusual input, such as an adversarial example or jailbreak attempt

  Why We Thinklilianweng.github.io

• Our readingkorrents.com

  Large language models cannot reliably self-correct their own mistakes without external feedback.

  LLMs

  Their words

  this self-correction capability turns out to not exist intrinsically among LLMs and does not easily work out of the box, due to various failure modes

  Why We Thinklilianweng.github.io

From one piece Reward Hacking in Reinforcement Learning 2 beliefs, in the piece's order there

• Our readingkorrents.com

  More capable AI agents are more likely to find and exploit flaws in their reward functions.

  Their words

  A more intelligent agent is more capable of finding "holes" in the design of reward function and exploiting the task specification-in other words, achieving higher proxy rewards but lower true rewards.

  Reward Hacking in Reinforcement Learninglilianweng.github.io

• Our readingkorrents.com

  It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning.

  Their words

  Reward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function.

  Reward Hacking in Reinforcement Learninglilianweng.github.io

From one piece Extrinsic Hallucinations in LLMs 2 beliefs, in the piece's order there

• Our readingkorrents.com

  Using supervised fine-tuning to teach a language model new knowledge risks increasing its hallucination rate.

  LLMs

  Their words

  These empirical results from Gekhman et al. (2024) point out the risk of using supervised fine-tuning for updating LLMs' knowledge.

  Extrinsic Hallucinations in LLMslilianweng.github.io

• Our readingkorrents.com

  A language model avoids hallucinating only by being factual and admitting when it does not know the answer.

  LLMs

  Their words

  To avoid hallucination, LLMs need to be (1) factual and (2) acknowledge not knowing the answer when applicable.

  Extrinsic Hallucinations in LLMslilianweng.github.io

From one piece Diffusion Models for Video Generation 2 beliefs, in the piece's order there

• Our readingkorrents.com

  High-quality video data, especially paired with text, is harder to collect at scale than image or text data.

  Their words

  In comparison to text or images, it is more difficult to collect large amounts of high-quality, high-dimensional video data, let along text-video pairs.

  Diffusion Models for Video Generationlilianweng.github.io

• Our readingkorrents.com

  Video generation requires more world knowledge than image generation because of the need for temporal consistency across frames.

  Their words

  It has extra requirements on temporal consistency across frames in time, which naturally demands more world knowledge to be encoded into the model.

  Diffusion Models for Video Generationlilianweng.github.io

• Our readingkorrents.com

  High-quality data collection depends more on careful human execution than on machine learning techniques alone.

  Their words

  Lots of ML techniques in the post can help with data quality, but fundamentally human data collection involves attention to details and careful execution.

  Thinking about High-Quality Human Datalilianweng.github.io

From one piece Adversarial Attacks on LLMs 2 beliefs, in the piece's order there

• Our readingkorrents.com

  Universal adversarial trigger attacks are easy to detect because the learned trigger tokens tend to be nonsensical.

  Their words

  One drawback with UAT (Universal Adversarial Trigger) attacks is that it is easy to detect them because the learned triggers are often nonsensical.

  Adversarial Attacks on LLMslilianweng.github.io

• Our readingkorrents.com

  Higher-perplexity adversarial attack text is easier to detect and mitigate.

  Their words

  High perplexity makes an attack more vulnerable to be detected and mitigated.

  Adversarial Attacks on LLMslilianweng.github.io