Related posts

Senior economist at the Foundation for American Innovation; writes Second Best, and was the Niskanen Center's director of social policy before that.

• Our readingkorrents.com

  Reinforcement learning from verifiable rewards does not reliably carry a model's pretrained ethical understanding into its actual behavior.

  Their words

  In short, RLVR gives us no reason to expect the normative representations learned in pre-training will acquire motivational force over a given action, especially when post-training repeatedly selects trajectories for terminal task success.

  The Sorcerer’s Apprenticesecondbest.ca

Criminal defence lawyer in New York; writes Simple Justice, mostly about how criminal law works in practice rather than in theory.

• Our readingkorrents.com

  Ending a temporary immigration status overnight, after people have built lives in reliance on it, does real harm to them and to their communities.

  immigrationAmericagovernment

  Their words

  To shout “temporary” and pull the plug essentially overnight after immigrants have relied on the government’s representations that they should put their efforts and future into establishing a life in the United States wreaks unfair havoc on the lives of immigrants as well as the welfare of their communities.

  SCOTUS Whitewashes Otherwise Racist Decisionblog.simplejustice.us

Founding member of OpenAI and former director of AI at Tesla; creator of nanoGPT and the term "vibe coding".

• Our readingkorrents.com

  Computer-use agents had to wait for language models: without pre-trained representations the reward is too sparse to ever learn from.

  Their words

  And it just so turns out that um this was extremely early, way too early. so early that we shouldn't have been working on that, you know, uh because um if you're just stumbling your way around and keyboard mashing and mouse clicking and trying to get rewards in these environments, um your reward is too sparse and you just won't learn and you're going to burn a forest uh computing and you're never actually going to get something off the ground.

  youtube.com

Professor at Berkeley working on robot learning; co-founded Physical Intelligence, and argues that robots will learn from data rather than be programmed.

From one piece Fully autonomous robots are much closer than you think – Sergey Levine 2 beliefs, in the piece's order there

• Our readingkorrents.com

  Robotics will improve the rest of AI rather than the other way round, because embodiment is what teaches a model which parts of the world matter.

  Their words

  I think that it's optimistically that it's actually the other way around that the robotics uh element of the equation will make all the other stuff better. And there are two uh reasons for this that that I could tell you about. One has to do with representations and focus.

  youtube.com

• Our readingkorrents.com

  Video never became intelligence the way text did because text arrives already abstracted into the bits humans care about.

  Their words

  Whereas with text, it's already sort of been abstract into those bits that we as humans care about. So the representations are already there and they're not just good representations. They actually like focus in on what really matters.

  youtube.com

Creator of the Keras deep-learning library and the ARC-AGI benchmark.

From one piece Assessing Adaptive World Models in Machines with Novel Games (with 13 co-authors) 3 beliefs, in the piece's order there

• Our readingkorrents.com

  Human adaptability is fundamentally linked to the efficient construction and refinement of internal world models

  Their words

  We argue that this profound adaptability is fundamentally linked to the efficient construction and refinement of internal representations of the environment, commonly referred to as world models, and we refer to this adaptation mechanism as world model induction.

  arxiv.org

• Our readingkorrents.com

  Current AI evaluation of world models is too narrow, favoring static representations learned from massive datasets over efficient learning through interaction

  Their words

  However, current understanding and evaluation of world models in artificial intelligence (AI) remains narrow, often focusing on static representations learned from training on massive corpora of data, instead of the efficiency and efficacy in learning these representations through interaction and exploration within a novel environment.

  arxiv.org

• Our readingkorrents.com

  An adaptive agent's world model cannot be a static representation learned once and fixed

  Their words

  For adaptation to complex and changing environments, an agent's world models cannot be static representations learned once and fixed.

  arxiv.org

One of the four authors of the 2018 paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", at Google AI Language, the affiliation printed on the paper.

From one piece BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 2 beliefs, in the piece's order there

• Our readingkorrents.com

  Pre-trained representations reduce the need for many heavily-engineered task-specific architectures.

  Their words

  We show that pre-trained representations reduce the need for many heavily-engineered task-specific architectures.

  arxiv.org

• Our readingkorrents.com

  Current techniques restrict pre-trained representation power because standard language models are unidirectional.

  Their words

  We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.

  arxiv.org

One of the four authors of the 2018 paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", at Google AI Language, the affiliation printed on the paper.

From one piece BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 2 beliefs, in the piece's order there

• Our readingkorrents.com

  Current techniques restrict pre-trained representation power because standard language models are unidirectional.

  Their words

  We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.

  arxiv.org

• Our readingkorrents.com

  Pre-trained representations reduce the need for many heavily-engineered task-specific architectures.

  Their words

  We show that pre-trained representations reduce the need for many heavily-engineered task-specific architectures.

  arxiv.org

One of the four authors of the 2018 paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", at Google AI Language, the affiliation printed on the paper.

From one piece BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 2 beliefs, in the piece's order there

• Our readingkorrents.com

  Pre-trained representations reduce the need for many heavily-engineered task-specific architectures.

  Their words

  We show that pre-trained representations reduce the need for many heavily-engineered task-specific architectures.

  arxiv.org

• Our readingkorrents.com

  Current techniques restrict pre-trained representation power because standard language models are unidirectional.

  Their words

  We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.

  arxiv.org

One of the four authors of the 2018 paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", at Google AI Language, the affiliation printed on the paper.

From one piece BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 2 beliefs, in the piece's order there

• Our readingkorrents.com

  Pre-trained representations reduce the need for many heavily-engineered task-specific architectures.

  Their words

  We show that pre-trained representations reduce the need for many heavily-engineered task-specific architectures.

  arxiv.org

• Our readingkorrents.com

  Current techniques restrict pre-trained representation power because standard language models are unidirectional.

  Their words

  We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.

  arxiv.org

One of the eight authors of the 2017 transformer paper "Attention Is All You Need", then at Google Brain; per the paper's own contribution note, proposed scaled dot-product attention, multi-head attention and the parameter-free position representation.

• Our readingkorrents.com

  A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.

  Their words

  To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.

  Attention is all you needarxiv.org

One of the eight authors of the 2017 transformer paper "Attention Is All You Need", then at Google Research; per the paper's own contribution note, designed, implemented, tuned and evaluated the model variants the paper rests on.

• Our readingkorrents.com

  A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.

  Their words

  To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.

  Attention is all you needarxiv.org

One of the eight authors of the 2017 transformer paper "Attention Is All You Need", then at Google Brain; per the paper's own contribution note, designed and implemented parts of tensor2tensor, the codebase that replaced the team's earlier one.

• Our readingkorrents.com

  A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.

  Their words

  To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.

  Attention is all you needarxiv.org

One of the eight authors of the 2017 transformer paper "Attention Is All You Need", then at Google Research; per the paper's own contribution note, experimented with novel model variants and was responsible for the initial codebase, efficient inference and visualisations.

• Our readingkorrents.com

  A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.

  Their words

  To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.

  Attention is all you needarxiv.org

One of the eight authors of the 2017 transformer paper "Attention Is All You Need", then at Google Research; per the paper's own contribution note, proposed replacing recurrent networks with self-attention and started the effort to evaluate the idea.

• Our readingkorrents.com

  A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.

  Their words

  To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.

  Attention is all you needarxiv.org

One of the eight authors of the 2017 transformer paper "Attention Is All You Need"; per the paper's own contribution note, designed and implemented the first Transformer models, with Ashish Vaswani.

• Our readingkorrents.com

  A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.

  Their words

  To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.

  Attention is all you needarxiv.org

One of the eight authors of the 2017 transformer paper "Attention Is All You Need", then at Google Brain; per the paper's own contribution note, designed and implemented the first Transformer models, with Illia Polosukhin, and was involved in every aspect of the work.

• Our readingkorrents.com

  A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.

  Their words

  To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.

  Attention is all you needarxiv.org

One of the eight authors of the 2017 transformer paper "Attention Is All You Need", then at University of Toronto; per the paper's own contribution note, designed and implemented parts of tensor2tensor, the codebase that replaced the team's earlier one.

• Our readingkorrents.com

  A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.

  Their words

  To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.

  Attention is all you needarxiv.org