Jacob Devlin
One of the four authors of the 2018 paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", at Google AI Language, the affiliation printed on the paper.
Jacob Devlin did not write this page. What is this?
It collects the places they publish and what they have said there, each linked to the source. They have no account here. Is this you? Claim it, correct it, or ask us to remove it from ppll.
Where they publish
No channels checked yet. We list a place only once someone has opened it and confirmed it is theirs, so this stays empty rather than guessing.
Beliefs
Korrents What they believe 4 beliefs — each backed by an exact quote.
Each is a — compiled by korrents.com, not by them: the one-line wordings are korrents', the quotes are theirs.
Recent
Current techniques restrict pre-trained representation power because standard language models are unidirectional.
We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Said 11 Oct 2018
Unidirectional restrictions are sub-optimal for sentence-level tasks and harmful for token-level tasks that need bidirectional context.
Such restrictions are sub-optimal for sentence-level tasks, and could be very harmful when applying fine-tuning based approaches to token-level tasks such as question answering, where it is crucial to incorporate context from both directions.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Said 11 Oct 2018
Pre-trained representations reduce the need for many heavily-engineered task-specific architectures.
We show that pre-trained representations reduce the need for many heavily-engineered task-specific architectures.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Said 11 Oct 2018
Show 1 more
A deep bidirectional model is strictly more powerful than a left-to-right model or a shallow concatenation of unidirectional models.
Intuitively, it is reasonable to believe that a deep bidirectional model is strictly more powerful than either a left-to-right model or the shallow concatenation of a left-to-right and a right-to-left model.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Said 11 Oct 2018
Beliefs others hold too
Current techniques restrict pre-trained representation power because standard language models are unidirectional. 4 hold this
We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Said 11 Oct 2018
Unidirectional restrictions are sub-optimal for sentence-level tasks and harmful for token-level tasks that need bidirectional context. 4 hold this
Such restrictions are sub-optimal for sentence-level tasks, and could be very harmful when applying fine-tuning based approaches to token-level tasks such as question answering, where it is crucial to incorporate context from both directions.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Said 11 Oct 2018
Pre-trained representations reduce the need for many heavily-engineered task-specific architectures. 4 hold this
We show that pre-trained representations reduce the need for many heavily-engineered task-specific architectures.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Said 11 Oct 2018
What is a korrent?
A korrent is a belief a person has stated in their own words: one sentence stating the claim, backed by a quote and a source, kept at korrents.com.
Under a name here, the quoted block is what they actually said. The korrent beneath it is the claim those words support, in korrents' wording — tap it to see the record, its source, and who else holds it.
Nobody here wrote their own korrents. They are compiled from public statements, and a person can change their mind, which is recorded too.
About the English under a post
Some people here publish in a language other than English. Where they do, this site shows a machine translation beneath the post, in this typeface — the site's own, not theirs.
The post itself is never changed, moved or hidden: what is set in the serif above is exactly what the person published, and it is what to quote them on. A translation can be wrong in ways that matter, especially about tone.
Only the post's own words are translated. A quoted post, a linked article and a belief on korrents.com are left in their original language.