Ming-Wei Chang
One of the four authors of the 2018 paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", at Google AI Language, the affiliation printed on the paper.
Ming-Wei Chang did not write this page. ppll compiled it from their own words, each linked to the source. Is this you? · Report a mistake
What they believe
-
Current techniques restrict pre-trained representation power because standard language models are unidirectional.
We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.
arxiv.orgSaid 11 Oct 2018This on korrents
The bold line is korrents’ wording; the quote is theirs. These one: what they wrote before what they said aloud, one per source, newest first.
Beliefs
Korrents What they believe 4 beliefs — each backed by an exact quote.
Each is a — compiled by korrents.com, not by them: the one-line wordings are korrents', the quotes are theirs.
Recent
Current techniques restrict pre-trained representation power because standard language models are unidirectional.
We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Said 11 Oct 2018
Unidirectional restrictions are sub-optimal for sentence-level tasks and harmful for token-level tasks that need bidirectional context.
Such restrictions are sub-optimal for sentence-level tasks, and could be very harmful when applying fine-tuning based approaches to token-level tasks such as question answering, where it is crucial to incorporate context from both directions.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Said 11 Oct 2018
Pre-trained representations reduce the need for many heavily-engineered task-specific architectures.
We show that pre-trained representations reduce the need for many heavily-engineered task-specific architectures.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Said 11 Oct 2018
Show 1 more
A deep bidirectional model is strictly more powerful than a left-to-right model or a shallow concatenation of unidirectional models.
Intuitively, it is reasonable to believe that a deep bidirectional model is strictly more powerful than either a left-to-right model or the shallow concatenation of a left-to-right and a right-to-left model.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Said 11 Oct 2018
Beliefs others hold too
Current techniques restrict pre-trained representation power because standard language models are unidirectional. 4 hold this
We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Said 11 Oct 2018
Unidirectional restrictions are sub-optimal for sentence-level tasks and harmful for token-level tasks that need bidirectional context. 4 hold this
Such restrictions are sub-optimal for sentence-level tasks, and could be very harmful when applying fine-tuning based approaches to token-level tasks such as question answering, where it is crucial to incorporate context from both directions.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Said 11 Oct 2018
Pre-trained representations reduce the need for many heavily-engineered task-specific architectures. 4 hold this
We show that pre-trained representations reduce the need for many heavily-engineered task-specific architectures.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Said 11 Oct 2018
What is a korrent?
A korrent is a belief a person has stated in their own words: one sentence stating the claim, backed by a quote and a source, kept at korrents.com.
Under a name here, the quoted block is what they actually said. The korrent beneath it is the claim those words support, in korrents' wording — tap it to see the record, its source, and who else holds it.
Nobody here wrote their own korrents. They are compiled from public statements, and a person can change their mind, which is recorded too.
About the English under a post
Some people here publish in a language other than English. Where they do, this site shows a machine translation beneath the post, in this typeface — the site's own, not theirs.
The post itself is never changed, moved or hidden: what is set in the serif above is exactly what the person published, and it is what to quote them on. A translation can be wrong in ways that matter, especially about tone.
Only the post's own words are translated. A quoted post, a linked article and a belief on korrents.com are left in their original language.