From one piece BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 4 beliefs, in the piece's order there
-
Their words
Intuitively, it is reasonable to believe that a deep bidirectional model is strictly more powerful than either a left-to-right model or the shallow concatenation of a left-to-right and a right-to-left model.
-
korrents.com
Pre-trained representations reduce the need for many heavily-engineered task-specific architectures.Their words
We show that pre-trained representations reduce the need for many heavily-engineered task-specific architectures.
-
Their words
Such restrictions are sub-optimal for sentence-level tasks, and could be very harmful when applying fine-tuning based approaches to token-level tasks such as question answering, where it is crucial to incorporate context from both directions.
+ 1 more
-
korrents.com
Current techniques restrict pre-trained representation power because standard language models are unidirectional.Their words
We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.