Sebastian Ruder
Everything, newest first — across every channel. Their profile → · Subscribe
Hiding
28 July 2024
13 May 2024
From one piece The Evolving Landscape of LLM Evaluation 2 beliefs, in the piece's order there
-
Their words
GPT models perform much better on coding problems released before their pre-training data cut-off.
-
Their words
The time when benchmarks lasted multiple decades has passed. Going forward, we will rely less on public benchmark results.
15 April 2024
From one piece Command R+ 2 beliefs, in the piece's order there
-
Their words
I'm excited about what this means for the open-source community and research, with the gap between closed-source and open-weight models closing and SOTA-level conversational models being more easily accessible.
-
korrents.com
Retrieval-augmented generation is the most practical approach for reducing LLM hallucinations.Their words
Retrieval-augmented generation (RAG; Lewis et al., 2020), which conditions on the LLM's generation on retrieved documents is the most practical paradigm IMO.
27 February 2024
From one piece True Zero-shot MT 2 beliefs, in the piece's order there
-
Their words
the embodied, interactive, and multi-modal nature of first language (L1) acquisition is challenging to replicate with current models.
-
Their words
many under-represented languages are primarily spoken; they do not have a written tradition or a standardized orthography. Text-based NLP technology is thus of limited use to them.
12 February 2024
From one piece Thoughts on the 2024 AI Job Market 3 beliefs, in the piece's order there
-
korrents.com
Large research team sizes on frontier AI projects create friction that makes execution and decision-making harder.Their words
Such size poses challenges for the effective execution and prioritization, increasing friction and making it more difficult to quickly make decisions.
-
Their words
In light of the applied nature of current research problems, there is another path that exposes you to cutting-edge AI work: joining a startup.
-
Their words
This lack of knowledge sharing may impede progress in AI development.
18 January 2024
-
korrents.com
At its best, research is a collaborative and sometimes argumentative conversation, not just its published output.Their words
At its best, research is a collaborative and sometimes argumentative conversation.
19 December 2023
From one piece NLP Research in the Era of LLMs 3 beliefs, in the piece's order there
-
Their words
recent LLMs are reaching the limits of text data online and repeating data eventually leads to diminishing returns
-
Their words
In the near term, the largest models using the most compute will continue to be the most capable.
-
korrents.com
Breakthroughs achieved with massive compute are typically achieved using it inefficiently.Their words
while massive compute often achieves breakthrough results, its usage is often inefficient.
5 December 2023
From one piece EMNLP 2023 Primer 3 beliefs, in the piece's order there
-
Their words
Full fine-tuning of LLMs has become prohibitive and requires parameter-efficient methods instead.
-
Their words
LLMs are still limited in non-English settings and that making LLMs more multilingual is an important direction.
-
Their words
In light of the increasing number of closed-source LLMs, it is important to continue to promote an open culture of sharing knowledge, data, and software, from which the NLP community has benefited greatly.
1 December 2023
From one piece NeurIPS 2023 Primer 3 beliefs, in the piece's order there
-
Their words
Overall, while initial plans of LLMs can be useful as a starting point, LLM-based planning currently works best mainly in conjunction with external tools.
-
korrents.com
Current large language models still struggle to compose operations into correct multi-step reasoning paths.Their words
Overall, current LLMs still struggle with composing operations into correct reasoning paths.
-
Their words
In sum, whenever we don't have infinite amounts of pre-training data, we should train smaller models for more (up to 4) epochs.
15 November 2023
From one piece An Overview of Instruction Tuning Data 2 beliefs, in the piece's order there
-
korrents.com
In instruction tuning, a smaller set of high-quality training examples outperforms a larger, noisier dataset.Their words
training on a small set of high-quality data outperforms instruction-tuning on larger, noisier data.
-
korrents.com
Models fine-tuned to imitate another model's outputs copy its style without acquiring its factual accuracy.Their words
Models that are instruction-tuned on ChatGPT-generated data mimic ChatGPT's style (and may thus fool human raters!) but not its factuality
25 February 2023
23 February 2023
From one piece Modular Deep Learning 2 beliefs, in the piece's order there
-
Their words
But modularity may also facilitate a shift away from a concentration of model development in a few institutions and to distributing the development of modular components across the community.
-
Their words
Given the trend of pre-training larger and larger models, we believe modularity will be crucial.
14 November 2022
7 November 2021
7 March 2019
28 January 2019
26 April 2018
30 November 2017
6 May 2016
Nothing matches.
What is a korrent?
A korrent is a belief a person has stated in their own words: one sentence stating the claim, backed by a quote and a source, kept at korrents.com.
Under a name here, the quoted block is what they actually said. The korrent beneath it is the claim those words support, in korrents' wording — tap it to see the record, its source, and who else holds it.
Nobody here wrote their own korrents. They are compiled from public statements, and a person can change their mind, which is recorded too.
About the English under a post
Some people here publish in a language other than English. Where they do, this site shows a machine translation beneath the post, in this typeface — the site's own, not theirs.
The post itself is never changed, moved or hidden: what is set in the serif above is exactly what the person published, and it is what to quote them on. A translation can be wrong in ways that matter, especially about tone.
Only the post's own words are translated. A quoted post, a linked article and a belief on korrents.com are left in their original language.