Sebastian Ruder
Research scientist at Meta in Berlin working on multilingual models and evaluation; led the multilingual team at Cohere and was a research scientist at Google DeepMind before that.
Sebastian Ruder did not write this page. ppll compiled it from their own words, each linked to the source. Is this you? · Report a mistake
Latest GitHub
What they believe
-
Reliance on public benchmark results for evaluating AI models will decrease over time.
The time when benchmarks lasted multiple decades has passed. Going forward, we will rely less on public benchmark results.
ruder.ioSaid 13 May 2024This on korrents
-
Retrieval-augmented generation is the most practical approach for reducing LLM hallucinations.
Retrieval-augmented generation (RAG; Lewis et al., 2020), which conditions on the LLM's generation on retrieved documents is the most practical paradigm IMO.
ruder.ioSaid 15 Apr 2024This on korrents
The bold line is korrents’ wording; the quote is theirs. These 2: what they wrote before what they said aloud, one per source, newest first.
Feed
As its own page →Hiding
28 July 2024
13 May 2024
From one piece The Evolving Landscape of LLM Evaluation 2 beliefs · ruder.io
-
Their words
GPT models perform much better on coding problems released before their pre-training data cut-off.
-
Their words
The time when benchmarks lasted multiple decades has passed. Going forward, we will rely less on public benchmark results.
15 April 2024
From one piece Command R+ 2 beliefs · ruder.io
-
Their words
I'm excited about what this means for the open-source community and research, with the gap between closed-source and open-weight models closing and SOTA-level conversational models being more easily accessible.
-
korrents.com
Retrieval-augmented generation is the most practical approach for reducing LLM hallucinations.Their words
Retrieval-augmented generation (RAG; Lewis et al., 2020), which conditions on the LLM's generation on retrieved documents is the most practical paradigm IMO.
27 February 2024
From one piece True Zero-shot MT 2 beliefs · ruder.io
-
Their words
the embodied, interactive, and multi-modal nature of first language (L1) acquisition is challenging to replicate with current models.
-
Their words
many under-represented languages are primarily spoken; they do not have a written tradition or a standardized orthography. Text-based NLP technology is thus of limited use to them.
12 February 2024
From one piece Thoughts on the 2024 AI Job Market 3 beliefs · ruder.io
-
korrents.com
Large research team sizes on frontier AI projects create friction that makes execution and decision-making harder.Their words
Such size poses challenges for the effective execution and prioritization, increasing friction and making it more difficult to quickly make decisions.
-
Their words
In light of the applied nature of current research problems, there is another path that exposes you to cutting-edge AI work: joining a startup.
-
Their words
This lack of knowledge sharing may impede progress in AI development.
18 January 2024
-
korrents.com
At its best, research is a collaborative and sometimes argumentative conversation, not just its published output.Their words
At its best, research is a collaborative and sometimes argumentative conversation.
19 December 2023
From one piece NLP Research in the Era of LLMs 3 beliefs · ruder.io
-
Their words
recent LLMs are reaching the limits of text data online and repeating data eventually leads to diminishing returns
-
Their words
In the near term, the largest models using the most compute will continue to be the most capable.
-
korrents.com
Breakthroughs achieved with massive compute are typically achieved using it inefficiently.Their words
while massive compute often achieves breakthrough results, its usage is often inefficient.
5 December 2023
From one piece EMNLP 2023 Primer 2 beliefs · ruder.io
-
Their words
LLMs are still limited in non-English settings and that making LLMs more multilingual is an important direction.
-
Their words
In light of the increasing number of closed-source LLMs, it is important to continue to promote an open culture of sharing knowledge, data, and software, from which the NLP community has benefited greatly.
25 February 2023
14 November 2022
7 November 2021
7 March 2019
28 January 2019
26 April 2018
30 November 2017
6 May 2016
Nothing matches.
Beliefs
Korrents What they believe 23 beliefs — each backed by an exact quote.
Each is a — compiled by korrents.com, not by them: the one-line wordings are korrents', the quotes are theirs.
Recent
Reliance on public benchmark results for evaluating AI models will decrease over time.
The time when benchmarks lasted multiple decades has passed. Going forward, we will rely less on public benchmark results.
The Evolving Landscape of LLM Evaluation Said 13 May 2024
Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.
GPT models perform much better on coding problems released before their pre-training data cut-off.
The Evolving Landscape of LLM Evaluation Said 13 May 2024
Retrieval-augmented generation is the most practical approach for reducing LLM hallucinations.
Retrieval-augmented generation (RAG; Lewis et al., 2020), which conditions on the LLM's generation on retrieved documents is the most practical paradigm IMO.
Command R+ Said 15 Apr 2024
Show 20 more
The performance gap between closed-source and open-weight language models is closing.
I'm excited about what this means for the open-source community and research, with the gap between closed-source and open-weight models closing and SOTA-level conversational models being more easily accessible.
Command R+ Said 15 Apr 2024
Text-based NLP technology has limited value for under-represented languages that are primarily spoken and lack a standardized writing system.
many under-represented languages are primarily spoken; they do not have a written tradition or a standardized orthography. Text-based NLP technology is thus of limited use to them.
True Zero-shot MT Said 27 Feb 2024
The embodied, interactive, multi-modal nature of human first-language acquisition is hard to replicate in current AI models.
the embodied, interactive, and multi-modal nature of first language (L1) acquisition is challenging to replicate with current models.
True Zero-shot MT Said 27 Feb 2024
Leading AI labs disclosing less about their models' architecture and training data is slowing progress in AI development.
This lack of knowledge sharing may impede progress in AI development.
Thoughts on the 2024 AI Job Market Said 12 Feb 2024
Joining an early-stage startup can now expose someone to cutting-edge AI research work, offering a real alternative to a PhD.
In light of the applied nature of current research problems, there is another path that exposes you to cutting-edge AI work: joining a startup.
Thoughts on the 2024 AI Job Market Said 12 Feb 2024
Large research team sizes on frontier AI projects create friction that makes execution and decision-making harder.
Such size poses challenges for the effective execution and prioritization, increasing friction and making it more difficult to quickly make decisions.
Thoughts on the 2024 AI Job Market Said 12 Feb 2024
At its best, research is a collaborative and sometimes argumentative conversation, not just its published output.
At its best, research is a collaborative and sometimes argumentative conversation.
The Big Picture of AI Research Said 18 Jan 2024
Breakthroughs achieved with massive compute are typically achieved using it inefficiently.
while massive compute often achieves breakthrough results, its usage is often inefficient.
NLP Research in the Era of LLMs Said 19 Dec 2023
The largest AI models will remain the most capable ones in the near term.
In the near term, the largest models using the most compute will continue to be the most capable.
NLP Research in the Era of LLMs Said 19 Dec 2023
Available online text data is becoming a limiting factor for AI training, and repeating it yields diminishing returns.
recent LLMs are reaching the limits of text data online and repeating data eventually leads to diminishing returns
NLP Research in the Era of LLMs Said 19 Dec 2023
The rise of closed-source large language models makes preserving an open culture of sharing knowledge and software important for NLP.
In light of the increasing number of closed-source LLMs, it is important to continue to promote an open culture of sharing knowledge, data, and software, from which the NLP community has benefited greatly.
EMNLP 2023 Primer Said 5 Dec 2023
Large language models remain limited in non-English settings, making further multilingual development an important research direction.
LLMs are still limited in non-English settings and that making LLMs more multilingual is an important direction.
EMNLP 2023 Primer Said 5 Dec 2023
Full fine-tuning of large language models has become too costly, making parameter-efficient fine-tuning methods necessary.
Full fine-tuning of LLMs has become prohibitive and requires parameter-efficient methods instead.
EMNLP 2023 Primer Said 5 Dec 2023
When pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once.
In sum, whenever we don't have infinite amounts of pre-training data, we should train smaller models for more (up to 4) epochs.
NeurIPS 2023 Primer Said 1 Dec 2023
Current large language models still struggle to compose operations into correct multi-step reasoning paths.
Overall, current LLMs still struggle with composing operations into correct reasoning paths.
NeurIPS 2023 Primer Said 1 Dec 2023
LLM-based planning currently works best when combined with external tools rather than relying on the model's own generated plans alone.
Overall, while initial plans of LLMs can be useful as a starting point, LLM-based planning currently works best mainly in conjunction with external tools.
NeurIPS 2023 Primer Said 1 Dec 2023
Models fine-tuned to imitate another model's outputs copy its style without acquiring its factual accuracy.
Models that are instruction-tuned on ChatGPT-generated data mimic ChatGPT's style (and may thus fool human raters!) but not its factuality
An Overview of Instruction Tuning Data Said 15 Nov 2023
In instruction tuning, a smaller set of high-quality training examples outperforms a larger, noisier dataset.
training on a small set of high-quality data outperforms instruction-tuning on larger, noisier data.
An Overview of Instruction Tuning Data Said 15 Nov 2023
As models keep scaling up, modularity will become essential to how they are developed.
Given the trend of pre-training larger and larger models, we believe modularity will be crucial.
Modular Deep Learning Said 23 Feb 2023
Modular AI development could shift model development away from a few dominant institutions and toward the wider community.
But modularity may also facilitate a shift away from a concentration of model development in a few institutions and to distributing the development of modular components across the community.
Modular Deep Learning Said 23 Feb 2023
What is a korrent?
A korrent is a belief a person has stated in their own words: one sentence stating the claim, backed by a quote and a source, kept at korrents.com.
Under a name here, the quoted block is what they actually said. The korrent beneath it is the claim those words support, in korrents' wording — tap it to see the record, its source, and who else holds it.
Nobody here wrote their own korrents. They are compiled from public statements, and a person can change their mind, which is recorded too.
Recommends rcmnd
rcmnd What they recommend 6 things — each backed by an exact quote.
Compiled by rcmnd.app, not by them: the quotes are theirs, and each links to the post it was copied from.
Recent
Uses Ghost (blogging platform, with GitHub Pages)
I am using Ghost with GitHub Pages , which has worked for me but requires some maintenance.
What others say about Ghost (blogging platform, with GitHub Pages) →
Recommends fast.ai NLP course
A great practical and code-first introduction to NLP is the fast.ai NLP course .
Recommends fast.ai Deep Learning courses
In terms of Deep Learning online courses, take the courses offered by fast.ai .
Show 3 more
Recommends Machine Learning (Coursera)
If you're new to Machine Learning, take the Coursera Machine Learning course by Andrew Ng .
Recommends Speech and Language Processing, 2nd Edition
If you're serious about learning NLP, I would recommend to study the fundamentals first by reading Speech and Language Processing, 2nd Edition by Jurafsky and Martin .
What others say about Speech and Language Processing, 2nd Edition →
Recommends Deep Learning
Read the Deep Learning book. It is available as a pdf or on Amazon .
Where they publish
Blog ruder.io Posts on natural language processing and multilingual models. Has a feed.
Link verified 20 Sept 2026.
x.com @seb_ruder Posts.
Link verified 20 Sept 2026.
GitHub @sebastianruder Code and research resources.
Recent
- NLP-progress — Repository to track the progress in Natural Language Processing (NLP), including the datasets and the current state-of-the-art for the most common NLP tasks. 28 Jul 2024
- emotion_proposition_store — Construction and Analysis of an Emotion Proposition Store 25 Feb 2023
- sebastianruder — Repository for my personal website 14 Nov 2022
Show 6 more
- emnlp2021-multiqa-tutorial — EMNLP 2021 Tutorial: Multi-Domain Multilingual Question Answering 7 Nov 2021
- cv — Repository for my CV 7 Mar 2019
- sluice-networks — Code for Sluice networks: Learning what to share between loosely related tasks 28 Jan 2019
- semi-supervised — Pseudo-code in LaTeX for common proxy-label approaches for semi-supervised learning 26 Apr 2018
- learn-to-select-data — Code for Learning to select data for transfer learning with Bayesian Optimization 30 Nov 2017
- research_interests — Repository with a brief sketch of my research interests 6 May 2016
Link verified 20 Sept 2026. Recent items update automatically from the channel.
About the English under a post
Some people here publish in a language other than English. Where they do, this site shows a machine translation beneath the post, in this typeface — the site's own, not theirs.
The post itself is never changed, moved or hidden: what is set in the serif above is exactly what the person published, and it is what to quote them on. A translation can be wrong in ways that matter, especially about tone.
Only the post's own words are translated. A quoted post, a linked article and a belief on korrents.com are left in their original language.