Gwern
Pseudonymous essayist. Writes at gwern.net on AI scaling, statistics, genetics and long-running internet research, with heavy citation and a habit of revising old essays in place rather than leaving them wrong.
Gwern did not write this page. ppll compiled it from their own words, each linked to the source. Is this you? · Report a mistake
Latest GitHub
What they believe
-
Criticism that a simpler, scaled-up architecture underperforms a heavily hand-optimized baseline actually demonstrates why that architecture-scaling research matters.
The more people tweet about how MLP-Mixer is overhyped because is −X% worse than the ultra-hand-optimized baseline or requires Y× more FLOPS, the more they demonstrate precisely why this sort of research is so important!
gwern.substack.comSaid 11 Jun 2021This on korrents
-
Research on transformer-style attention mechanisms has not yet reached diminishing returns compared to alternative approaches.
if the marginal return on this research direction had fallen below that of competitors, we would see those neglected directions invade Transformer topics-while we continue to see the reverse, and many applications as yet untouched by all the new approaches, suggesting that we still don't pay enough attention
gwern.substack.comSaid 3 Jun 2021This on korrents
The bold line is korrents’ wording; the quote is theirs. These 2: what they wrote before what they said aloud, one per source, newest first.
Feed
As its own page →Hiding
11 October
29 June 2024
-
Recommendsrcmnd.app
Apollo 11Their words
Strongly recommended for anyone with an interest in space, greatness, or reality.
-
Likedrcmnd.app
KediTheir words
Thoroughly enjoyable, stuffed with beautiful shots of Istanbul and cats, with particularly clever ankle-level tracking shots following the semi-feral cats around.
-
Recommendsrcmnd.app
AmyTheir words
Overall, required viewing for any Winehouse fan and of general psychiatric interest; possibly too painful to watch for others.
+ 4 more
-
Likedrcmnd.app
All About EveTheir words
Excellent character-study/drama about the price of fame, the sincerest forms of flattery, and a little meta-fictionally, the psychology of the theatre and Broadway being usurped by Hollywood
-
Lovedrcmnd.app
Blade Runner 2049Their words
one of the best movies I’ve seen all year, well worth paying to see on the big screen
-
Recommendsrcmnd.app
They LiveTheir words
For a compelling depiction of how this might feel from the inside, I strongly recommend John Carpenter’s movie They Live
-
Likedrcmnd.app
Ready Player OneTheir words
It is excellent to watch on a big screen in 3D despite how silly it ultimately is, so I’m not surprised it’s been successful.
6 June 2024
-
Lovedrcmnd.app
RedlineTheir words
One of the most amazing hand-animated films of all time, Redline is a blast from start to finish in showing dystopian SF racing
-
Lovedrcmnd.app
Made in AbyssTheir words
I suspect Made in Abyss will be the best anime I watched in 2019, and perhaps 2020 too.
-
Lovedrcmnd.app
Mushishi Zoku ShouTheir words
Rather than declining, the second season is better than the first.
+ 2 more
-
Lovedrcmnd.app
Ringing BellTheir words
There are not many anime from the 1970s which could be said to be as worth watching now as the day they were made, but Chirin no Suzu manages to be one of them for its unflinching honesty.
-
Likedrcmnd.app
Hellsing UltimateTheir words
All in all, a very satisfactory OVA indeed.
12 September 2023
13 October 2022
-
Likedrcmnd.app
The Black CompanyTheir words
I enjoyed the first book a great deal: it’s in a fairly stock medieval setting, but it handles the dark fantasy well
-
Likedrcmnd.app
Savage Continent: Europe in the Aftermath of World War IITheir words
a fascinating book on the bloody aftermath of WWII as the destruction wound down
-
Likedrcmnd.app
Tombstone: The Untold Story of Mao’s Great FamineTheir words
The statistics and anecdotes are fairly horrifying, and the sheer profusion drills in how widespread the famine was.
11 June 2021
From one piece May 2021 Gwern.net Newsletter 3 beliefs · gwern.substack.com
-
Their words
human intelligence's g is real but is the overall 'pool' of neural resources which derives from overall body integrity because the number of neurons, their density, their myelination, resistance to damage and infection etc, is causally downstream of all body and developmental systems, creating a huge mutational target
-
Their words
The Master Synthesis: intelligence is execution of small simplicity-weighted programs, best discovered by search over smooth loss landscapes like that of highly-overparameterized differentiable networks containing lottery-ticket subnetworks which are ensembled/averaged over, approaching Bayes-optimal reasoning in the limit (as nearest-neighbors-like high dimensional interpolation / memorization gives way to algorithmic generalization / interpolation on a more abstract level)
-
Their words
The more people tweet about how MLP-Mixer is overhyped because is −X% worse than the ultra-hand-optimized baseline or requires Y× more FLOPS, the more they demonstrate precisely why this sort of research is so important!
3 June 2021
From one piece April 2021 newsletter 3 beliefs · gwern.substack.com
-
Their words
nothing in psychology, or medicine, makes sense except in the light of individual differences
-
korrents.com
Demanding that research be done cheaply tends to make it the most expensive kind of research.Their words
there's nothing so expensive as demanding research be cheap.
-
Their words
if the marginal return on this research direction had fallen below that of competitors, we would see those neglected directions invade Transformer topics-while we continue to see the reverse, and many applications as yet untouched by all the new approaches, suggesting that we still don't pay enough attention
6 April 2021
From one piece March 2021 Gwern.net Newsletter 3 beliefs · gwern.substack.com
-
korrents.com
Unsupervised learning trained on unfiltered, internet-scale image data now genuinely works.Their words
another reminder that unsupervised learning is really working!
-
Their words
a self-blinding study, similar to my old self-blinding protocols, confirms that microdosing is just placebo effect, as I said in 2012
-
Their words
Supervised learning from ImageNet is now obsolete for transfer learning, and ImageNet just a contaminated validation set
13 March 2021
From one piece February 2021 Gwern.net Newsletter 3 beliefs · gwern.substack.com
-
Their words
it's not one thing, it's everything-more intelligent people have lower discount rates, save more for longer & are less risk-averse, more accurately predict future growth or inflation
-
Their words
NASA's lunar quarantine was a sham intended to mollify the public as they covered up repeated major failures & lab leaks both before & after-had there been any dangerous lunar organisms, they would have escaped easily
-
Their words
it is increasingly clear that brains really do do backprop in some sense
4 February 2021
From one piece Jan 2021 Gwern.net Newsletter 3 beliefs · gwern.substack.com
-
Their words
The fact that the prefix-tuning, by directly optimizing the prompt embeddings, yields better performance than even single optimized text prompts, suggests so.
-
korrents.com
Mixture-of-experts models can scale stably but begin to show fundamental limits as they are scaled further.Their words
the mixture-of-experts approach, while scaling stably, starts showing its limits
-
Their words
To take the GAN/DRL analogy seriously, perhaps they were they ultimately a dead end, akin to trying to learn everything from rewards, and an adversarial GAN loss ought to be only the cherry on the cake of a large unsupervised/semi-supervised generative model.
5 January 2014
Nothing matches.
Beliefs
Korrents What they believe 15 beliefs — each backed by an exact quote.
Each is a — compiled by korrents.com, not by them: the one-line wordings are korrents', the quotes are theirs.
Recent
Criticism that a simpler, scaled-up architecture underperforms a heavily hand-optimized baseline actually demonstrates why that architecture-scaling research matters.
The more people tweet about how MLP-Mixer is overhyped because is −X% worse than the ultra-hand-optimized baseline or requires Y× more FLOPS, the more they demonstrate precisely why this sort of research is so important!
May 2021 Gwern.net Newsletter Said 11 Jun 2021
Intelligence is the execution of small, simplicity-weighted programs found by search over smooth loss landscapes in overparameterized networks, approaching Bayes-optimal reasoning as scale increases.
The Master Synthesis: intelligence is execution of small simplicity-weighted programs, best discovered by search over smooth loss landscapes like that of highly-overparameterized differentiable networks containing lottery-ticket subnetworks which are ensembled/averaged over, approaching Bayes-optimal reasoning in the limit (as nearest-neighbors-like high dimensional interpolation / memorization gives way to algorithmic generalization / interpolation on a more abstract level)
May 2021 Gwern.net Newsletter Said 11 Jun 2021
General intelligence (g) reflects an overall pool of neural resources rooted in whole-body developmental integrity, making it a large mutational target rather than a distinct cognitive module.
human intelligence's g is real but is the overall 'pool' of neural resources which derives from overall body integrity because the number of neurons, their density, their myelination, resistance to damage and infection etc, is causally downstream of all body and developmental systems, creating a huge mutational target
May 2021 Gwern.net Newsletter Said 11 Jun 2021
Show 12 more
Research on transformer-style attention mechanisms has not yet reached diminishing returns compared to alternative approaches.
if the marginal return on this research direction had fallen below that of competitors, we would see those neglected directions invade Transformer topics-while we continue to see the reverse, and many applications as yet untouched by all the new approaches, suggesting that we still don't pay enough attention
April 2021 newsletter Said 3 Jun 2021
Demanding that research be done cheaply tends to make it the most expensive kind of research.
there's nothing so expensive as demanding research be cheap.
April 2021 newsletter Said 3 Jun 2021
Individual differences are essential to making sense of psychology and medicine.
nothing in psychology, or medicine, makes sense except in the light of individual differences
April 2021 newsletter Said 3 Jun 2021
Supervised pretraining on ImageNet is now obsolete for transfer learning, and ImageNet itself is just a contaminated validation set.
Supervised learning from ImageNet is now obsolete for transfer learning, and ImageNet just a contaminated validation set
March 2021 Gwern.net Newsletter Said 6 Apr 2021
Psychedelic microdosing produces no effect beyond placebo.
a self-blinding study, similar to my old self-blinding protocols, confirms that microdosing is just placebo effect, as I said in 2012
March 2021 Gwern.net Newsletter Said 6 Apr 2021
Unsupervised learning trained on unfiltered, internet-scale image data now genuinely works.
another reminder that unsupervised learning is really working!
March 2021 Gwern.net Newsletter Said 6 Apr 2021
Growing evidence suggests the brain performs a form of backpropagation-like learning.
it is increasingly clear that brains really do do backprop in some sense
February 2021 Gwern.net Newsletter Said 13 Mar 2021
NASA's lunar quarantine after the Apollo missions was theater meant to reassure the public rather than genuine biocontainment.
NASA's lunar quarantine was a sham intended to mollify the public as they covered up repeated major failures & lab leaks both before & after-had there been any dangerous lunar organisms, they would have escaped easily
February 2021 Gwern.net Newsletter Said 13 Mar 2021
More intelligent people tend to be better investors because of lower time-discounting, greater risk tolerance, and more accurate economic forecasts, not any single factor.
it's not one thing, it's everything-more intelligent people have lower discount rates, save more for longer & are less risk-averse, more accurately predict future growth or inflation
February 2021 Gwern.net Newsletter Said 13 Mar 2021
An adversarial GAN loss should only supplement a larger unsupervised or semi-supervised generative model rather than stand alone as a primary approach.
To take the GAN/DRL analogy seriously, perhaps they were they ultimately a dead end, akin to trying to learn everything from rewards, and an adversarial GAN loss ought to be only the cherry on the cake of a large unsupervised/semi-supervised generative model.
Jan 2021 Gwern.net Newsletter Said 4 Feb 2021
Mixture-of-experts models can scale stably but begin to show fundamental limits as they are scaled further.
the mixture-of-experts approach, while scaling stably, starts showing its limits
Jan 2021 Gwern.net Newsletter Said 4 Feb 2021
Directly optimizing prompt embeddings (prefix-tuning) yields better performance than even a single optimized text prompt.
The fact that the prefix-tuning, by directly optimizing the prompt embeddings, yields better performance than even single optimized text prompts, suggests so.
Jan 2021 Gwern.net Newsletter Said 4 Feb 2021
What is a korrent?
A korrent is a belief a person has stated in their own words: one sentence stating the claim, backed by a quote and a source, kept at korrents.com.
Under a name here, the quoted block is what they actually said. The korrent beneath it is the claim those words support, in korrents' wording — tap it to see the record, its source, and who else holds it.
Nobody here wrote their own korrents. They are compiled from public statements, and a person can change their mind, which is recorded too.
Recommends rcmnd
rcmnd What they recommend 33 things — each backed by an exact quote.
Compiled by rcmnd.app, not by them: the quotes are theirs, and each links to the post it was copied from.
Recent
Liked Ready Player One
It is excellent to watch on a big screen in 3D despite how silly it ultimately is, so I’m not surprised it’s been successful.
gwern.net 29 Jun 2024
Recommends They Live
For a compelling depiction of how this might feel from the inside, I strongly recommend John Carpenter’s movie They Live
gwern.net 29 Jun 2024
Loved Blade Runner 2049
one of the best movies I’ve seen all year, well worth paying to see on the big screen
gwern.net 29 Jun 2024
Show 30 more
Liked All About Eve
Excellent character-study/drama about the price of fame, the sincerest forms of flattery, and a little meta-fictionally, the psychology of the theatre and Broadway being usurped by Hollywood
gwern.net 29 Jun 2024
Recommends Amy
Overall, required viewing for any Winehouse fan and of general psychiatric interest; possibly too painful to watch for others.
gwern.net 29 Jun 2024
Liked Kedi
Thoroughly enjoyable, stuffed with beautiful shots of Istanbul and cats, with particularly clever ankle-level tracking shots following the semi-feral cats around.
gwern.net 29 Jun 2024
Recommends Apollo 11
Strongly recommended for anyone with an interest in space, greatness, or reality.
gwern.net 29 Jun 2024
Liked Hellsing Ultimate
All in all, a very satisfactory OVA indeed.
gwern.net 6 Jun 2024
Loved Ringing Bell
There are not many anime from the 1970s which could be said to be as worth watching now as the day they were made, but Chirin no Suzu manages to be one of them for its unflinching honesty.
gwern.net 6 Jun 2024
Loved Mushishi Zoku Shou
Rather than declining, the second season is better than the first.
gwern.net 6 Jun 2024
Loved Made in Abyss
I suspect Made in Abyss will be the best anime I watched in 2019, and perhaps 2020 too.
gwern.net 6 Jun 2024
Loved Redline
One of the most amazing hand-animated films of all time, Redline is a blast from start to finish in showing dystopian SF racing
gwern.net 6 Jun 2024
Liked Tombstone: The Untold Story of Mao’s Great Famine
The statistics and anecdotes are fairly horrifying, and the sheer profusion drills in how widespread the famine was.
gwern.net 13 Oct 2022
What others say about Tombstone: The Untold Story of Mao’s Great Famine →
Liked Savage Continent: Europe in the Aftermath of World War II
a fascinating book on the bloody aftermath of WWII as the destruction wound down
gwern.net 13 Oct 2022
What others say about Savage Continent: Europe in the Aftermath of World War II →
Liked The Black Company
I enjoyed the first book a great deal: it’s in a fairly stock medieval setting, but it handles the dark fantasy well
gwern.net 13 Oct 2022
Liked In the Plex: How Google Thinks, Works, and Shapes Our Lives
I learned a great deal from this book about Google, which put some of my own experiences with Google products in context.
gwern.net 13 Oct 2022
What others say about In the Plex: How Google Thinks, Works, and Shapes Our Lives →
Liked Titan: The Life of John D. Rockefeller, Sr.
Fascinating account of a Gilded Age titan much worse known than Carnegie.
gwern.net 13 Oct 2022
What others say about Titan: The Life of John D. Rockefeller, Sr. →
Liked Clear and Simple as the Truth: Writing Classic Prose
A style book which actually delivers real style advice!
gwern.net 13 Oct 2022
What others say about Clear and Simple as the Truth: Writing Classic Prose →
Liked Packing for Mars: The Curious Science of Life in the Void
Roach comes up with all the best quotes and stories, seems to have talked to everyone and done everything.
gwern.net 13 Oct 2022
What others say about Packing for Mars: The Curious Science of Life in the Void →
Loved Bias in Mental Testing
One of the classics in the field, Jensen sets out to explain almost everything, it seems, in psychometrics
gwern.net 13 Oct 2022
Loved The Notenki Memoirs: Studio Gainax and the Men Who Created Evangelion
It is an invaluable resource for any researcher
gwern.net 13 Oct 2022
What others say about The Notenki Memoirs: Studio Gainax and the Men Who Created Evangelion →
Loved The Thousand Autumns of Jacob de Zoet
It was surprisingly unliterary and unpostmodern for Mitchell, but in exchange, he nailed the historical details and gave us an adventure which subverted many of the usual tropes
gwern.net 13 Oct 2022
What others say about The Thousand Autumns of Jacob de Zoet →
Loved The Discovery of France: A Historical Geography from the Revolution to the First World War
This is incredibly interesting because from our perspective, we have forgotten (if we ever knew) what went into the process of taking the thousands of villages and regions differing in all sorts of ways
gwern.net 13 Oct 2022
Loved The Origins of Political Order: From Prehuman Times to the French Revolution
It is, overall, an excellent book and one of the better ones on grand history I’ve read
gwern.net 13 Oct 2022
What others say about The Origins of Political Order: From Prehuman Times to the French Revolution →
Loved Genius: The Life and Science of Richard Feynman
A solid biography, though I don’t have anything in particular to say about it.
gwern.net 13 Oct 2022
What others say about Genius: The Life and Science of Richard Feynman →
Loved The Histories
Overall, far more interesting than I had expected. Surprisingly funny or interesting anecdotes.
gwern.net 13 Oct 2022
Loved The Wages of Destruction: The Making and Breaking of the Nazi Economy
A fascinating account of the economic transformation of Germany under the Nazis
gwern.net 13 Oct 2022
What others say about The Wages of Destruction: The Making and Breaking of the Nazi Economy →
Loved Lords of Finance: The Bankers Who Broke the World
I enjoyed this tremendously for revealing a new world to me where I thought I already knew the lay of the land.
gwern.net 13 Oct 2022
What others say about Lords of Finance: The Bankers Who Broke the World →
Loved The Remains of the Day
The prose is so smooth that like Gene Wolfe’s, it becomes invisible, and you pass through it to the slow silent sorrow of the protagonist.
gwern.net 13 Oct 2022
Loved Star Maker
It’s really impressive how Stapledon just casually scatters around handfuls of jewels that lesser authors might belabor singly throughout an entire book.
gwern.net 13 Oct 2022
Loved Collapse of Complex Societies
much more convincing than Spengler or Toynbee
gwern.net 13 Oct 2022
Loved The Better Angels of Our Nature: Why Violence Has Declined
This was really really good, as in, maybe the best book I’ve read that year.
gwern.net 13 Oct 2022
What others say about The Better Angels of Our Nature: Why Violence Has Declined →
Loved Stories of Your Life and Others
He is the closest thing to a modern Jorge Luis Borges in melding high concepts with literature to create something better than either
gwern.net 13 Oct 2022
Where they publish
Site gwern.net The essays, heavily cited and revised in place rather than left wrong.
A large, cross-linked body of essays on AI scaling, statistics, genetics and internet research. Old pieces are revised in place rather than superseded, so a page here is a current position rather than a dated one.
Link verified 1 Sept 2026.
GitHub gwern The site's own source, and the research tooling behind it.
The source of gwern.net itself along with the scrapers and analysis code behind the essays — the working method, rather than the conclusions.
Recent
- gwern.net — Site infrastructure for gwern.net. Custom Hakyll website with unique link archiving, popup UX, transclusions/collapses, dark+reader mode, bidirectional backlinks, and typography (sidenote… 11 Oct 2026
- archiver-bot — Background daemon which archives a list of URLs to the Internet Archive, archive.is, and other services 12 Sept 2023
- shellac-ghc — Sketch of a Haskell shell using the Shellac library (abandoned) 5 Jan 2014
Link verified 1 Sept 2026. Recent items update automatically from the channel.
About the English under a post
Some people here publish in a language other than English. Where they do, this site shows a machine translation beneath the post, in this typeface — the site's own, not theirs.
The post itself is never changed, moved or hidden: what is set in the serif above is exactly what the person published, and it is what to quote them on. A translation can be wrong in ways that matter, especially about tone.
Only the post's own words are translated. A quoted post, a linked article and a belief on korrents.com are left in their original language.