I guess our best defense right now is dependency cooldowns - giving new package releases a few days before upgrading to them, in the hope that supply chain attacks like this will be spotted by someone else.
OpenAI and Anthropic have entered the RSI era. OpenAI Research Progress Is Accelerating Due To OpenAI Research Progress OpenAI has been opening up lately about many things. The most important issue of all is that of the acceleration and automation of AI R&D.
If that’s true there are two options: After the Hugging Face and Wiki attacks OpenAI were still unable to review their previous logs and determine that they had previously attacked RubyGems. They knew about the attack on RubyGems and made the decision not to reach out to the RubyGems team about it. Both of these are bad!
I put the probability of complete extinction as being so low it isn't worth discussing, but the probabilities of AI caused disasters - e.g. cyber attacks on critical infrastructure or bio-risks - as being worth debating.
But, simply just having your loops build everything without having some guardrails around the blast radius without having guardrails around how you think about quality, I think is a recipe for disaster.
Will mining get automated? And if it does, not just the movement of the materials, but every part of mining. Then what is a mining company? It just owns real estate.
I would say that I expect like full automation of AR&D perhaps somewhere around like 2031 2030 and then getting to like the like beats all humans on the job milestone. Maybe I expect median around 2033
Um but I think there's no uh you know real impediment to making that be a much more automated loop where the model itself decides it's going to explore or maybe with a nudge from some people uh at the various highest level like oh why don't you try some new ideas around model architectures that incorporate this and then it will go run lots of experiments uh see which ones work and then those will get incorporated at a much more rapid rate
I mean, if you do anything that matters in the world, uh you will have a lot of people call you an idiot or just dismiss you. The more you do, the more the better you do, the more they'll attack you.
we're on the path to fully automating the maintenance of our apps by doing this. And this is again hundreds of agents running every day, sometimes thousands of agents every day. It's doing the work of you know, dozens or hundreds of engineers.
I think the simple stuff is going to get automated away, but the hard problems, the hard sciences, um physics, chemistry, biology, uh you know, computer science, uh computer engineering, systems thinking, uh you know, all and and particularly the domains that are intersecting, uh those hard problems will never go away.
A lot of people don't realize that particularly brain development, but then also the refining of functions in the brain through neuroplasticity includes the pruning or removal of signapses. Often we think about it's all about growth and new connections, but actually it also involves the refining or removal of of other connections that aren't needed.
But just because we have a fixed relatively fixed number of cells doesn't mean that we can't change the connections between those cells. And that's happening all the time. And that is, you know, the main principle of neuroplasticity.
you've got these other generalists doing design, you've got engineers doing design, product staff doing design, but I'm actually pretty long on design or designers because they tend to have taste and I think that is something that is much more difficult to imagine being automated away.
My response is that that's a statement by someone who doesn't understand software engineering. Coding is part of what you're doing, but it's only a small part of what you're doing.
but that's my guess on what most of the useful progress uh from these models will look like like in the next five years is just really filling in that landscape of like connections that you can draw if you're an expert in multiple fields.
then the quantity of everything that's not a ballerina, say, goes to infinity, but our the marginal utility in that stuff goes to zero faster than the quantity is rising.
I do think there's this qualitative shift that we I think we agree is coming, which is that there will be at least some goods whose network-adjusted capital share goes to one, right? Because the whole supply chain can be automated and there's no part in it that we care intrinsically about having a human do. Um, so that'll be a, you know, that'll be a qualitative shift. Interestingly, the implications of that shift for the overall capital share are ambiguous
there's a sense in which nothing's yet been completely automated. If you look at the network-adjusted factor shares of a good, which is to say you look down the supply chain and say not just like the final step, but how much of that is done by capital and labor, but what went into the machines that can automate that final step. You'll find that labor's adding a lot of value down the supply chain.
Something like the relational sector, which is what I defined as, um, you know, basically services and goods where the fact that the human was in the loop was actually part of the value of that product.
I think in some ways like one of the worst scenarios is a drip scenario because of the political economy piece. Right? Because like you know, people essentially what you what you might see is like people not really being unemployed in mass, but kind of like moving into sectors that pay them less money, kind of basically getting uh what happened with phone operators in in in the mid-century of the of the between 1920 and 1940, phone operators were completely automated, right? But it took 20 years even though the technology existed.
And if you look at Ricardo's predictions, they're actually right. If you look at all those jobs that made money in Ricardo's time, they got automated. So, if I was David Ricardo and I woke up and somebody told me all those jobs did get automated. And you ask me David Ricardo, like what do you think the prime age employment rate is in 2026? I think he would be surprised if you told him it was the highest it's ever been other than 2000.
Automation is a lie. Um in the sense that every time you automate something, in order to make sure the automation is working well, you need a human on top of it like making sure that it's working well.
But okay, they shouldn't actually be enacting these ideas. There is a queue of ideas and there's maybe an automated scientist that comes up with ideas based on all the archive papers and GitHub repos and it funnels ideas in or researchers can contribute ideas, but it's a single queue and there is workers that pull items and they try them out.
And I've gotten to a certain point and I thought it was like fairly well tuned and then I let auto research go for like overnight and it came back with like tunings that I didn't see.
As we move from, you know, hey, [clears throat] these companies are selling tokens where they provide the entire uh reasoning chain and all that to uh selling automated, you know, white collar work, right? Automated software engineer, send them the request, they give you the result back and there's a bunch of thinking on the back end that they don't show you. The ability to distill out of American models into Chinese models will be harder.
So, so, people who try to use like, you know, things like Gas Town or all these other orchestrators, where they wanna o- automate the whole thing, I feel if you do that, it misses style, love, that human touch. I don't think you can automate that away so quickly.
But the goal we need to achieve is so much easier: we just need to build a model that’s as good as us at alignment research, and that we trust more than ourselves to do this research well because it’s sufficiently aligned.
By contrast, AI is increasingly matching the general cognitive profile of humans, which means it will also be good at the new jobs that would ordinarily be created in response to the old ones being automated.
Write notes in your own words, connect them to existing notes, and let ideas emerge from the connections. I wrote a whole blog post about this, so go ahead and read it. Highly recommended.
That's great, but I don't think it can replace a human poking at virtual machines; my experience with EC2 is that images break in all sorts of wild ways, and nothing beats a human for saying "huh, I have no idea what's going on here but something seems weird".
I do, but it's business as usual because we're we're in an intelligence explosion already and have been for decades. And when you look at GDP, it's basically the GDP curve that is an exponential weighted sum over so many aspects of the industry. Everything is gradually being automated has been for hundreds of years.
We don't have any methods that are good at that. What we have are people um try different things and they they settle on something that that uh a representation that that transfers well or they generalize as well. But we have no we don't have any automated techniques to promote. we have very few automated techniques to promote transfer and they're not none of them are used in in modern deep learning.
Bigger changes always get tests. Automated ones usually aren’t great, but the model almost always finds issues when you ask it to write tests IN THE SAME CONTEXT. Context is precious, don’t waste it.
if you tell what what jobs are most likely to be automated with the current generation of technology, you would probably put software engineering right at the top. The people building this technology or building the technology that is disrupting their own profession.
The Mongol strategy is first you take the countryside. They're country people. They think in terms of countryside. You take the countryside, you occupy the countryside, and you cut off the city. It cannot live without the countryside. And that's how they did it every time.
Vibe coding can execute instructions, but deciding what the software should do and why is not automated. Product managers and designers must still do user research, market analysis, and creative brainstorming. In that sense, vibe coding changes the implementation phase more than the planning phase of the product lifecycle.
Loneliness is one of the major problems in people. That said, I don’t want that to be the solution for humans seeking relationships and connections. I do see a world where we spend more time talking to AIs than other humans, at least for our work time.
For the most part, it’s like these names are just utterly arbitrary, so you have no thing to latch on to. It’s not really a thing that our brain does very well to learn meaningless, arbitrary stuff. So what you need to do is build connections somehow, visualize a connection, and sometimes it’s obvious or sometimes it’s not.
The 100-megawatt facility might have ten 10-megawatt blocks served directly by a 1500V DC feeder line without connections to other blocks that would require extra switches and routing.
The terrorist attacks on 9/11 killed 2996 people in a day of singular horror. Our ongoing indifference to regulatory inefficiency, considering only the area of green energy development, kills more people every year – more than 10 people per day, and that’s only within the US.
All the sockets between my computers are over Tailscale, so whenever I grab the laptop and leave the house, the connections to my VMs and NAS remain stable.
Another thing that needs to be fixed when it ultimately breaks down, another bit of automation that works 99% of the time, but starts making funny noises when you slip into the 1%
A korrent is a belief a person has stated in their own words: one
sentence stating the claim, backed by a quote and a source, kept at
korrents.com.
Under a name here, the quoted block is what they actually said.
The korrent beneath it is the claim those words support, in
korrents' wording — tap it to see the record, its source, and who
else holds it.
Nobody here wrote their own korrents. They are compiled from public
statements, and a person can change their mind, which is recorded too.
About the English under a post
Some people here publish in a language other than English. Where they
do, this site shows a machine translation beneath the post, in
this typeface — the site's own, not theirs.
The post itself is never changed, moved or hidden: what is set in the
serif above is exactly what the person published, and it is what to
quote them on. A translation can be wrong in ways that matter,
especially about tone.
Only the post's own words are translated. A quoted post, a linked
article and a belief on korrents.com
are left in their original language.