Lots of risk with AI. I worry a lot about cybersecurity for example. However, “extinction risk” — literally 100% of humans die — is not remotely one of them.
If there really is a high chance of AI leading to the extinction of humanity within years/decades, then the only rational stance towards safety monitoring and research pacing should be stringent, top-down government involvement and universally ratified international treaties.
The concerns over AI safety and cybersecurity are legitimate, but we’re risking talking America, the global AI leader, into self-inflicted obsolescence and the obscurity of bureaucracy.
I don’t think AI is going to usher in an extinction event. In fact, even if nobody were to slow down, I really don’t think humanity would have much to worry about.
most surveys find that the median AI researcher forecasts about a 5% chance of extinction or similar "doom" from AI, while the mean probability is much higher -- around 15-20% -- due to a smallish group who give very high probabilities of doom.
I put the probability of complete extinction as being so low it isn't worth discussing, but the probabilities of AI caused disasters - e.g. cyber attacks on critical infrastructure or bio-risks - as being worth debating.
A clear risk discussed throughout this year is to cybersecurity: the models are becoming superhuman in their ability to break in and out of computer systems.
Astra is now the best cybersecurity model in the world on @vercel DeepsecBench. It pulls off in 49 minutes what took Sol ~4 hours, with a better score, at nearly the same cost.
I do think I want to push back on the cyber on the brain hypothesis that you raised a couple of times. We didn't find like particular evidence for the cyber nature of the task making all the hacking and crimes that they did more likely versus the impossible nature of the task.
even in this incident we saw there was a lot of pressure um as a result of this incident to stop doing cyber security evaluations and I really don't think that stopping doing evaluations and like sort of blinding ourselves to the result of evaluations is the right reaction to this problem.
Extremely severe extinction events happen every 100M years or so and just switching to sustainable energy will not be enough to stop them. Space satellites to control temperature and massive geo-engineering will be needed or it will be game over.
But yeah, there's been a back and forth about whether dinosaurs were already on their way out or were they a bit limited by the very end of the Cretaceous. I think the more recent analyses have shown that's probably not the case, because in other words, they were basically doing fine right up to the extinction event. And so yeah, if the asteroid hadn't hit, there's no reason to think that they were on some kind of terminal decline.
So birds did not appear as a result of the dinosaurs going extinct. Birds lived alongside the dinosaurs for 100 million years. The birds were not new on the scene, and it's not like, "Oh, the dinosaurs died, and from the ashes rose the birds." No, they've been knocking around forever.
You're dominant in contests, particularly with conspecifics. If you're trying to win a territory or win mating rights, bigger things usually beat up smaller things. So there's going to be selection favoring them. But then big things don't usually do well in extinction events, so that tends to reset the clock by killing off the big stuff, and then smaller stuff does better again.
A korrent is a belief a person has stated in their own words: one
sentence stating the claim, backed by a quote and a source, kept at
korrents.com.
Under a name here, the quoted block is what they actually said.
The korrent beneath it is the claim those words support, in
korrents' wording — tap it to see the record, its source, and who
else holds it.
Nobody here wrote their own korrents. They are compiled from public
statements, and a person can change their mind, which is recorded too.
About the English under a post
Some people here publish in a language other than English. Where they
do, this site shows a machine translation beneath the post, in
this typeface — the site's own, not theirs.
The post itself is never changed, moved or hidden: what is set in the
serif above is exactly what the person published, and it is what to
quote them on. A translation can be wrong in ways that matter,
especially about tone.
Only the post's own words are translated. A quoted post, a linked
article and a belief on korrents.com
are left in their original language.