Andrej Karpathy

Their words

But I want there to be a thing that's behind and that uh is kind of like a common working space for intelligences that the entire industry has access to.

Their words

And I think if you're outside of that frontier lab, your your judgment fundamentally will start to drift because you're not part of the you know, what's coming down the line. And so I feel like my judgment will inevitably start to drift as well.

Their words

You're you're not a completely free agent and you can't actually like be part of that conversation in a fully autonomous um free way. Like if you're inside one of the frontier labs. Like there's some things that you can't say. Uh and conversely there are some things that the organization wants you to say.

Their words

I do have like cautiously optimistic view of this in software engineering where I do think um it does seem to me like the demand for software will be extremely large. Um and it's just become a lot cheaper.

Their words

a swarm of agents on the internet could collaborate to improve LLMs and could potentially even like run circles around frontier labs. Like who knows, you know? Um yeah, like maybe that's even possible. Like frontier labs have a huge amount of trusted compute but the earth is much bigger and has huge amount of untrusted compute.

Their words

I do think we should expect more speciation in the intelligences. Um like, you know, the animal kingdom is extremely diverse in the brains that exist and there's lots of different niches of of nature and some animals have overdeveloped visual cortex or other part kind of parts and I think we we should be able to see more speciation and um you don't need like this oracle that knows everything.

Their words

So even though the models have improved tremendously and if you give them an agentic task, they will just go for hours and move mountains for you. And then you ask for like a joke and it has a stupid joke. It's crappy joke from five years ago and it's because it's outside of the it's outside of the RL.

Their words

And so you're kind of like you're either on rails and you're part of the super intelligence circuits or you're not on rails and you're outside of the verifiable domains and suddenly everything kind of just like meanders.

Their words

I simultaneously feel like I'm talking to an extremely brilliant PhD student who's been like a systems programmer for their entire life and a 10-year-old.

Their words

But okay, they shouldn't actually be enacting these ideas. There is a queue of ideas and there's maybe an automated scientist that comes up with ideas based on all the archive papers and GitHub repos and it funnels ideas in or researchers can contribute ideas, but it's a single queue and there is workers that pull items and they try them out.

Their words

And I've gotten to a certain point and I thought it was like fairly well tuned and then I let auto research go for like overnight and it came back with like tunings that I didn't see.

Their words

So the question is how do I refactor all the abstractions so that I'm not I have to arrange it once and hit go. The name of the game is how can you get more agents running for longer periods of time without your involvement doing stuff on your behalf?

Their words

So I think the industry just has to reconfigure in so many ways that's like the customer is not the human anymore. It's like agents who are acting on behalf of humans and this refactoring will be will probably be substantial in a certain sense.

Their words

these apps that are on the app store for using these smart home devices, etc. Uh, these shouldn't even exist kind of in a certain sense. Like shouldn't it just be APIs and shouldn't agents be just using it directly?

Their words

But now it's not about flops, it's about tokens. So what is your token throughput and what token throughput do you command?

Their words

I think to a large extent you feel like it's a skill issue. It's not that the capability is not there. It's that you just haven't found a way to string it together of what's available.

Their words

I kind of went from 80/20 of like, you know, uh to like 20/80 of writing code by myself versus just delegating to agents. And I don't even think it's 20/80 by now. I think it's a lot more than that. I don't think I've typed like a line of code probably since December basically.

Their words

Use apps like BobbyApproved and Ivy to scan the bar codes and get a lot of information about all ingredients and a score (I like and use both).

Their words

Consider getting a home delivery service, e.g. I currently like and use Locale

Their words

Look for very few and simple ingredients and ideally "fragrance free" and "dye free". I currently use Blueland for all of these.

Their words

Recommended reading: "Poison like no other" (on plastics), "In Defense of Food"

Their words

Recommended reading: "Poison like no other" (on plastics), "In Defense of Food"

Their words

(on food vs. "edible food-like substances"), "Poison Squad"

Their words

I do, but it's business as usual because we're we're in an intelligence explosion already and have been for decades. And when you look at GDP, it's basically the GDP curve that is an exponential weighted sum over so many aspects of the industry. Everything is gradually being automated has been for hundreds of years.

Their words

so I think there's there's an interesting point here because I do believe coding is like the perfect first thing for uh for a for uh these LLMs and uh agents and that's because coding has always fundamentally uh worked around text.

Their words

But even there I'm not actually looking at full automation yet. I'm looking for an autonomy slider and I almost expect that we are not going to instantly replace people. We're going to be swapping in AIs that do 80% of the volume. They delegate 20% of the volume to humans and humans are supervising teams of five AIs doing the call center work that's more rote.

Their words

so a good example recently was um Jeff Hinton's prediction that radiologists would not be a job anymore and this turned out to be very wrong in a bunch of ways right so radiologists are alive and well and growing even though computer vision is really really good at recognizing all the different things that they have to recognize

Their words

Now, number one, the first concession that people make all the time is they just take out all the physical stuff because we're just talking about digital knowledge work. I feel like that's a pretty major concession compared to the original definition which was like any task a human can do.

Their words

Um so I almost feel like because the internet is so terrible, we actually have to sort of like build really big models to compress all that. Uh most of that compression is memory work instead of like cognitive work. But what we really want is the cognitive part actually delete the memory

Their words

I also think humans collapse over time. Uh I think this is uh again these analogies are surprisingly good but humans collapse during the course of their lives. This is why children have completely u you know they haven't overfit yet and they will say stuff that will shock you

Their words

all of the samples you get from models are silently collapsed. They're silently, this is not obvious if you look at any individual example of it. They occupy a very tiny manifold of the possible space of um sort of thoughts about content.

Their words

the reason that I think this is kind of tricky is quite subtle. And it's the fact that anytime you use an LLM to assign a reward, those LLMs are giant things with billions of parameters and they're gameable.

Their words

A human would never do this. Number one, a human would never do hundreds of rollouts. Uh number two, when a person sort of finds a solution, they will have a pretty complicated process of review of like, okay, I think these parts that I did well, these parts I did not do that well, I should probably do this or that.

Their words

reinforcement learning is a lot worse than I think the average person thinks reinforcement learning is terrible. It just so happens that uh everything that we had before is much worse

Their words

And I kind of feel like the industry it's it's um it's over it's it's making too big of a jump and it's trying to pretend like this is amazing and it's not. It's slop and I think they're not coming to terms with it and maybe they're trying to fund raise or something like that.

Their words

I think um yeah, they're not very good at code that hasn't never been written before maybe is like one way to put it, which is like what we're trying to achieve when we're building these models.

Their words

So, don't write blog posts, don't do slides, don't do any of that. Like, build the code, arrange it, get it to work. It's the only way to go. Otherwise, you're missing knowledge.

Their words

These models don't really have this distillation phase um of taking what happened, analyzing it, obsessively thinking through it, um basically doing some kind of a synthetic data generation process and distilling it back back into the weights

Their words

anything that's in the weights it's kind of like a hazy recollection of what you read a year ago anything that you give it as a context uh at test time is directly in the working memory

Their words

And actually, you don't actually need or want the knowledge. I actually think that's probably actually holding back the neural networks overall because it's actually like getting them to rely on the knowledge a little too much sometimes.

Their words

So that's why I kind of call pre-training this kind of like crappy evolution. It's like the practically possible version with our technology and what we have available to us to get to a starting point where we can actually do things like reinforcement learning and so on.

Their words

a lot of what looks like learning is actually a lot more maturation of the brain and I think that actually very little reinforcement learning for animals and I think a lot of the reinforcement learning is actually like more like motor tasks. It's not intelligence tasks. So I actually kind of think humans don't actually like really use RL roughly speaking is what I would say.

Their words

we're not doing training by evolution. Uh we're doing training by basically imitation of humans and the data that they've put on the internet. And so you end up with these like sort of ethereal spirit entities because they're fully digital and they're kind of like mimicking humans.

Their words

And it just so turns out that um this was extremely early, way too early. so early that we shouldn't have been working on that, you know, uh because um if you're just stumbling your way around and keyboard mashing and mouse clicking and trying to get rewards in these environments, um your reward is too sparse and you just won't learn and you're going to burn a forest uh computing and you're never actually going to get something off the ground.

Their words

So like they don't have enough intelligence. They're not multimodal enough. They can't do computer use and all this kind of stuff. And they don't do a lot of things. You know, they don't have continual learning. You can't just tell them something and they'll remember it. And they're just cognitively lacking. And it's just not working. And I just think that it will take about a decade to work through all of those issues.

Their words

AutoSleep is basically a random number generator. Maybe there is a better app on Apple Watch for sleep tracking, but I haven't found it. Do not use.

Their words

I recommend Brave browser

Their words

Brave Search is a privacy-first search engine with its own index, unlike e.g. Duck Duck Go which is basically a nice skin for Bing

Their words

I like and use privacy.com to mint new credit cards for every single transaction or merchant.

Their words

I currently use Earth Class Mail but tbh I'm a bit embarrassed by that and I'm looking to switch to Virtual Post Mail

Their words

I currently use Earth Class Mail but tbh I'm a bit embarrassed by that and I'm looking to switch to Virtual Post Mail

Their words

I still use gmail just due to sheer convenience, but I've started to partially use Proton Mail

Their words

If you wish to hide your IP/location to services, you can do so via VPN indirection. I recommend Mullvad VPN

Their words

You can block ads by blocking entire domains at the DNS level. I like and use NextDNS

Their words

For more advanced users who like to tinker, pi-hole is the physical alternative.

Their words

I like and use The Little Snitch