From one piece Jeff Dean: The 1% Rule for Building in AI 15 beliefs · youtube.com
-
designprotein structure prediction
Their words
I mean I think like if you look at uh my colleagues work on say alpha fold that was a very specific model for uh protein folding and it was highly successful um and was able to really handle that domain quite well so that all of a sudden you now have this amazing tool and model that can give you answers to questions about proteins and their structure um really effectively um but it's not a general model it's a very specific one and there are other I domains where that kind of approach can work really well. Uh maybe in material science or chip design or things like that that uh will enable you to leverage the capabilities of a very accurate but but niche model uh to do things that are hard today.
-
korrents.com
A startup's durable advantage over a general model is having visibility into data the general model will never see.Their words
Yeah, I mean I think uh sometimes it's uh a product that you build that might have access to particular kind of data that the underlying model might not the a general model. So it might be you're building something to help users organize all their own personal information and the model won't necessarily have access to that. And so there you can have a big advantage because all of a sudden your model has visibility or your product has visibility into important data.
-
Their words
And if they're completely failing, that's probably a good sign. If they're kind of able to do some of it but not very well, that's maybe not a great sign because that's a probably a a sign that the capability is starting to be present in those models and with more training data or larger scale models or or whatever it's likely to get better. So um you know look for something where the model succeeds 0% or 1% of the time not not 20%.
+ 12 more
-
korrents.com
Before building on a gap in the general models, work out whether that gap survives six months or three years.Their words
Um but I I would also caution that the general models are definitely getting better at a broader and broader range of things. So you have to figure out, you know, is that thing you're working on, is that going to be a durable thing or do you think the models uh at the forefront are going to get better at that in the next six months or 12 months or is it something they're not going to be able to do for a couple years or three years? And you know, you you want to weigh that as you're as you're deciding what to work on.
-
Their words
Um, and that's a very very useful general technique is you know inference time compute to perform search over plausible ways of solving the problem that can get much much higher performance or much more reliability in longunning agent flows.
-
Their words
Um, and sometimes that's because the model is trying to do something it doesn't have a lot of experience doing. So it's been trained on a whole set of things and as soon as you get a little bit off the distribution of things it knows how to do then like most machine learning models it will you know its performance will suddenly will start to degrade and the farther you get off the comfort zone of what it knows how to do the the more likely it is to to not work as well.
-
Their words
And often you can actually make the model work better and succeed at that kind of problem by not just adjusting the model parameters which is hard to do from the outside but from you know creating better guidelines for the model you know writing skills for the model to know how to use different tools that would be incredibly useful for solving this particular class of problem.
-
Their words
And the nice thing about that is that information is really clear to the model, unlike the training data the model was trained on where it's all kind of like trillions of tokens stirred together into a soup of of hundreds of billions or trillions of parameters, but it's all less clear than the actual context uh that the model sees directly for this particular problem or uses use case.
-
korrents.com
Batching exists only because moving data costs about a thousand times more energy than computing on it.Their words
Um because if you didn't have that thousandx difference then you know you wouldn't have to do batching but you have to do batching of you know many examples or maybe many tokens at once in order to amortize that data movement
-
korrents.com
The numbers an AI engineer should know by heart are memory bandwidths and the energy cost of a single operation.Their words
Yeah, I mean I think if you looked at what is important in AI systems these days, you would want to know things like the bandwidth between you know your main memory system on your accelerator to the onchip memory to the um you know the multiplier unit or whatever. You want to know how much energy does it take to do a single multiplier operation.
-
Their words
And um if you build a specialized chip for low precision dense linear algebra and can't do anything else that turns out to be really useful for machine learning inference uh even though it can't run Chrome or Word or whatever.
-
korrents.com
Agents can already be run for days or weeks on a single hard problem, and almost nobody has internalised that.Their words
I mean I think um probably one thing is people don't quite realize how possible it is to have you know agent-based systems that can run not just for an hour or two hours on a problem you care about but for some problem domains and with highly capable models underlying them you can get them to run for days or weeks and do really really complicated tasks
-
Their words
Yeah, I mean it's a little different, but I think uh you're going to see more and more uh uh high performance and um low energy uh inference hardware systems because I think everyone is now realizing that inference is the key to making you know these agent-based systems be available to more and more people and that latency is really important and that specialization of the hardware is a really key way you can make uh things that are more energy efficient and lower latency than more general purpose uh computational devices like say GPUs or TPUs
-
Their words
you know, because sometimes if you just squint at a problem and you think about not necessarily being anchored on exactly how that problem is solved today, but how you would solve it from first principles, you can come up with really good ideas that are, you know, maybe not what other people are thinking about.
-
korrents.com
AI models have reached the level of a junior engineer on longer-running, agent-based coding tasks.Their words
Yeah, I mean I feel like uh the models have been getting a lot better at sort of agent-based longer running coding tasks and it seems pretty clear that they are now actually pretty capable and depending on exactly your definition of of junior engineer it seems pretty spot-on I would say.