Jeff Dean

Their words

Um, another way you can get more experience for yourself is to just write down a bunch of things you think might be important in the next 12 months. And maybe you pick one of them to work on, but go back and evaluate in 12 months of these other things, which ones actually seemed important or which ones did other people in the world go out and and create and which ones did they did not seem to do yet.

Their words

Yeah, I mean I think it's really having incredibly good taste in what you ask your agents to work on, right? That is the the crux of you know from my background uh a research problem. You know, a researcher can have all the tools and all the techniques, but often most of the battle is what problem are you gonna spend your time on?

Their words

Um, and to give you an example of a a a use of a coding agent that works extremely well is you can ask today's models to translate software from one computer language to another very effectively because in that case you actually have a incredibly detailed specification.

Their words

So we've always told computer scientists from the very beginning that really it's really important to specify what it is, what's the software that you're writing is trying to accomplish before then going and writing it. And so now we actually have agent-based systems that can do the writing, but the importance of specifying what what it is you want has actually gone up because before you'd be handing it off to a very intelligent human who maybe has context or can ask you follow-up questions.

Their words

I mean I think like if you look at uh my colleagues work on say alpha fold that was a very specific model for uh protein folding and it was highly successful um and was able to really handle that domain quite well so that all of a sudden you now have this amazing tool and model that can give you answers to questions about proteins and their structure um really effectively um but it's not a general model it's a very specific one and there are other I domains where that kind of approach can work really well. Uh maybe in material science or chip design or things like that that uh will enable you to leverage the capabilities of a very accurate but but niche model uh to do things that are hard today.

Their words

Yeah, I mean I think uh sometimes it's uh a product that you build that might have access to particular kind of data that the underlying model might not the a general model. So it might be you're building something to help users organize all their own personal information and the model won't necessarily have access to that. And so there you can have a big advantage because all of a sudden your model has visibility or your product has visibility into important data.

Their words

And if they're completely failing, that's probably a good sign. If they're kind of able to do some of it but not very well, that's maybe not a great sign because that's a probably a a sign that the capability is starting to be present in those models and with more training data or larger scale models or or whatever it's likely to get better. So um you know look for something where the model succeeds 0% or 1% of the time not not 20%.

Their words

Um but I I would also caution that the general models are definitely getting better at a broader and broader range of things. So you have to figure out, you know, is that thing you're working on, is that going to be a durable thing or do you think the models uh at the forefront are going to get better at that in the next six months or 12 months or is it something they're not going to be able to do for a couple years or three years? And you know, you you want to weigh that as you're as you're deciding what to work on.

Their words

Um, and that's a very very useful general technique is you know inference time compute to perform search over plausible ways of solving the problem that can get much much higher performance or much more reliability in longunning agent flows.

Their words

Um, and sometimes that's because the model is trying to do something it doesn't have a lot of experience doing. So it's been trained on a whole set of things and as soon as you get a little bit off the distribution of things it knows how to do then like most machine learning models it will you know its performance will suddenly will start to degrade and the farther you get off the comfort zone of what it knows how to do the the more likely it is to to not work as well.

Their words

And often you can actually make the model work better and succeed at that kind of problem by not just adjusting the model parameters which is hard to do from the outside but from you know creating better guidelines for the model you know writing skills for the model to know how to use different tools that would be incredibly useful for solving this particular class of problem.

Their words

And the nice thing about that is that information is really clear to the model, unlike the training data the model was trained on where it's all kind of like trillions of tokens stirred together into a soup of of hundreds of billions or trillions of parameters, but it's all less clear than the actual context uh that the model sees directly for this particular problem or uses use case.

Their words

Um because if you didn't have that thousandx difference then you know you wouldn't have to do batching but you have to do batching of you know many examples or maybe many tokens at once in order to amortize that data movement

Their words

Yeah, I mean I think if you looked at what is important in AI systems these days, you would want to know things like the bandwidth between you know your main memory system on your accelerator to the onchip memory to the um you know the multiplier unit or whatever. You want to know how much energy does it take to do a single multiplier operation.

Their words

And um if you build a specialized chip for low precision dense linear algebra and can't do anything else that turns out to be really useful for machine learning inference uh even though it can't run Chrome or Word or whatever.

Their words

I mean I think um probably one thing is people don't quite realize how possible it is to have you know agent-based systems that can run not just for an hour or two hours on a problem you care about but for some problem domains and with highly capable models underlying them you can get them to run for days or weeks and do really really complicated tasks

Their words

Yeah, I mean it's a little different, but I think uh you're going to see more and more uh uh high performance and um low energy uh inference hardware systems because I think everyone is now realizing that inference is the key to making you know these agent-based systems be available to more and more people and that latency is really important and that specialization of the hardware is a really key way you can make uh things that are more energy efficient and lower latency than more general purpose uh computational devices like say GPUs or TPUs

Their words

you know, because sometimes if you just squint at a problem and you think about not necessarily being anchored on exactly how that problem is solved today, but how you would solve it from first principles, you can come up with really good ideas that are, you know, maybe not what other people are thinking about.

Their words

Yeah, I mean I feel like uh the models have been getting a lot better at sort of agent-based longer running coding tasks and it seems pretty clear that they are now actually pretty capable and depending on exactly your definition of of junior engineer it seems pretty spot-on I would say.

Their words

For writing code, I use emacs and Google's internal distributed build system (a version of which was open sourced as Bazel) and our version control system

Their words

For writing, commenting and collaborating on internal technical documents, I use Google Docs.

Their words

For technical papers, I and my coauthors usually use LaTeX. For giving presentations, I almost exclusively use Google Slides.

Their words

For giving presentations, I almost exclusively use Google Slides.

Their words

For handling email, I use GMail with lots of filters to automatically file and label message (I get about 1400 email messages per day, so I have to be quite efficient to deal with it all in a reasonable amount of time).

Their words

Most of the code I write is in C++, although I've written fair amounts of code in Java, Python, Perl, Self, Cecil, x86 assembly, and Pascal over the years.