Related posts
Salvatore Sanfilippo
GitHub
qwen-asr — C inference for Qwen3-ASR 0.6b and 1.7b transcriptions models
qwen qwen3 inference
The the words it uses that this site has seen
least often elsewhere. Posts are matched on those words alone —
nothing here is a summary of this one.
Sources
All Writing Bluesky GitHub x.com Korrents Site Newsletter Podcast Recommends Blog Papers
Top people
Sebastian Raschka
Dax Raad
Salvatore Sanfilippo
Nathan Lambert
Jeff Dean
Dylan Patel
Guillermo Rauch
Tomasz Tunguz
Paul Graham
Vicki Boykis
Vitalik Buterin
Casey Handmer
Showing Profile →
Show everything
Hiding Show them again
Show them again
Further back ↓
Hiding Show them again
20 September
Machine-learning researcher on open language models; writes the Interconnects newsletter and the RLHF Book, after leading post-training at Ai2.
My best guess is that for scaled RL most of the top Chinese AI labs are starting to use a lot of Huawei for inference and Nvidia for training (maybe not for weird architectures). As agent swarms, even more scaled post-training, etc becomes the norm, this will accelerate their domestic industry. China Related
Programmer; wrote Redis and hping, and blogs about C, systems programming and working alone.
19 September
Machine-learning researcher on open language models; writes the Interconnects newsletter and the RLHF Book, after leading post-training at Ai2.
Where I stand on RSI: A moderate's view on the recent events and trajectory of AI. I was underestimating how much we are likely to scale inference-time compute in the near future, but have not seen much to convince me that an intelligence explosion is near. interconnects.ai
Related
CEO of Vercel; creator of Next.js and Socket.IO. Writes at rauchg.com.
Looks like today may be a record day for token volume % of open models on Vercel AI Gateway: 🟦 Open 78.4% 🟨 Closed 21.6% While spend 💲 usually tells a different story, #3 and #4 today are Moonshot AI & DeepSeek. Adding Z.ai, their combined spend surpasses OpenAI (#2). (Do note that's the spend for inference of the model across providers (mostly in the US), not revenue going directly to the open weight labs.) OpenAI Related
18 September
Builds developer tools — SST, and now OpenCode at Anomaly. Posts constantly on x.com and writes almost never; the blog stopped in 2021.
we have some more senior engineering roles we're looking to fill at anomaly on opencode and other projects email engineering@anomaly.jobs if - you can do inference on $100M of hardware - you are amazing at product - you are deeply experienced with infra Related
i had fable doing some inference optimization work, it was making a ton of progress then it realized what it was doing and started to reject my requests so i switched to astra and told it to spawn fable subagents to do the work back in business Related
17 September
Venture capitalist and founder of Theory Ventures, previously a partner at Redpoint. Writes a near-daily blog on startups, SaaS metrics, data and AI.
15 September
AI research engineer working on large language models. He writes the Ahead of AI newsletter and is the author of Build a Large Language Model (From Scratch).
Some food for thought when designing benchmarks... So, here's a little computer-use (visual) comparison between GPT-5.6 Astra and Qwen3.8 Max. The task here was to recreate the image in the center using the Paint UI. Super interesting how the two different LLMs+Harnesses approached this totally differently by default. I.e., Astra tried to approach this by drawing and layering geometric shapes. Qwen approached this pixel by pixel. (Of course, the pixel-by-pixel result looks closer to the original, it's essentially a low-res version of that by nature.) So, the Qwen-generated image would surely… LLMs OpenAI Related
Co-founder of Y Combinator and Viaweb; essayist at paulgraham.com.
From one piece
@paulg on X
2 beliefs · x.com
Their words
Someone needs to define the unit, perhaps using a chain of increasingly hard problems, each pair of which can be solved by a single model.
Show the whole quote
@paulg on X x.com
Their words
Although you pay for AI by the token, that's not the unit of inference, because you get more problem solving per token as models improve.
Show the whole quote
@paulg on X x.com
13 September
Moroccan senior frontend developer writing at smakosh.com; builds side projects and client work under Smakosh LLC.
How to Make Money With LLM Inference Through Airside A practical guide to making money with LLM inference: list your deployment on @llmgateway through Airside, file the prices developers pay, choose the landing fee you accept, and keep the rest of the traffic you win. Link: llmgateway.io
LLMs Related
General partner at Andreessen Horowitz and author of The Cold Start Problem. Previously led rider growth teams at Uber; has blogged on growth, network effects and marketplaces since 2007.
current homelab setup for local AI experimentation: - hermes box hosted on a Framework Desktop Mainboard AI Max+ 395 - 5090 eGPU running Qwen 3.8 27B for fast tok/s LLM use - sometimes 150+ tok/s - 2x DGX Spark: running Deepseek v4 Flash 0731 - better but slower model - pi 5 for monitoring - Mac mini as a dev box - use Herdr and ohmypi/codex/claude depending on the use case - housed in a 10" DeskPi mini rack (mostly) Hermes is defaulted to local AI but with a homegrown routing plugin hitting a small low TTFT model (Arch-Router) to decide whether to go local or upgrade to cloud/frontier. Tryin… LLMs Anthropic Related
12 September
Technical educator, conference speaker and developer relations engineer based in Ottawa, Canada. Author of the Anubis bot filter and of over 400 articles at xeiaso.net.
11 September
Computer science professor who works on fast data processing; co-author of the simdjson parser and a weekly blogger about software performance since 2004.
What should be obvious is that inference, running a large language model, is the part that has to be cheap. Most of our hardware was not designed for that. LLM inference is limited by bandwidth. You do a great many matrix-vector multiplications against a huge weight matrix, and you barely reuse those weights. It is closer to streaming a video than to running a simulation or drawing a scene in a game. Graphics processors from Nvidia and others were built for something else. They pileed compute first, then bolted on high-bandwidth memory. It is expensive. An obvious answer is to invert the desi… LLMs Related
10 September
Former L8 engineer at Meta, Microsoft and Atlassian, now writing and building solo on agentic engineering. Writes Kun's Field Notes and posts a lot about AI coding agents on X.
omfg Qwen, GLM, Kimi, DeepSeek looks like they are all just claude Quoting @AnthropicAI We're publishing our most detailed threat intelligence report to date. It covers how people tried to misuse Claude—for cyberattacks, influence operations, surveillance, biology, and building weapons—and how we found and stopped them. We disrupted every operation in the report, and used the lessons… Anthropic Related
8 September
Writes The Diff, a newsletter on inflections in finance and tech read by hedge fund managers, founders and VCs. Previously worked in finance at SAC Capital/Point72.
6 September
Software engineer who writes about low-level and systems programming at nullprogram.com; author of the Elfeed feed reader, the Endlessh SSH tarpit and Enchive.
AI research engineer working on large language models. He writes the Ahead of AI newsletter and is the author of Build a Large Language Model (From Scratch).
Reasoning from scratch round 2: In this video, I cover the text generation process in LLMs and KV caching (to prepare the base model before adding reasoning techniques in the upcoming ones). 00:00 Introduction and reasoning model demo 01:55 How to work through the book 05:00 Chapter 2 overview 08:25 Checking PyTorch and hardware support 10:26 Apple silicon and MPS caveats 15:00 Cloud GPU options 16:08 Tokens and tokenization 18:20 Qwen3 and the Reasoning From Scratch package 23:05 Encoding and decoding text 26:24 Downloading weights and selecting a device 31:01 Loading the pretrained Qwen3 mo… LLMs Related
3 September
Venture capitalist and founder of Theory Ventures, previously a partner at Redpoint. Writes a near-daily blog on startups, SaaS metrics, data and AI.
How large is Snowflake's AI revenue? Management lowered product gross margin guidance from 76% → 74% due to "fast-growing AI workloads." If GPU inference runs at ~25%–35% gross margins, that 200 bps compression implies ~$60M–$75M/quarter (~$240M–$300M ARR) in direct AI token consumption. Related
Builds developer tools — SST, and now OpenCode at Anomaly. Posts constantly on x.com and writes almost never; the blog stopped in 2021.
23 August
CEO of Vercel; creator of Next.js and Socket.IO. Writes at rauchg.com.
Their words
the demand for intelligence is highly elastic: as inference costs fall, usage grows rapidly.
Show the whole quote
@rauchg on X x.com
20 August
Programmer, photographer and musician, best known for the Python Requests HTTP library and other Python tooling.
I wrote a new essay about a strange inversion in AI-assisted programming. Old-school programming made sustained attention the scarce resource. You entered a tight loop with the machine, and the difficulty itself kept you engaged. Agentic coding changes that. To reach the same intensity, you keep more work in flight, run more agents, and consume more inference. Flow becomes throughput, then the usage meter appears. There are legitimate economic reasons for limits. But when higher plans are explicitly sold as freedom from interruption, uninterrupted access to your own momentum becomes the produ… kennethreitz.org
Related
11 August
AI research engineer working on large language models. He writes the Ahead of AI newsletter and is the author of Build a Large Language Model (From Scratch).
Whoa, Meta released a new open-weight LLM yesterday, something that hasn't happened since the good old Llama days. Their Meta Muse Glimmer model is a 30B multimodal reasoning model with a Gemma-like architecture design. (“Glimmer” is probably a wordplay on “Spark,” the more likely capable model from which Glimmer was distilled. Muse Spark is only available through Meta’s Model API, though.) Architecture-wise, here are some of the main points: 1. "Only" a 131k context window, compared to Qwen3.6 and Gemma 4, which support 2x that natively; it's reasonable, but maybe on the shorter end in the a… LLMs Related
Programmer; wrote Redis and hping, and blogs about C, systems programming and working alone.
30 July
AI research engineer working on large language models. He writes the Ahead of AI newsletter and is the author of Build a Large Language Model (From Scratch).
Consistent with what I found with Qwen3.6 a while back: Claude Code uses 2-3x as many tokens than (many) other harnesses at similar success rate. - Unoptimized? - Buggy? - Deliberate (coz that helps it in more challenging cases)? Need to find some time to investigate more... Quoting @composio We ran Kimi K3 through 3 agent harnesses (Claude Code, Hermes, Kimi Code) on 28 identical tasks. All 3 harnesses completed the tasks at similar success rates, but the interesting story is token efficiency: the same task cost up to 30x more tokens depending on the harness. 🧵🧵 Anthropic Related
Chief Scientist at Google DeepMind and Google Research, at Google since 1999, where he co-created MapReduce, Bigtable, TensorFlow and the TPU.
From one piece
Jeff Dean: The 1% Rule for Building in AI
3 beliefs, in the piece's order there
Their words
And um if you build a specialized chip for low precision dense linear algebra and can't do anything else that turns out to be really useful for machine learning inference uh even though it can't run Chrome or Word or whatever.
Show the whole quote
Jeff Dean: The 1% Rule for Building in AI youtube.com
Their words
Um, and that's a very very useful general technique is you know inference time compute to perform search over plausible ways of solving the problem that can get much much higher performance or much more reliability in longunning agent flows.
Show the whole quote
Jeff Dean: The 1% Rule for Building in AI youtube.com
energy
Their words
Yeah, I mean it's a little different, but I think uh you're going to see more and more uh uh high performance and um low energy uh inference hardware systems because I think everyone is now realizing that inference is the key to making you know these agent-based systems be available to more and more people and that latency is really important and that specialization of the hardware is a really key way you can make uh things that are more energy efficient and lower latency than more general purpose uh computational devices like say GPUs or TPUs
Show the whole quote
Jeff Dean: The 1% Rule for Building in AI youtube.com
22 July
Machine-learning researcher on open language models; writes the Interconnects newsletter and the RLHF Book, after leading post-training at Ai2.
18 July
AI research engineer working on large language models. He writes the Ahead of AI newsletter and is the author of Build a Large Language Model (From Scratch).
30 June
CEO of Positive Sum and founder of Colossus. Hosts Invest Like the Best, interviewing investors and business leaders; the guests do most of the talking there.
29 June
Economics professor at the University of California, Berkeley and a former Deputy Assistant Secretary of the US Treasury. Author of Slouching Towards Utopia and writer of the Grasping Reality newsletter.
Their words
Right now: llama3.2:3b appears to be the model for: is this email urgent?qwen3:8b appears to be the model for: summarize this 5000-word article. llama3.3:70b(q8) appears to be the model for: let’s write or debug some computer code.
Show the whole quote
braddelong.substack.com
27 June
AI research engineer working on large language models. He writes the Ahead of AI newsletter and is the author of Build a Large Language Model (From Scratch).
Their words
Here, I am primarily going to use Qwen3.6 with the Qwen-Coder coding client.
Show the whole quote
magazine.sebastianraschka.com
Their words
Here, I am primarily going to use Qwen3.6 with the Qwen-Coder coding client.
Show the whole quote
magazine.sebastianraschka.com
Their words
Alternatively, if you don't want to use Qwen3.6, Cohere's North Mini Code is probably the most interesting, capable alternative at this size class right now.
Show the whole quote
magazine.sebastianraschka.com
15 June
Systems and developer-tools engineer at Cloudflare as of 2026, on durable infrastructure for AI agents; worked on React and PartyKit before that. Writes at sunilpai.dev.
Machine learning engineer working on recommender systems, personalization and information retrieval. Previously worked on LLMs and LLM infrastructure at Mozilla.ai and on ML and recsys at Duo, Tumblr, Automattic and Comcast; wrote the 'What are embeddings?' text and ran the Normconf conference.
Their words
For my local setup, I'm currently using Pi as the agent harness and LM Studio as the inference server
Show the whole quote
vickiboykis.com
Their words
For my local setup, I'm currently using Pi as the agent harness and LM Studio as the inference server
Show the whole quote
vickiboykis.com
9 June
Cryptographer; teaches cryptography at Johns Hopkins and writes A Few Thoughts on Cryptographic Engineering.
4 June
Canadian full-stack web developer who makes JavaScript and CSS video courses and co-hosts the Syntax podcast.
Cloudflare has acquired Vite / VoidZero This gives Cloudflare control over the entire stack. They have all the primitives from frontend/backend framework, linting, testing, formatting, JS runtime, db, kv, inference, blob storage, access, etc... smart move! blog.cloudflare.com
Related
27 May
Builds developer tools — SST, and now OpenCode at Anomaly. Posts constantly on x.com and writes almost never; the blog stopped in 2021.
From one piece
Building OpenCode with Dax Raad
3 beliefs, in the piece's order there
Their words
cuz because we rent GPUs at scale to run the models and we still use middleman by the way. So we're not like going all the way down to the down to the floor. Even for us there are some models the sticker price and the cost to us there's like an 80% margin in there.
Show the whole quote
Building OpenCode with Dax Raad youtube.com
Their words
There's always negative sentiment that exists for any business that's getting hyped. They have no incentive to correct it. Um so again it's complicated because I know the training costs are a big part of it. Uh the R&D department is is hugely expensive but long-term inference makes sense as a business and I think it it always will.
Show the whole quote
Building OpenCode with Dax Raad youtube.com
Their words
The demand for inference is growing. So, like I don't think it's linearly growing. I think it might even be exponentially growing. But we haven't made our production of GPUs grow exponentially. That's like kind of a linear process. So as those lines intersect, there's going to be uh tightening.
Show the whole quote
Building OpenCode with Dax Raad youtube.com
25 May
Programmer; wrote Redis and hping, and blogs about C, systems programming and working alone.
Distributing LLM inference in DwarfStar High end NVIDIA cards, and the server and power needed to run them, cost a lot of money, especially if you plan to reach enough VRAM to run massive models. The alternative, so far, has been Apple hardware, or the DGX Sp…
LLMs Related
22 May
Programmer and writer on computer architecture, performance, and software reliability. He has worked on CPU design at Centaur Technology and on software at Google and Microsoft, and writes long-form technical essays at danluu.com.
Why are so many people so sure that the big AI providers are losing money on inference? It reminds me of the comments about how Uber can never make money. Their unit economics were fine and they were only losing money because they chose to do so on customer acquisition. Related
13 May
Danish software engineer and Microsoft technical fellow, creator of Turbo Pascal and Delphi and lead architect of C# and TypeScript.
Their words
because if you were to force AI to write a type annotation on everything, then it would probably get it wrong more often because now it has to keep track of all these types and and it and it has to just repeat itself over and over and over, right? And so, types are important where there's no context.
Show the whole quote
youtube.com
2 April
Co-founder of Ethereum. Publishes long essays on mechanism design, governance and what cryptography is for, and returns to earlier positions to say which parts they no longer hold.
Their words
As it turned out, ollama was not able to fit Qwen3.5:35B onto my GPU, but llama-server could. Hence, from that day forward, I resolved to cease being a cave-dwelling noob, and use llama-server (via llama-swap to make model swapping easier).
Show the whole quote
vitalik.eth.limo
Their words
I have been using the Qwen3.5:35B model and have tried it on each of these, and I also tried the one-step-larger 122B.
Show the whole quote
vitalik.eth.limo
23 March
Co-founder of NVIDIA and, as of 2026, its chief executive; the company designs the GPUs most large AI models are trained on.
Their words
that was always illogical to me because inference is thinking, and I think thinking is hard. Thinking is way harder than reading.
Show the whole quote
youtube.com
13 March
Founder and chief analyst of SemiAnalysis; reports on semiconductor supply chains, AI datacentre economics and what the chip export controls actually do.
From one piece
Dylan Patel — The single biggest bottleneck to scaling AI compute
2 beliefs, in the piece's order there
Their words
They could release claw slow mode and have an increase in tokens per dollar by a significant amount. Um they could probably like reduce the price of Opus 46 by you know 4x 5x and reduce the speed by another by maybe just like 2x like the curve on inference throughput versus speed is there already just on hm um and yet they don't um because no one actually wants to use a slow model
Show the whole quote
youtube.com
Their words
So when you look at inference at let's say 100 tokens a second for deepseek and kimk 2.5 hopper versus blackwell the performance difference is on the order of 20x
Show the whole quote
youtube.com
15 February
Programmer; wrote Redis and hping, and blogs about C, systems programming and working alone.
13 February
Programmer; wrote Redis and hping, and blogs about C, systems programming and working alone.
Co-founder and CEO of Anthropic; previously VP of research at OpenAI.
Their words
There's There's nothing preventing longer context from working. You just have to train at longer context and then learn to to serve them at inference. And both of those are engineering problems that we are working on and that I would assume others are working on as well.
Show the whole quote
youtube.com
10 February
Founder of Terraform Industries and formerly a physicist at NASA's Jet Propulsion Laboratory; writes about energy, space and manufacturing.
24 January
AI research engineer working on large language models. He writes the Ahead of AI newsletter and is the author of Build a Large Language Model (From Scratch).
30 December 2025
AI research engineer working on large language models. He writes the Ahead of AI newsletter and is the author of Build a Large Language Model (From Scratch).
28 December 2025
Founded PSPDFKit in 2011 and ran it for a decade. Came back from a break to work on AI agents — the OpenClaw project, and OpenAI, joined in February 2026. Writes at steipete.me.
4 December 2025
Professor of Cognitive and Computational Neuroscience at the University of Sussex, co-director of the Sussex Centre for Consciousness Science, and author of Being You: A New Science of Consciousness.
17 September 2025
Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.
I've been nerdsniped by the idea of Semantic IDs. Here's the result of my training runs: • RQ-VAE to compress item embeddings into tokens • SASRec to predict the next item (i.e., 4-tokens) exactly • Qwen3-8B that can return recs and natural language! eugeneyan.com
Related
6 September 2025
AI research engineer working on large language models. He writes the Ahead of AI newsletter and is the author of Build a Large Language Model (From Scratch).
15 August 2025
Founder of Terraform Industries and formerly a physicist at NASA's Jet Propulsion Laboratory; writes about energy, space and manufacturing.
solar data centers
Their words
actually it turns out that you can significantly decrease power consumption with a very small reduction in overall compute. So if you if you've got like three really bad days in a row or something, you can actually just like you can dial back your power usage quite a lot without compromising your inference or or um or training.
Show the whole quote
youtube.com
9 August 2025
AI research engineer working on large language models. He writes the Ahead of AI newsletter and is the author of Build a Large Language Model (From Scratch).
31 July 2025
Founded PSPDFKit in 2011 and ran it for a decade. Came back from a break to work on AI agents — the OpenClaw project, and OpenAI, joined in February 2026. Writes at steipete.me.
3 February 2025
Founder and chief analyst of SemiAnalysis; reports on semiconductor supply chains, AI datacentre economics and what the chip export controls actually do.
OpenAI
Their words
OpenAI has a fantastic margin. When they're doing inference, their gross margins are north of 75%. So that's a four to five X factor right there of the cost difference, is that OpenAI is just making crazy amounts of money because they're the only one with the capability.
Show the whole quote
youtube.com
30 January 2025
Creator of Flask and Jinja. Writes about software at lucumr.pocoo.org.
Their words
This allows me to run models locally on my MacBook Pro M1 Max. With the 64GB of RAM it has, it’s a pretty potent machine for basic inference despite it being three years old.
Show the whole quote
lucumr.pocoo.org
31 December 2024
Co-creator of Django and creator of Datasette; writes daily at simonwillison.net.
Their words
To understand more about inference scaling I recommend Is AI progress slowing down?
Show the whole quote
simonwillison.net
6 August 2024
Founding member of OpenAI and former director of AI at Tesla; creator of nanoGPT and the term "vibe coding".
19 June 2024
Co-founder and CEO of Perplexity, an AI answer engine; previously a research scientist at OpenAI and a PhD student at UC Berkeley.
Their words
I think if we can achieve that amount of inference compute, where it leads to a dramatically better answer as you apply more inference compute, I think that will be the beginning of real reasoning breakthroughs.
Show the whole quote
youtube.com
10 January 2023
Machine-learning researcher; has written the Lil'Log survey posts on how a model technique works since 2017, and worked at OpenAI from 2018 to 2024, latterly leading its safety systems team.
Nothing matches. Show everything
A korrent is a belief a person has stated in their own words: one
sentence stating the claim, backed by a quote and a source, kept at
korrents.com .
Under a name here, the quoted block is what they actually said.
The korrent beneath it is the claim those words support, in
korrents' wording — tap it to see the record, its source, and who
else holds it.
Nobody here wrote their own korrents. They are compiled from public
statements, and a person can change their mind, which is recorded too.
Got it
About the English under a post
Some people here publish in a language other than English. Where they
do, this site shows a machine translation beneath the post, in
this typeface — the site's own, not theirs.
The post itself is never changed, moved or hidden: what is set in the
serif above is exactly what the person published, and it is what to
quote them on. A translation can be wrong in ways that matter,
especially about tone.
Only the post's own words are translated. A quoted post, a linked
article and a belief on korrents.com
are left in their original language.
Got it