Very cool paper and some interesting things for local serving. Though its a little apples-to-oranges we do publish live energy stats for all models on our service here https://portal.neuralwatt.com/energy-pricing in case you are interested in what this looks like on the cloud side. FWIW DSV4.1 flash is really getting popular due to its IPW.

Some of the items like model routing, if you do it per request instead of per session, can break down on the cloud from an energy and cost POV since one of the best things you can do for both is to maintain the KV cache which both reduces time component of energy and the quite expensive prefill energy.

I am keen on the future where we have local/cloud hybrid serving which is cache aware. I do think that could be the best use of energy resources for AI.

Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winter time the GPU also doubles as a 300W in-house heater. Not to mention avoiding personal data collection and re-selling.

Saw some measurements on SBC NPUs (3588) and that did seem to have a decent win on power over CPU...but also a perplexity loss relative to CPU so think this will prove quite hard to reliably quantify in practice.

Unless I misread it, are they saying local GPUs use less energy?

That’s surprising, almost unbelievable, due to batching. Local is usually not batched.

this is the metric i've been waiting for. we run everything local (ollama + neo4j) for compliance reasons, so 'quality per watt' is literally our budget line. one data point from our setup: qwen2.5:3b on an m2 macbook handles nl-to-cypher for simple graph schemas at ~3-5s per answer, and the energy cost is a rounding error compared to shipping the same queries to a frontier api. the hard part was never the model though, it was parsing pdfs locally without a vision model. would love to see parsing/ocr covered in future benchmarks.

> We propose intelligence per watt (IPW), task accuracy per unit of power

Stupid metric. It's not because a model is better performing that it necessarily requires more energy or compute.