Cerebras is good on paper but all I can reasonably access is "GPT OSS 120b, Qwen 3.8 27B" which aren't very good for my use.

Openrouter on nitro gets me there. Although the TPS is 5X less still.

I have tried GPT OSS 120b on cerebras and it flies. But it's mostly wrong.

Citing full precision benchmarks then just going with 4 bit quants

Ah, this prompted a few questions from me. Perhaps someone with more hardware background could chime in.

They don't really show it here, but supposedly a Cerebras' chip is sub-divided into a 2D mesh of what they call PEs (cores). So data flow generally flows in two axis, and maintains consistent timings to its 4 neighboring cores. They show the layers flowing from left-to-right being streamed by PEs, but I wonder how true that ends up being: maybe the communications are easier to stream via diagonals, or some other shape that has better topology for data flow.

Then, even from die shots of CPUs and GPUs we often see that cores are tiled as squares, and those don't have the same cross-core connection and streaming requirement. Now, I assume that some parts of what we call a core can be moved around more freely while maintaining whatever power/heating/timing requirements we have, so perhaps it's easier to 'force' a square by moving those loose parts around. But why do we confine those to squares? Is super-core layout just something not worth investing time into, like how interconnects are timed, non-convex tilings, and distances to I/O?

And for streaming architectures like Cerebras, why aren't other types of tilings considered, something like a hexagon grid? This could have 3 inputs / 3 outputs, which could allow more complex data flow (?). Similarly, we could have 8 connections with longer diagonals in a square tiling, etc.

I guess, what are the tradeoffs in place such that so many chip designs seem to reuse square tilings both locally and globally?

I tried Cerebras this week for my project and their 27b model at 2000tks would be perfect for it. Unfortunately it doesn't do the 2k tokens per second which is what I am looking for.

They've shown the calculation for number of concurrent requests limited by kv cache size. I guess for each new request loaded into the queue, the kv cache for that context will need to be loaded from a supporting storage server, that will need to have a lot of ram to store the active caches of multiple requests?

> The total number of concurrent requests represents the number of simultaneous requests moving through the pipeline at any point in time. Unlike a GPU batch, those requests don’t wait for one another to finish, they stream through the pipeline continuously.

What do they mean by “unlike a GPU batch”? It’s not a hardware limitation of GPUs right? It’s just the scaffolding to do this in software isn’t available right

Lol these guys have another stupid quantization scheme they had mentioned earlier. Most models are stupider at least from them gateways.

Such a weird company with a terminally online on linkedin ceo, garbage devrel on X. Cant really be friends with CSPs coz everyone has their own chip, no consumer solution either, no neocloud would want to onboard them either coz its baggage no one wants to risk dev time/customer money with.

A wafer looks pretty big. I'm surprised it has only 44GB of memory available. Doesn't that seem a tad .... low? I'm not a hardware person, but I would have assumed much more memory would fit into a wafer?

Ok I'm sold, where to buy? :D

Extreme speeds are actually interesting if you think about it. We are apparently getting some interesting results from LLMs nowadays. If they would put 100x effort in 10% the time it takes, or imagine 1% where how long tool calls take become the bottleneck, that would actually probably feel like a really special kind of "super intelligence".

At 20 kW average consumption per WSE-3T wafer this gives roughly 1.18 MW for the whole system. Assuming 1500 tokens per second and 190 concurrent requests this gives 4.1 J/token. Which is good I suppose for the size of the model?

Cerebras is the Transmeta of the AI wave.

could push kv-cache to off chip cheaper storage like dram. Inference engine manages decode for a single user to run to completion before decode a different users so kv-cache fetch from dram to sram is limited.

[flagged]

I'd want to know is what a 59-wafer pipeline costs in failure domain. The GPU deployment can lose a node and degrade; Because, here the model is the rack, so one bad wafer takes the whole thing down, and a pipeline that has to be live end to end doesn't get a rolling upgrade. How do you drain one?

[dead]