The quantization is probably the most important thing to know and its not clear at all from the dense word salad

To put this on perspective, consider that with 128GB of unified RAM it can run even fast: https://x.com/antirez/status/2119392658661749175 Like, coding-agent acceptably fast. See later in the reply for the prefill at ~300 t/s but can go even faster for larger prefills.

Something to keep in mind with aggressive quantisation, is that it's been empirically proven [1] that (1) quantisation has non-linear performance behaviour, and (2) generally the peak size-to-performance ratio occurs at 4-bit; anything lower and you will see significant reduction in performance.

Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1.

[1] "Reliability Scaling Laws for Quantized Large Language Models", TUM, https://arxiv.org/pdf/2607.10855

At which quantization? That's the only thing I want to know.

EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.

This is a very impressive achievement in an unexpected direction. I'm really looking forward to trying out qwen on this system.

Strata, Project Maya, the FP8 3x speedup on memory-bound workloads on Ampere cards and now llama-mod chip. I’m amazed with all these optimisations and thinking outside the box. Well done, don’t stop!

hm... this is the first time. TIL something from the HN comments more than the actual article. I'm relatively new to the AI space and the idea of quantization is something I'll have to take forward when evaluating future models.

I wish people would clean up the ai spammed emoji from the github readme. The readability is just horrible.

This is quite an obnoxious Readme.md with AI slop. It wraps around llama.cpp but without seeing what platform assumptions it makes its unclear for other readers whether this is worth testing.

I very recently spent a lot of time using deepseek v4 flash 0731 before it was superseded by newer more capable things (glm 5.3-flash, v4.1 flash 1017, etc) and I am exceedingly skeptical of anyone that claims to get good QUALITY results out of it at that size. Whatever they're doing to shrink it will make it worse than an unsloth Q2XS quantization of it or equivalent. That size of model needs something like 150GB of RAM to run properly with full unquantized kv cache and not with a ton of training data discarded.

I remain exceedingly skeptical of people who are claiming to run fairly large models in ridiculously tiny sizes. For comparison a 'regular' version of v4 flash 0731 has file sizes more like this: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF

Even the 97GB Q2XS will show very significant divergence from the full precision thing. Running a lobotomized model just to say you're doing it is really not accomplishing anything special.

Anyone else finding these AI generated pages really hard to read?

It is 2 bit? anybody tested this? is any good?

[flagged]