Inkling-2-Mini in 5.4 GB Memory

209 points - today at 6:02 PM

Source

Comments

an0malous today at 7:10 PM
Our future hinges on the capability gap between frontier lab models and open source local models
syntaxing today at 7:54 PM
I really wanted to like this model but it is sooooo slow. Third of the speed of Qwen 27B on my R9700. I don’t know if it’s an artifact of looped transformer or unoptimized kernel.
manlymuppet today at 7:24 PM
My Quantitive!
Dwedit today at 6:32 PM
How does this method compare to the use of an importance matrix?
IronWolve today at 6:49 PM
Only mx version?
woadwarrior01 today at 7:06 PM
From the model card:

> The model is distributed as a single checkpoint of about 20GB, in mixed NVFP4 and MXFP8 precisions.

What's the point in quantizing a QAT model? Just for social media karma?

cheapcheerful today at 7:12 PM
"Look at my quant 2"

/igotthatreference.jpg

hugmynutus today at 7:48 PM
*TL;DR* Promising initial results but shoddy research. OP's quant did not get 1st or 2nd prize in that math competition.

----

Gave it a quick browse to verify what my local clanker told me, it seemed to check out

I believe it is Dynamically allocating weights based on fisher information of conversation traces & For some K-quant look up table stuff it can store in an importance matrix to get better compression.

The results are complete nonsense and really don't back up any of the README claims. For Inkling-2-Mini; at 5.4 GB its KL is worse than Unsloth's UD-IQ1_S: 0.0895 vs 0.0808 on English/chat/code, and 0.287 vs 0.124 across 13 languages (e.g.: horrible). It then achieved better results by post-training the qaunt'd model on sample inputs (AFAIT).

According to anthropomorphic-clud their results for GSM8K & HumanEval are within the noise regime and vastly over stated.

collabspherestu today at 8:06 PM
[flagged]
peter_d_sherman today at 8:27 PM
[dead]