I've found that using structured outputs solves this problem much better. Instead of letting a model generate only "A", "B" or "C" and looking at the probs, have it directly generate "Legitimate", "Spam" or "Phishing" or any other pre-defined option from a set of multi-token sequences. Behind the scenes it boils down to something quite similar, but you're not running into the risk that the model actually wanted to say "A phishing attempt seems likely, so answer (C) is correct.", which would lead "A" to have the highest probability in the first token. You can even use a reasoning budget this way either via inherent reasoning or a free-form part preceding the remaining output structure. You can also have it assign probabilities (either in words or numbers) using more complex output structures, but I would not rely on them much more than the token logprobs (they can still be quite good though).

Another trick that works is to repeat the question two times: "I'm repeating the task and labels for clarity: ..."

Especially ridiculous is how the hacker news crowd seems to be taking these at face value…

There was one the other day with a compelling demo. But when you looked closely at it, it was feeding in the options with the word “best” on the option to pick and a fine tuned model designed to recognise that word…

So you might ask: how do I obtain the training examples? Just collect samples and use a coding agent to classify them as match or no match. From time to time you can add more examples to the dataset to have your concept adapt to changes in input distribution. It's all automated, but it only uses LLMs to train concept vectors, after that it works like a regular embedding model with a calibrated classifier on top. It's also 20-30x faster than Jev, free, and runs on CPU.

An illustration of how it defines a concept as opposed to simple cosine similarity: https://github.com/horiacristescu/semlabel/raw/main/images/c...

P.S. I am evaluating that model for a production use case where I would have used Jev

But then at the end it says it’s parody. Maybe HN title should say it’s a joke.

But aren’t calibrated predictions one of the defining features?

Still good. In practice for Jev the devils in the details. As you all know by now, it's easy to write PoC and understand with AIs (or even manually, which is now a prestious practice).

That demo will get you 80% there

Getting to that 100% or even 99% to JEV level will be hard with all the edge cases, infra, API, communications, etc.

Still a good article.

here's 7 lines

  import os
  import dspy

  lm = dspy.LM("openrouter/z-ai/glm-5.3-flash", api_key=os.environ["OPENROUTER_API_KEY"])
  jev = dspy.Predict('email:str -> choice:Literal["Legitimate", "Spam", "Phishing"]')
  email = "Payroll asks for your password on a non-company sign-in page."
  pred = jev(email=email, lm=lm)
  print(pred.choice)

  1. draw a circle
  2. import the rest of the owl

Ends with referring to a product, and saying "this is a parody post", after pretending to make a serious point.

In real life, a human doesn't do classification tasks with the System One part of their brain, they use System Two. So by definition what Jev does isn't System One thinking.

If anything, regular programming that automatically executes based on logic, without requiring "thinking" would be "System One".

If you're comparing with something, you need to state 'fast' in relative terms. Jev is definitely fast, and if this Python takes the same time to get a decision then it's also fast. If it's 100* slower than Jev though, you shouldn't be calling it 'fast', because relatively speaking it's really, really slow.

Here is an archive: https://web.archive.org/web/20260923122959/https://www.nobod...

I guess it's due to the calibrated decision part (and that's what LLMs tell me).

But I figure some supervised classification post training would still improve the model.

p(y = next thinking+decision token | x = question) != p(y = next decision token | x = question)

The former is what LLMs are trained for, the latter is what Jev was likely trained on (likely used thinking alignment as an auxiliary loss, but not explicitly included in the probability calibration).

just found this one https://huggingface.co/spaces/multimodalart/jev-decision-ind...

is a legal cya a la "Nathan For You" 's Dumb Starbucks

About Jev:

> We didn't train a model with Reinforcement Learning for Calibrated Decisions (RLCD) to calibrate the decisions and probabilities (even though they are not always correct).

Only 99% correctness! Borderline unusable!

About their model:

> It classifies: it gets a prompt with choices and outputs probabilities.

You want numbers, it gives you numbers! What more could you want?

Since most chat models want to answer with a human-readable message i think their logprobs are not as meaningful. It would be interesting to see if one choice is like "correct" and if the model wants to choose it more often, cause it might not answer the question but to prose to the user.

    <think>\n\n</think>

You can test Jev like model at 26B parameter count here (built few weeks ago): https://gambler-relay-us-west1.leo-fish.ts.net/demo (might not stay up for long)

Typesafe compatible API

This is just running on old hardware.

Speed and cost are obvious reasons, but isn’t this a tradeoff?

Why do I have to feed my e-mail into the model?

In my experience even structured LLM output performs poorly on classifier tasks. LLMs are trained to talk and think longer. If you don't give LLM enough space to reason it would become very dumb.

I'm not saying that Jev is way better, but that people way overindexed cost and speed.

ok

You have built something like jev but not jev (for starters, the output of what you've built will be absolutely worthless, the whole reason Jev is getting so much hype is because the output is good enough)

Add visual understanding, add reasoning and bring down the size to run on my computer. That's when it will be interesting.

So many people that don't understand the tech jumped on the hype train because "it cannot hallucinate" and else. It's crazy.