It is true that they don't tend to give simple answers.
But it reveals something about human biases that we tend to seek out simple answers to complex questions. We tend to think that because a question is short, that it's simple. We forgot how many details are involved and how much is actually going on underneath. Reality itself is often annoyingly complex.
All living creatures, including humans, are conditioned to simplify extremely complex things into simple concepts which we can react to. We only understand things to the extent that we believe they affect us.
saejoxtoday at 10:41 PM
I have been using structured json outputs since gpt 4 days. Never understood this jev hype
dzinktoday at 9:44 PM
PostHog should not be using them on customer data, period.
aabditoday at 10:13 PM
Contrarian take.
I think everyone should use LLMs and the most expensive possible, unless they know how to run evals (both on line and off line).
Yes its expensive, ugly, etc.
But...
I've seen enough IRL that people just create really bad evals that do not reflect real world. Then the product is just bad. Outright broken for the p90 customer.
People don't know statistics. Offline Evals are hard. Sampling the entire population distribution is hard. Sampling the fat tail is harder. Online evals are big ops projects that are giant balls of death.
dieselgatetoday at 10:12 PM
Completely unrelated to the article itself but I landed on the posthog.com homepage after viewing their open positions and the landing page is cool. It's sort of a desktop UI and different from what I usually see, was expecting some other HR domain to come up.
gaal_dornicktoday at 7:51 PM
[dead]
Rapzidtoday at 10:40 PM
The approach of using decision models to gate more expensive work.. I mean, yeah.
But using protecting PII as an example use case is a bit bonkers?
jakeydustoday at 10:04 PM
"Not everything needs an LLM! Use a decision model!"
And what is a decision model exactly? An LLM with instructions to limit responses?
Huh.
deletedtoday at 7:29 PM
johnh-hntoday at 10:43 PM
[dead]
pj_mukhtoday at 10:33 PM
tl;dr Not everything needs an LLM, but everything needs a typesafe wrapper around an LLM.