90% of US working hours are spent on tasks not covered well by AI benchmarks
305 points - today at 5:08 PM
SourceComments
> ...
> But even after controlling for AI-exposure, O*NET tasks from occupations with larger wage bills are significantly overrepresented among benchmark prompts:
So, in short: they want AI to take the good jobs. AI is not coming for jobs with tasks like "gather and empty trash" or "service, clean, or supply restrooms," or "mix water and detergents or acids in containers to prepare cleaning solutions according to specifications." We software engineers and lawyers would be smart to start retraining (and adjusting our living standards) now!
And this points to one of the ways AI will help improve society and living standards for all: we'll have cleaner, better-maintained bathrooms, as labor one wasted on engaging intellectual tasks is redirected towards menial, low-pay tasks that still need a human touch.
Reason to be optimistic, amirite?
For example, the “social workers & care -> counsel parents with child rearing problems” category is considered “well covered” because there are 13 prompts.
It’s self-evident that the spectrum of work carried out by a social worker undertaking that work cannot be well represented by thirteen prompts.
I suspect if you really strictly defined “covered well”, to the extent that any normal person would use that phrase, the headline would be above 99.9%.
I've found discussions on HN tend to quickly get pulled toward polar extremes between those who are justifiably impressed at the increasing effectiveness of LLMs and those who highlight the brittle jaggedness and counter-intuitive gaps of LLM 'intelligence' in daily practice. One side keeps citing more examples of 'uniquely human traits' LLMs don't seem close to replicating while the other complains of eternally moving goalposts.
Today I think both extremes are partly right but also partly wrong, and end up largely talking past each other because so much remains ill-defined, uncertain or currently unknowable. IMHO, the most accurate position is a carefully nuanced middle-ground built on well-defined scopes, terms, time frames and probabilities with frustratingly wide error bars. Partisans at either extreme seem uninterested in more than drive-by hot takes.
It can be simultaneously true that "Someday AI will be able to functionally replace all human knowledge work" and "That day does not appear to be soon, and even when it arrives, there will still be uniquely human traits which are valuable and that LLMs can't satisfactorily replicate."
I've started to notice this on a micro scale as well. I am 20x more productive at coding and scripting, but I don't spend more of my time coding. My time is now filled with things like presentations, Jira, QA, etc. Ironically, despite all of the automation, the things that are painstaking and time consuming disproportionately require more time now.
It's really easy to fall into the McNamara fallacy when rolling out these kinds of solutions. Focusing on the low value stuff is attractive. It tends to be easier to do, and also easier to quantify outcomes. But it also tends to direct attention away from important but difficult to quantify considerations, which may mean you don't notice negative externalities. So we often found ourselves in a verschilmmbessern situation where automating what we could automate made things worse overall.
We have 3 companies claiming something like a $35T market for their services. The world's entire GDP is something like $126T. So 10% of the productive work that is done on this planet, that AI can actually perform makes up fully 1/4 of the entire world's economic output? best guesses for output solely of tech companies is something like $12T. Where are they getting these figures?
I understand the resistance to having nearly all labor devalued because it can be automated. We're still in the denial stage of grief.
We only tax capital gains, not the destruction of wealth or property. So someone can steal the livelihoods of millions of people with no recourse. Just like they can tear down a 100 year old house and build 3 townhouses in its place and make hundreds of thousands of dollars, despite driving up housing costs in the neighborhood to the point that they're unaffordable. Large swaths of profit in the economy are based on externalities and captured markets now.
To think we'll make it through this without tax reform is frankly naive. Tax reform as in: human labor should not be taxed, and non-human labor and capital gains should be taxed more to cover the difference - including unemployment checks and UBI when it comes to it.
Expect howls of derision from moneyed interests who will never let common sense reasoning or deduction enter the law books for the public good.
Not to defeat my own argument, but I predict we will experience a roughly 3x gain in productivity by using AI, so the output of every job would need to triple to maintain current employment levels. Will that happen anytime soon? Of course not. Which is why layoffs in the short term are inevitable. I wouldn't be surprised if unemployment hits 15+% in 2027 or 2029 just after the elections, with 25+% unemployment for the young in entry-level roles being highly likely.
All this being said, I don't think AI usage will meaningfully go down in the long run cause it's a very powerful dopamine loop and tech companies are very good at making those. How many people report social media making them unhappy and how many people do you know who have actually quit? Or to paraphrase Jim Gaffigan, nobody says they eat at McDonalds, but the sign outside says "billions served". Somebody's lying (and it's not McDonalds).
Apparently Epic thinks otherwise.
I guess if I wanted to put my skepticism about the AI stock bubble into one little phrase, it would be that I don't think the bubble has accounted for Amdahl's Law. Costs don't actually go to zero with AI, and there's a huge difference between costs falling even 90% and costs falling a full 100%. And the difference between "30% improvement this year, trending to 85% in five years" and "90% this year" is pretty significant too.
Yep, makes sense...
You know what? The truth doesn't matter much. If people believe in the resale value of tulips, each tulip would sell for a million.
It's the beliefs and perception that create the reality. Reality and truth don't exist on their own.
hint: it's neither