As AI eats the web, the internet’s collective memory is disappearing

330 points - yesterday at 10:36 PM

Source

Comments

ghm2199 today at 10:52 AM
My sister, a journalist, mentioned to me that she only uses google search because she had learned how to get information typically only Google indexed in the country she lives in, in a way it was not exposed on chat bots. She often has to search for information like Old govt forms released as public record with a fixed a certain format photo scanned into a pdf and indexed by Google were often on the second page of the search and beyond. But they are there. She knew how the forms looked and what bigrans and trigrams matching a certain part of form for a certain piece of information to search for and Google search has it. Like an official order on a tender notice for some government department which is no longer in the .gov.* website gave her the official's name and then she could track down who to contact in an office... ChatGPT and other bots don't have it. Some how all these government documents became part of the government record and are the key for her to do her job.

I sincerely hope google wont stop indexing that stuff just because of a PM in search "de/re-prioritizing" ranking in a way that makes this impossible.

figassis today at 10:31 AM
I called this maybe 3y ago, but I think so did everyone else that was sane. Sure, we get immense value from AI, but indiscriminately injecting into everything, the one thing we know to be unreliable above the threshold we used to fire people for, is probably the greatest undoing of all the good companies like Google brought to the internet. I mean what a way to destroy your legacy of democratizing information. The amount of harm (direct and indirect) this will cause, and the cost to return to baseline will be so immense, and yet we will not be able to point to the root cause. They won't be there to take responsibility.
erfgh today at 10:40 AM
AI has killed reading-anything-written-after-AI for me. Due to this effect it is probably the worst invention in human history or pre-history.
ChiMan today at 8:11 AM
AI will kill the internet because it is killing the incentive to make it. It is an industrial-strength example of why we don’t allow stealing.
sgt today at 7:27 AM
Funny, I was just thinking this morning that Google searches are absolutely horrible these days. It's like it has amnesia, a lot of recent history seems to be just gone. Especially on non US specific sites too.
ilamont today at 1:06 AM
After publishers successfully sued the Internet Archive over its digital lending program, calling it unauthorized copying

No. The court specifically determined that the Internet Archive was guilty of unauthorized copying. It was not simply an unfounded or unproven allegation. The Authors Guild, the National Writers Union, the European Writers Council, and the Society of Authors in the UK all came out against the Internet Archive, and supported the suit.

Each new restriction limits the archive’s ability to act as a comprehensive backstop.

This self-inflicted damage to the wayback machine is the real tragedy of this entire affair. When IA was asked to stop CDL - many times - founder Brewster Kahle continued. The National Writers Union tried to open a dialogue as early as 2010 but was ignored:

The Internet Archive says it would rather talk with writers individually than talk to the NWU or other writers’ organizations. But requests by NWU members to talk to or meet with the Internet Archive have been ignored or rebuffed.

https://nwu.org/nwu-denounces-cdl/

When the requests to abandon CDL turned into demands, Kahle dug in his heels. When the inevitable lawsuits followed, and IA lost, he insisted that he was still in the right and plowed ahead with appeals. And here we are today.

comrade1234 yesterday at 11:22 PM
I spent the last three days (off and on) using Gemini to configure my edge router 4 with my iOS devices on a vpn and it's been awesome. In the past I'd do a google search and read a few sources of documentation, do another google search and read another set of documentation. Now, Gemini aggregates multiple pages together so all of the work of reading source docs from multiple locations is now n a single step.

Oh, I should mention though. There was no advertising at all. They didn't make any money off me. It was 100% Gemini which I recognize as not long-term feasible.

bsammon today at 7:17 AM
> While the web has always been organized around intermediaries that shape what survives online and who sees it,

This statement, from the sixth paragraph of the article, is something that I would have liked to see addressed more in the article. The article implies that this is something that must always be true, or cannot be changed, and simply focuses on how we could have better/better funded/better protected intermediaries (AKA gatekeepers), and doesn't discuss the possibility of an internet (or part of the internet) without gatekeepers (and doesn't ask if it has ever existed/does exist/should exist)

novafunc yesterday at 11:19 PM
I occasionally use Google Search when DuckDuckGo fails to give me relevant. Almost always, Google has better results.

Though I can find its AI answers annoying aggressive. I'll look up like two search terms and the AI will bullshit multiple paragraphs out of despite having zero context of what I am looking for.

DuckDuckGo seems to have detection of whether it should give an AI answer. And it allows you to have more granular control of when you want to get an AI answer. And is overall less distracting than Google's.

CommieBobDole today at 1:07 AM
The article touches on something that I've been thinking about with regards to Google's AI strategy; the automatically-generated AI search summaries are not great. They very frequently confidently misinterpret what the user is searching for and generate half a page of useless information that pushes actual results down the page, and they are occasionally hilariously incorrect, with hallucinated facts.

This is probably a difficult-to-solve problem; given that they generate billions of these a day, not even Google can afford to devote enough compute to each query to reliably generate quality results. You can see this by selecting the "AI mode" from the search interface after getting the mediocre summary - the results are much better and generally perfectly usable. Though even that is probably a special minimal-compute version of the lowest tier of Gemini, it's still maybe an order of magnitude more capable than whatever generates the search summaries.

The bigger problem is that these search summaries are the default and by far the most common interaction that the general public has with "AI", and because this experience sucks, they just assume that all LLMs are similarly stupid and mostly useless. In non-technical spaces I frequently see the argument that "AI" is not useful for anything, all it generates is garbage hallucinations, and almost invariably they cite some actual terrible experience with the Google AI search summary. I would argue that the strategy of adding LLM summaries to every search is the worst of both worlds - it makes classic search worse while poisoning users against the idea of actual LLM-assisted search.

umvi yesterday at 11:30 PM
I feel like collecting, curating, and protecting high quality corpuses of "truth" is going to become increasingly important for high quality AI.

There will come a day (and probably soon) when "training on the public internet" (Reddit, etc) will taint your model with metric tons of corporate contamination, political poison, and other adversarial content intentionally crafted to bias AIs for various reasons (corporate gain, geopolitical information warfare, etc). Basically the AI-equivalent of SEO.

smackeyacky today at 4:07 AM
Gemini has been a hilarious companion to my while I fixed the balance shaft chain guides in my old Mitsubishi triton (mighty max for US readers).

First it told me I could just remove said balance shaft chain as an emergency repair. Sorry Gemini, it also drives the oil pump.

Then it told me I could remove the water contaminated oil caused by removing the timing case by filling the crankcase with hot, soapy water and running the engine. Lord no.

Then it gave the wrong instructions for putting new gears on the balance shafts which meant the chain guides didn’t align with the chain. I’ll do it my way thanks Gemini.

The rest of the mistakes are too trivial to recount and sure it’s a pretty obscure subject but if I trusted it with a topic I’m not familiar with there is a huge potential for damage if you blindly follow it’s overconfidence. I miss normal searching.

luciana1u today at 7:05 AM
we're building the world's largest library and then locking the doors, letting the bots photocopy everything before the lights go out.
schaefer yesterday at 11:28 PM
Kagi search today is better than Google search ever was.

And it’s clear that Google’s Ad model ultimately created a priority inversion. The advertisers became the customer.

I am so glad Kagi came along with a business model that is actually working.

keiferski today at 8:21 AM
The concept of an almanac seems relevant again: a yearly printed book with verified, accurate information. No manipulation at a later date, no AI hallucinations, etc.

The most famous one was probably Benjamin Franklin’s:

https://en.wikipedia.org/wiki/Poor_Richard%27s_Almanack

Paper encyclopedias might make a comeback for the same reason.

anigbrowl today at 12:27 AM
Overbroad claim. Dramatic corollary

This clickbaity headline format cannot die fast enough

kukkeliskuu today at 9:00 AM
The world needs an anti-SEO search engine, that explicitly penalizes for something that appears SEO.
lopis today at 8:42 AM
My personal anecdote is that I used to search for "xyz nutrition" quite often on Google. It used to provide a data table with lots of information. Sure, nutritional information is hard to get right, but at least that data was consistent.

Now Google just gives you an AI answer with random values pulled from blogs and Reddit. It's almost always blatantly incorrect. I genuinely can't understand why Google would destroy its most valuable search features. Disclaimer: I work for Ecosia, so I know for a fact that users really value these search widgets, and it was often cited as a reason they couldn't leave Google.

ElProlactin yesterday at 11:33 PM
There's too much going on in this article and while I think some of the points are valid, others go too far.

A lot of the "cultural record" the author refers to is just digital junk. Random digital content that very few people care about, if we're being honest. Trying to hoard every bit of digital information ever produced is not the same thing as preserving "culture".

Case in point:

> Even the increasing use of ephemeral formats like Instagram Stories and WhatsApp status updates means that large portions of cultural, social, and political communication are never conserved in the first place. As a society, we can probably survive bad search results and come up with another way to schedule a sunset make-out session. But we can’t aspire to sovereignty if we can’t retain and retrieve our collective memory.

For most of human history, nobody was trying to "conserve" every cultural, social or political communication ever produced, and I fail to see how Instagram Stories and WhatsApp status updates, many of which aren't even truly broadcast publicly for all to see, are part of some imaginary "collective memory."

If you find a web page, see an Instagram Story or receive a message that's important to you, save it or take a screenshot. But let's not pretend all these things belong in a global Digital Civilizational Archives.

p0w3n3d today at 8:10 AM
Hello negative feedback loop.
danpalmer yesterday at 10:57 PM
It's hard to get past the beginning of this article and take it at all seriously. The quote someone who missed a sunset because they asked google and supposedly got the wrong time... but they didn't think to look at the sun or lack thereof to check? Also when I put "when does the sun set today" I get a single exact figure at the top of my results, not from AI, which is honestly the best kind of result – an exact correct answer.
khalic today at 10:01 AM
It seems like a lot of techies should have chosen acting as a specialty, because I’ve never seen that much melodrama in one industry. FFS people, adapt, stop complaining
gaigalas today at 9:03 AM
Perhaps we should let the content economy crash so that the big monsters eating it starve and die. A new content economy could be built on their carcasses instead.
noosphr today at 12:13 AM
Gemini's highest tier of plan has the utility of Google search circa 2010. I'm paying $400 for the privilege.
JohnMakin today at 6:55 AM
we built extremely powerful plausible bullshit machines targeted toward and trained on an electorate and population that has historically bad education and reading levels, built on top of an already fraught and fragile web which was also built off predatory basically unregulated behavior with a shaky relationship with “truth” and are surprised people have no idea what’s going on?

This was the whole point of it all and why the people in power have bet the farm on it.

larodi today at 8:35 AM
What a joke. Google did that already 10 year ago when they started evicting stuff from its page rank caches.

AI is merely amplifying what social media and FAANG in general already have done to the 90s web.

flinux yesterday at 11:58 PM
I see only one simple solution (though we should discuss the more complex ones): if Google directly answers a search query, then it must be held accountable for it, for better or for worse, and therefore assume all the benefits and (legal) liabilities that this entails
monster_truck today at 9:31 AM
I just don't care man. Did these people just log on yesterday or something? The MUDs and MMOs I grew up with disappeared. The IRC networks, and especially forums I learned so much from are gone (these were largely killed for something even worse than AI: commercial blogs!). Effectively every social network I've ever cared about has been ruined, failed, or sold. Even games have become bottom line chasing, live service slop that you never truly own.

If you want it so bad stop crying and make it, that's what I've been doing. It works a lot better than whatever this post and many of these comments are. There are dozens of us !

cookiengineer today at 7:47 AM
I'm working on using the OpenZIM format to archive the web and to make the wikis seedable (and locally hostable for LLMs) so that the ongoing cat and mouse game anubis defense can stop.

My hope is that with the torrent protocol we can make the archived knowledge discoverable and seedable, because currently there's only the web archive and the kiwix download servers for archived contents. Both of them still are centralized servers that bear the cost of hosting those files.

- [1] https://github.com/cookiengineer/gozim

- [2] https://github.com/cookiengineer/zimdex

zkmon today at 7:39 AM
It happens when the deterministic precision was given away in return for the probabilistic guesses. It was one extreme until now (deterministic code), and we are swinging to the other extreme (probabilistic slop), but what the world wants could be somewhere in between. Some information does not need too much precision, while others do need precision.
qudat yesterday at 11:45 PM
Funny as I just cancelled my Kagi sub to get the Gemini ai pro sub. The deal was too good to pass up
amazingamazing yesterday at 11:32 PM
[2011] The Atlantic - Why Google Won't Survive the Facebook Threat.

I’ll add this article to the list of incorrect predictions lol

georgemcbay yesterday at 11:01 PM
From a user perspective, Google search is the most useful it has been in years, though that doesn't feel entirely like intentional improvement, just a lucky side effect of the move to "AI mode".

And yes, if you take what the AI tells you at face value it could be wrong. But if you are aware of this and aware of the ways in which LLMs are likely to shit the bed, it is quicker to get from request to useful information than it has been with Google search since like 2017.

And also, yes, the old balance of Google driving clicks to sites that will then generate revenue off more Google Ads being shown after you click through to them creating a virtuous cycle is completely busted, and that sucks. It does not impact me directly but it certainly seems like unless a better system is devised that it is one of a few ways in which AI is likely to stall out its own training funnel.

charcircuit yesterday at 11:24 PM
It would be beneficial for the author to look at the revenue for Google Search. Revenue is still growing.
deleted today at 8:18 AM
forgetfreeman yesterday at 11:37 PM
The most frustrating part of all of this is the underlying premise that the internet is, has been, or could ever be a credible cultural record is deeply stupid. Or maybe more charitably it's both historically and technically illiterate. It has always taken continuous unwavering effort on some person's part to keep any given piece of content online. And while managing a simple hosting account and updating domain registration periodically doesn't take a tremendous amount of effort 20 years is a long time to expect anyone to maintain enthusiasm. The internet has always been a frothy, ever changing blend of the odd nugget of truth drifting in a sea of unadulterated bullshit. Treating this, or worse what comes from statistically averaging it, as a source of capital T truth is totally unhinged. From whence did this mythology of online truth spring?
peyton today at 12:04 AM
What’s come next for me has been much better. I use ChatGPT cranked to Pro with “extended” thinking to one-shot whatever I would’ve spent time looking into with Google. It’ll plan the whole sunset bike ride or promposal or whatever from TFA.
BrucecarlL today at 9:54 AM
[dead]
glum64 today at 7:13 AM
[dead]
madhu_ghalame today at 7:16 AM
[dead]
jheriko today at 7:16 AM
[dead]