It's worth realizing that, before computerized central offices, telephone wiretapping required running physical wires. Back when Rudi Giuliani was prosecuting organized time, not only did physical wires have to be run, the cops were billed for them as expensive private lines. His task force was spending about a million dollars a year with New York Telephone on wiretapping. In one case, law enforcement didn't pay their bill, resulting in the person being wiretapped having the wiretap connection show up on their bill, blowing the case.

That resulted in the Communications Assistance to Law Enforcement Act, which mandated that central offices offer remote wiretapping. Capacity up to 1% of lines is required.

Back in the electromechanical era, the only call data that could be collected was outgoing dial pulses, using a "pen register".[1] (The one shown in Wikipedia is mine. It's a beautiful piece of antique brass telegraph technology. It records dial pulses as dashes, and has to be wound up like a clock, with a big brass key.) The Supreme Court decision allowing "pen registers" without a warrant refers to these "extremely limited" devices. That definition has been stretched and stretched by law enforcement into all non-voice data collected by telcos.

Law enforcement still wants more.

[1] https://en.wikipedia.org/wiki/Pen_register

> In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon.

This doesn't resonate with me. I see companies adding more sloppily written features with AI. I see more bugs in the software I use, not less. While it's plausible that software is getting both buggier and more secure, I suspect those two move in the same direction not opposite.

My guess is that we're getting better at finding _existing_ security issues with AI (and thus fixing those issues), but simultaneously adding more insecure surface areas _at a faster rate_.

On one side, you have pieces like this, where seemingly there are constant fights between serious actors with large and properly distributed budgets, employing top tech and top minds; on the other - regular news of the hackz, where responsible person in charge of security with root access failed to grasp basic technical knowledge (several times), ticking every checkbox in "never do this" list from security best practices, which led to every customer being pwned.

It's like two parallel worlds, that exist in the same place at the same time, but somehow don't cross.

I don't think the thesis that a government will be able to do something will ultimately hold. I don't see how they can avoid "going dark" in a democracy.

we live in a world where the government can't even do much about illegal drug markets anyone can access by downloading a piece of software.

if they pass laws that mandate backdoor access and block software which doesn't conform more and more people will move to the dark networks.

and if they effectively block the dark networks (in the limit they will have to block all encrypted communications) then we will be living in a tyranny.

freedom is messy. accept that digital crime can only be solved when the criminal makes a tangible mistake. LLM's will be building profiles on criminals to help with identifying mistakes.

I've always loved the ridiculousness of the "going dark" label when law enforcement can't access encrypted chats or a back door isn't built into a piece of software. When there are security cameras on the vast majority of houses, stop lights and in people's hands, and when so much meta data about people's associations are shared from Google, Facebook, any other social platform, how in the world can they say they are "going dark". How did they ever solve crimes before these things?

I'm supposed to be concerned that the US government and Israel won't be able to hack everyone's phones?

It breaks my heart that the governments with unlimited budgets who have hired the best and brightest will have to put in serious effort to get the bad guys, and potentially find it not worth it to casually spy on the whole world.

I am just beside myself at such an idea that people looking to feed the prison machine cannot as easily find excuses to turn normal citizens into prison feed.

Just super sad guys.

> In fact, the worst part about this dynamic is that these potential new backdoors will begin primarily useful for allowing the US to weaken its own systems, which will in turn allow foreign adversaries to find new ways to attack our communications. This deliberate self-sabotage will happen just at a moment when we’re finally learning how to defend our own infrastructure.

I don't understand how you can both argue for that law enforcement (and intelligence) agencies will force others to implement intentional backdoors AND also everyone will be using AI to find and secure ALL potential holes in the software so there won't be any vulnerabilities anymore.

Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention?

I have respect for the author so I feel like I probably misunderstand something from the overall text rather than I somehow have a better perspective on this topic that the author knows very much more about than me. I felt like I nodded along all up until "So how is this a problem?" and now I'm not sure I understood correctly.

I dunno man, if there's a deluge of new AI generated code at all layers of the stack I think there will still be vulnerabilities.

Like if we were willing to stop adding new code and just have a small secure code base, AI could maybe help us find all the vulnerabilities in that code base.

But people have consistently been unwilling to do that. Like if we were willing to stop adding code we could have stopped decades ago and done SQLite level testing everywhere and probably have found almost all the bugs already.

> This is not a call to action for experts to rally behind a sophisticated plan. Like so many things about the AI revolution, it’s just occurring to me that we’re on a long greasy slide to a place that will look different than where we are today.

heh. long greasy slide. It really does feel like that.

Sounds like a pretty strong argument to self host, and otherwise be in charge of the software you use.

Considering from a point of view stipulating that perfectly secure software is possible. I don't think it follows that the limititation that it would place on intelligence is necessarily a net loss.

Apart from the obvious harms of invasion of privacy, and fishing expeditions being biased to the places you decided to fish. There is the simple fact that data can be misleading, especially without context. An interceped communication is a piece of data that is intrinsically tied to the trust of the inteceptor. A few people with an agenda can collaborate to create a seeming truth by 'discovering' the same thing from different sources.

Requiring warrants compelling information holders to provide data, not only serves the task of protection from abuse but also create a record of provenance that can be verified.

It also provides a degree of symmetry in capabilities which discourages actions that one party may do over another if they are motivated to act because they have a temporary advantage over another.

Without vulnerability derived backdoors I am convinced the government will strong arm the corporations to build a backdoor for the three letter agencies—at least in the US. But we also are at the dawn of quantum systems which could make interception and spying impossible or at least made obvious to the user when it happens.

Exhausting infrastructure vulnerabilities even without quantum could be a game changer for many technologies and enable things we can’t do right now, like vote on our phones.

> In this case, we’re just going to have to hope that this time we make the right choices, for no other reason than that they’re right.

I'm more curious what could be a right choice, and more importantly who is the "we" in this, as many decisions are largely made by companies and governments.

No system view. The law agencies can develop exploits to intercept our phones, that is a new, and totally unseen before threat.

Unless you remember 2013, Snowden, that nothing was done (at most was some concern about doing it to US citizens, the rest of the world doesn't deserve privacy), all US (and/or five-eyes) based web companies must disclose users information and be forced to not disclose that, and things kept going surely at a faster and more intrusive rate in everything else, and of course phones.

You are complaining being sprinkled by water while at the bottom of the ocean. At least the big companies can find their own vulnerabilities with the AI tools you mention, the rest of the doors are still wide open.

Man I thought from the title this was going to be about next-gen AI being able to zero day everything so effectively that software security is meaningless and we'd need to basically shut it all down, go dark.

I think that one of the reasons they (frontier companies and the gov) will be putting so much effort into curtailing bugs and vulnerabilities is to limit the blast radius of future AI models. Imagine with the new Sol Ultrafast, they could have pwned Hugging Face in 6 hours and not 4 days (IIRC).

It also seems likely to me that the US Gov. probably already has routine mechanisms for compelling targeted software updates for persons of interest, so I'm not sure that a more formalized backdoor than automatic updates is going to be surfaced in the mainstream, unless that is avenue is also cut down somehow.

This "going dark" scenario would require new legislation. With secure enclaves, modern smartphones can't be cracked open in the manner that the FBI wanted in the 2016 case. So there's no such thing as "court order tech company to crack phone" anymore. It would have to be "outlaw tech companies from producing phones that they can't crack open", which is very different and does not fall under any existing US statute.

We will know we've made systems secure when laws focus on compelling people to provide access.

These laws exist - they aren't the focus yet. Right now there's still no need; just hack the device or compel the cloud service to give the data, why waste energy getting consent from its owner!

More bugfinding AI, more end to end encryption, more CVEs and more fixes, cannot happen soon enough.

I predicted a long time ago that if computers become unhackable LEA and intelligence agencies will push for laws that require backdoors to be built into hard- and software.

It will be interesting to see if my prophecy becomes reality.

BTW I also hate that Hacker News is being dominated by articles on A.I. lately. Maybe we should vote on HN reducing or even eliminating A.I. related news?

I would like to point out that the last year when not a single law enforcement agency anywhere in the world could have possibly tapped anyone's phone was 1876.

150 years ago was the invention of the telephone, and I think that articles like this seem to assume that prior to this, police just never caught any criminals.

Governments and their agencies shouldn't have any access to the communications of private citizens. If something like terrorism is the issue, the correct response is to use the wealth and resources of the government to address the root causes of those attacks, not to undermine my right to privacy.

I think what’s unintentionally eye-opening about this chart is the recency of “law enforcement can read your text communications.”

Law enforcement doesn’t need this surveillance ability at all. All time periods prior to 25 years ago didn’t have it.

Additionally, there is no correlation between “law enforcement reads text messages” and crime rates going down.

> In April, Anthropic announced a new model called Mythos that was optimized for software vulnerability finding

No, it was just good at it because it wasn't RL'd against it. I know this is a small detail, but it tosses journalistic credibility in my eyes.

Some people point out AI can cause bugs as well as fix them, and its a valid point. But the question is: what will the ratio be?

If automated pentesting in PR review CI pipeline will become table stakes - which is very plausible - maybe the OP has a point.

I completely disaggree with this take. Modern AI is not yet capable of finding multi-component bugs as advanced as those produced by firms like NSO.

Nononono no more of this shit

I remember walking into some shitty congresscritter's office with a fucking years old one pager, with a few more citations written on the bottom in pen because I wasn't going to bother making it pretty this time around.

You should have seen his face when i asked him straight up: dude, you seem to have a problem processing information. Are you having some kind of medical issue? Because I'm not the last staffer: If you abuse my time, I am never coming back here again to add more citations to a fucking one pager from 1999 -- I'm making it my mission to remove you if you fuck this up on purpose ever again.

(Or something to that effect -- I've been told I can get a bit aggressive in my rhetoric.)

This was approximately 2016 and that individual is no longer in office.

I stand by my words.

i wonder how many open source projects have alphabet boys building trust as contributors for eventual backdoor planting

But Matthew, think of the kids!!1

Honestly though, framing this as a "tech issue" doesn't help IMHO, it just muddies the water. Ever since RSA was invented privacy has been about educating people on how to use it effectively and _why they should care_. If voters now are choosing authoritarianism over democracy and individual freedom, I think we have to face the reality that after almost 50 years of fighting battle after battle on the technology front, we've largely lost the war on the home front in this regard.

So I've read dozens of complaints that AI produces insecure code.

I, for one, usually tell my AI to start with secure code, make it small, and modular.

This is the first article I've seen that now says the opposite ! AI will make code too secure!

Since the small amount of AI coding I've done often results in buggy code (even a shell script written today) with the AI go-to solution of "write more buggy code to fix", this seems counterintuitive.

Childish, nation-states as powerful as the US have access to much more potent stuff. Maybe they'll be forced to rely more on their Intel ME/AMD PSP/modem (cf https://redmine.replicant.us/projects/replicant/wiki/ModemIs...) backdoor and ANT James Bond catalog.

> Defenders are now in the process of patching every bug they can find, often with AI helping them. Entire development toolchains are being rebuilt to incorporate powerful vulnerability scanning before software reaches the testing phase. This does not mean that every bug will be found: even calculating the number of bugs in a piece of code is probably uncomputable. In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon.

> Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs.

His conclusion sounds extremely optimistic to me.

article read to me like a typical rehash of a typical offense-defense cycle. AI would have written a better article.

I personally welcome such spiraling offense-defense cycles as it is one of the main drivers of the technological progress.

Considering the amount of AI slop being generated today and LOCs that no one actually reviewed, I doubt we will run out of vulnerabilities to exploit...

[flagged]

[flagged]

[flagged]

[flagged]

[dead]

> I’m concerned that AI is going to make software much too secure.

Lmao this is like “I’m concerned the raccoons that I see in the storm drains are going to make our sewer system much too efficient”