> Our investigation has revealed that the incident originated from a third-party AI tool whose Google Workspace OAuth app was the subject of a broader compromise, potentially affecting hundreds of its users across many organizations.
> We are publishing the following IOC to support the wider community in the investigation and vetting of potential malicious activity in their environments. We recommend that Google Workspace Administrators and Google Account owners check for usage of this app immediately.
Claude Code defaulting to a certain set of recommended providers[0] and frameworks is making the web more homogenous and that lack of diversity is increasing the blast radius of incidents
I've been part of a response team on a security incident and I really feel for them. However, this initial communication is terrible.
Something happened, we won't say what, but it was severe enough to notify law enforcement. What floors me is the only actionable advice is to "review environment variables". What should a customer even do with that advice? Make sure the variable are still there? How would you know if any of them were exposed or leaked?
The advice should be to IMMEDIATELY rotate all passwords, access tokens, and any sensitive information shared with Vercel. And then begin to audit access logs, customer data, etc, for unusual activity.
The only reason to dramatically overpay for the hosting resources they provide is because you expect them to expertly manage security and stability.
I know there is a huge fog of uncertainly in the early stages of an incident, but it spooks me how intentionally vague they seem to be here about what happened and who has been impacted.
_jabyesterday at 6:40 PM
> Vercel did not specify which of its systems were compromised
I’m no security engineer, but this is flatly unacceptable, right? This feels like Vercel is covering its own ass in favor of helping its customers understand the impact of this incident.
nettlinyesterday at 7:18 PM
They just added more details:
> Indicators of compromise (IOCs)
> Our investigation has revealed that the incident originated from a third-party AI tool whose Google Workspace OAuth app was the subject of a broader compromise, potentially affecting hundreds of its users across many organizations.
> We are publishing the following IOC to support the wider community in the investigation and vetting of potential malicious activity in their environments. We recommend that Google Workspace Administrators and Google Account owners check for usage of this app immediately.
> Vercel has reportedly been breached by ShinyHunters.
nike-17yesterday at 4:47 PM
Incidents like this are a good reminder of how concentrated our single points of failure have become in the modern web ecosystem. I appreciate the transparency in their disclosure so far, but it definitely makes you re-evaluate the risk profile of leaning entirely on fully managed PaaS solutions.
Izmakiyesterday at 7:16 PM
A "limited subset of customers" could be 99% of them and the phrase would still be technically true.
swingboyyesterday at 4:41 PM
Is this one of those situations where _a lot_ of customers are affected and the “subset” are just the bigger ones they can’t afford to lose?
OsrsNeedsf2Pyesterday at 2:39 PM
The lack of details makes me wonder how large this "subset" of users really is
landl0rdyesterday at 6:43 PM
Wow, maybe Cloudflare can help them secure their systems? I hear they have a pretty good WAF.
jtokophyesterday at 4:55 PM
This announcement in its current form is quite useless and not actionable. As least people won’t be able to say “why didn’t you say something sooner?” They said _something_
zuzululuyesterday at 5:07 PM
What is the rationale for using vercel ? I'm getting a lot of value out of cloudflare with the $5/month plan lately but my bare metal box with triple digit ram has seen zero downtime since 2015.
jngiam1yesterday at 10:33 PM
I don't get why everything is not marked as sensitive in env vars by default instead.
So, the Vercel post says a number of customers were impacted, but not everyone, and they will contact the people that were impacted.
I wasn't contacted so does that mean I'm safe?
adithyasrinyesterday at 3:44 PM
We run on Vercel and I wonder if / how long before we're alerted about a leak. Quick look online suggests environment variables marked as sensitive are ok, but to which extent I wonder.
rrmdpyesterday at 9:00 PM
Use VPS, nowadays with the help of AI it's a lot easier to set everything up, you don't need Versel at all. And of course way cheaper
leetroutyesterday at 6:31 PM
Porter also had a breach recently. I assume it is as tightly scoped as they say to not have publicized it.
monirmamounyesterday at 8:06 PM
Well when the CEO of Vercel poses with Netanyahu, a war criminal, in the middle of a genocide... it's going to imply that Vercel has valuable war crime data that people will want to intercept just to bring down Israel's genocidal program.
philip1209yesterday at 7:39 PM
We proactively rotated keys. Even if you haven’t received an official email, expect customers to inquire about this tomorrow morning.
oxag3nyesterday at 7:17 PM
> incident response provider
So they use third-party for incident management? They are de-risking by spending more, which is a loose-loose for the customers.
james-clefyesterday at 8:23 PM
The point I am taking away here is to never use Vercel's environment variables to store secrets.
He also suggests in another post that Linear and GitHub could also be pwned?
Either way, hugops to all the SRE/DevOps out there, seems like it's going to be a busy Sunday for many.
_pukyesterday at 5:09 PM
Hmmm, the dashboard 404 I got 6 hours ago now makes a bit more sense..
jheitzebyesterday at 5:23 PM
Missing from Glasswing
jamesfisheryesterday at 7:58 PM
Reminder the Vercel CEO is a genocide supporter, if you need more reasons to move away from it.
nothinkjustaiyesterday at 5:11 PM
Looks like their rampant vibe coding is starting to catch up to them. Expect to see many pre vulns like this in the future.
raw_anon_1111yesterday at 8:05 PM
Why does anyone running a third party tool have access to all of their clients’ accounts? I can’t imagine something this stupid happening with a real service provider.
I see Vercel is hosted on AWS? Are they hosting every one on a single AWS account with no tenant isolating? Something this dumb could never happen on a real AWS account. Yes I know the internal controls that AWS has (former employee).
Anyone who is hosting a real business on Vercel should have known better.
I have used v0 to build a few admin sites. But I downloaded the artifacts, put in a Docker container and hosted everything in Lambda myself where I controlled the tenant isolation via separate AWS accounts, secrets in Secret Manager and tightly scoped IAM roles, etc.
jimmydoeyesterday at 6:16 PM
what's the cause of the breach?
deletedyesterday at 2:38 PM
0xyyesterday at 3:10 PM
This is why you pay a real provider for serious business needs, not an AWS reseller. Next.js is a fundamentally insecure framework, as server components are an anti-pattern full of magic leading to stuff like the below. Given their standards for framework security, it's not hard to believe their business' control plane is just as insecure (and probably built using the same insecure framework).
Next.js is the new PHP, but worse, since unlike PHP you don't really know what's server side and what's client side anymore. It's all just commingled and handled magically.
There is no serious reason to use Vercel, other than for those being locked into the NextJs ecosystem and demo projects.
victor9000yesterday at 10:22 PM
[dead]
renan_warmlingyesterday at 7:49 PM
[dead]
nryooyesterday at 4:42 PM
[dead]
Yash16yesterday at 9:39 PM
[dead]
jccx70yesterday at 2:53 PM
[dead]
ArcherLyesterday at 5:37 PM
[dead]
sreekanth850yesterday at 4:05 PM
[flagged]
ksajadiyesterday at 5:37 PM
[flagged]
hansmayeryesterday at 8:13 PM
[flagged]
deletedyesterday at 4:41 PM
mikert89yesterday at 3:18 PM
Much as I want to rip on vercel, its clear that ai is going to lead to mass security breaches. The attack surface is so large, and ai agents are working around the clock. This is a new normal. Open source software is going to change, companies wont be running random repos off github anymore