It's 5:20 AM here in Europe and I just rotated a bunch of keys.
Only to read,
> Because deployments are paused, you will not be able to redeploy your services to pick up the new credentials until builds are restored. We will post here when builds resume with any additional details.
So now my service does not work. Which is worse: a stolen key or a broken product? I'm genuinely regretting following their advice on rotating keys, because logs showed no sign of incorrect usage, and our service would at least have remained online.
I hope that they automatically redeploy their customer services. I just worry that I won't have tested it. I mean: 5AM. Keys changed. Nothing more I can do. Should go back to sleep. No timeline to wake up to check deployments, so... I... wait?
vldsznyesterday at 11:11 PM
Crazy day for cybersecurity: PHP RCE, eMudhra certificates, and now Railway.
Yup, got my OpenRouter key. They managed to run up a whole $2 on Opus 5.5 and Fable. Well, time for a new hosting provider I guess...
Incidentally, I'm testing Muse as an email triage agent today - it buried "oh also, railway and anthropic say yours keys have been leaked and you need to rotate asap" under 50 routine deploy / admin emails notifications. Oh to live through such an AI powered software renaissance!
theturtletalksyesterday at 10:20 PM
Postiz founder posted on X that his service is being brought down, his keys were already used and Railway was quiet about this until he spoke up:
Right just as I take my first day off (for the entire year!) from work to fly out for a vacation. The mother of all security bombs starts dropping all around stacks and services that I am responsible for. How else could have taken time off go for me…
MattIPv4yesterday at 10:13 PM
Seen multiple folks in the Railway Discord confirming that their secrets stored in Railway have been used by third parties, so it definitely sounds like the attackers were able to read decrypted environment variables.
martinaldyesterday at 10:27 PM
I have a bad gut feeling that every day is going to look a bit like today with endless urgent patches and breaches.
Fwiw it sounds similar to the hack over the weekend that opencode suffered.
alpinemanyesterday at 10:31 PM
Train wreck. People were writing on discord 3 hours ago, 10 hours after the incident started, that keys created just for railway were running up usage. Meanwhile builds have been down for more than 12 hours. Only NOW did they just send out an email to all customers. I wouldn't have found out until then if I wasn't checking the discord periodically.
averroustoday at 2:26 AM
Anyone can help?
I changed he api key in the env var as openrouter revoke the previous one based on this incident. However, as the redeploy cannot be done in my region, the env cannot be updated and now the product cannot be used. Any command in railway to fix this?
jasona123today at 2:59 AM
sighs
Thankfully I'm only using Railway for a hobby project (aptly, for a real-time transit dashboard) but still
fyzixyesterday at 10:45 PM
How is it possible rotate keys if you disable deployments? You're basically telling us to take our services offline...
DrawTRyesterday at 10:21 PM
Jeez, what a rough past 24 hours for all security folk. Hang in there
AJRFyesterday at 10:31 PM
Do cloud providers have a lock-down mode where you can press 1 button and stop all spend?
tstrebitzertoday at 3:03 AM
meanwhile on railway.com:
Ship software peacefully
deletedyesterday at 11:08 PM
minrawsyesterday at 10:25 PM
how are the secrets decrypted, I would like some explanation so that I can have some understanding here, I am starting to feel like all these companies are just meme-ing with their security posture.
Like did someone vibe code too hard, how do env secrets leak for everyone...
cowlbytoday at 1:49 AM
This is so fucked up. I was reviewing my account and I just saw this in the logs... Claude says database credentials and bucket credentials go in via env vars so likely they had access to database and bucket contents.
Audit Logs
Oct 21, 2026 05:54:13 PM CDT
BucketInstanceCredential.regenerated in my-project production by Railway Staff
Oct 21, 2026 05:54:13 PM CDT
BucketInstanceCredential.regenerated in my-project production by Railway Staff
Oct 21, 2026 05:54:13 PM CDT
BucketInstanceCredential.regenerated in my-project production by Railway Staff
1-6yesterday at 11:33 PM
It's time for competent programmers to get rehired to help patch massive exploits that are about to unfold before us.
rpmismstoday at 2:51 AM
Aw shit. I just got done patching our PHP servers, now I gotta do the Railway too? I swear I'm gonna wake up to a Vercel SNAFU.
paxyyesterday at 10:24 PM
It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!” as if it absolves you from having to maintain a secure system. If anything it makes the situation more damning. If it was possible to find exploits using AI then your own red teaming should have found them before the attacker did.
dylanztoday at 12:06 AM
I'm rotating production secrets all night... great. I confirmed that they've actually used many of my secret keys already.
arbayiyesterday at 11:27 PM
i revoked my api keys, changed with new ones but guess what? railway does not allow deployments at the moment, so my app is down.
gbnwlyesterday at 10:25 PM
The past two days have been awful as someone who uses Railway.
eecksyesterday at 10:41 PM
Huh, I've been using Railway tonight and this was the first I've seen of this. Just got an email about it now too at 23:36
someone managed to get the environment variables AND the encryption keys for those? i feel for the secops engineers over there right now. brutal
wateralienyesterday at 10:24 PM
Is this our new AI daily reality?
sergiotapiayesterday at 10:41 PM
If you're on Railway do not wait, ROTATE every API key at the source, and do not set new values because Railway is still compromised and not secured.
AndrewKemendotoday at 2:04 AM
Yeah I’m not sure what happened but there was a huge bot swarm the last few days
I had an open gitea host pulled into some botnet
Luckily digitaloceam sent an alert.
shinycodeyesterday at 11:08 PM
I’m lucky I moved away from it to host my own db with coolify
barelysapienttoday at 12:25 AM
Ugh. Love Railway.
sergiotapiayesterday at 11:27 PM
Actually I will move my projects away from Railway. With AI what it is today, no reason to shy away from hosting on actual VPS.
What is the best VPS to use today? Is it still hetzner or ovh?
lijokyesterday at 10:27 PM
Damn I guess all those l33t hiring practices were for nothing
Keloranyesterday at 10:10 PM
Well this is certainly interesting, I wonder how their secrets were being stored, I assumed they were stored in vault and then injected with a sidecar, but I guess not
essephyesterday at 10:24 PM
Fuck, I thought this was about trains
goforbgyesterday at 11:24 PM
the worst service I've signed up for has to be railway. What a terrible product.
dayyanyesterday at 11:42 PM
[flagged]
jon128today at 12:27 AM
[dead]
torben-friisyesterday at 11:19 PM
>We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers.
Is that an accusation? As in, major AI players taking out competitors ?
vischtoday at 12:42 AM
This is exactly why we should own our deployment stacks. Going to push up the launch here as pdt is a railway alternative thttps://github.com/autoidm/pdt for scheduled jobs
AI makes this so much easier to do now, we should all own our own code, and deployment infrastructure (use a cloud instead of someone wrapping aws for you)