Railway's $740,000 Bill Shows AI API Keys Are Now the Cash in the Register — and Your Hosting Platform Holds the Till
Railway's $740,000 Bill Shows AI API Keys Are Now the Cash in the Register — and Your Hosting Platform Holds the Till
Railway's full postmortem shows an attacker went from a build escape to the company's decryption key in about a day, with AI provider keys as the main target, and it changes how teams should think about storing secrets on a platform.
What is new
Railway first disclosed its October breach in a preliminary report on 22 October. The full findings, released this week, change the picture in three ways. First, the attack started earlier than the company first thought: at 07:28 UTC on 20 October an attacker used a previously unknown flaw to partly escape build isolation, about six and a half hours before an alert about unusual registry calls paged an engineer. Second, there is now a price. Railway estimates the stolen AI provider keys were used for about $740,000 of unauthorised usage, and says it will pay where the AI providers will not refund. Third, the chain is now fully documented, and it is a textbook example of how one small foothold turns into the keys to every tenant.
The chain, in plain terms
The full report sets out a sequence every platform team should recognise:
- Build escape. The partial escape gave the attacker a credential that build servers use to push images and reach some internal systems.
- Over-trusted internal service. Railway's internal workflow system (Temporal) accepted that build credential as a trusted client. From there the attacker read other builds' histories, including the secrets passed to 353 builds, and pulled cached code snapshots that held a copy of the internal monorepo.
- Reading the source to find the next door. The monorepo showed a misconfigured OIDC binding in a cloud environment. Exploiting it gave a short-lived cloud credential, which was then used to obtain two more: one able to read encrypted production variables, and an SSH key.
- The decryption host. The SSH key opened an orchestration machine holding a credential that could call the cloud KMS to unwrap per-environment keys. Between 04:42 and 07:15 UTC on 21 October the attacker decrypted variables. They were fully removed at 09:24.
Railway puts the exposure at no more than 1.85% of workspaces, about 100,000. That is an upper bound, because its logs show which environments were touched but not which variables were read. Linked-account tokens for 1,852 users, including GitHub OAuth tokens and bring-your-own-key agent keys, were also exposed and have been reset. Railway says it found no access to customer databases and no sign that customer repositories were downloaded.
Why the attacker went for AI keys
The detail that should change behaviour is what the attacker searched for. Railway says the database queries filtered for variable names matching AI provider keys. Some queries also matched cloud, database and payment credentials, but AI keys came first.
That makes economic sense. A cloud key leaves a trail and triggers alarms. A database password needs a network path. An AI API key is a bearer token that turns straight into resellable compute. Railway's working theory is that the keys fed services that relay or resell model access for distillation, though it has not shown that the attacker ran those services. Either way, an AI key works like cash: anyone holding it can spend it immediately and from anywhere, and the owner gets the bill.
Railway is not alone. Cloud security researchers count at least nine AI developer platforms that disclosed incidents in October, with attackers hunting stored environment variables and especially AI keys. No single flaw links them. The pattern is the target, not the technique.
The uncomfortable part: envelope encryption did its job, and it didn't matter
Railway did the textbook thing. Each environment's variables are encrypted with their own key, wrapped by a root key in a cloud KMS. That design defeats someone who steals a database dump. It does not defeat someone who reaches a machine that is allowed to ask the KMS to decrypt. Encryption at rest protects against a narrow threat. The real control is which identities can call decrypt, from where, and how quickly an unusual burst of decrypt calls gets noticed.
The refund picture also shows who carries the risk. According to Railway, AWS Bedrock and Google Gemini handle these cases under existing fraud policies, OpenAI reviews them case by case, talks with Anthropic are ongoing, and OpenRouter does not refund, so Railway covers those costs. The charge report form says Railway "is not responsible for usage of these keys" while the company covers the costs anyway. Customers should not count on that generosity next time, from Railway or anyone else.
What Railway changed
Each build now runs in its own short-lived virtual machine, build credentials can push only one specific image, and every user was moved to the new build system at once. Railway says its own hardening team built a second escape within an hour of the first patch, which is a candid admission that build isolation is hard. The company has filed a complaint with the FBI's IC3 and says indicators of compromise will be published once law enforcement agrees.
Practical takeaways
- Rotate at the issuer, not the platform. If you stored secrets on Railway before 21 October, revoke and reissue them with the provider that issued them. Filing a charge report does not revoke a key.
- Cap every AI key. Set hard spend limits, per-key budgets and usage alerts with each model provider. A $50 daily cap turns a $740,000 problem into an annoyance.
- Scope and restrict keys. Use per-project keys, IP or workload restrictions where the provider supports them, and short-lived tokens over long-lived ones.
- Keep the most valuable secrets off the PaaS. Fetch them at runtime from a secrets manager you control, with its own audit trail, instead of storing them as platform environment variables.
- Audit linked accounts. Review GitHub audit logs and authorised OAuth apps, and check registry audit logs such as Docker Hub for pushes you did not make.
- Close the side doors. Remove public network access from managed databases, so a leaked connection string is not enough on its own.
- If you run a platform, treat build credentials as untrusted, alert on KMS decrypt volume per identity, and assume the build sandbox will eventually be escaped.
