A story post about the time a Stripe account I used years ago had thousands of pounds worth of unexpected payments flood in.
Part 1: Unexpected Cash
What is this about?
An evening in early August 2022 and I notice an unexpected email come through on my phone. It is a Stripe notification telling me I’ve received a payment! That should be exciting news, except I’m not selling anything online that I know about. Initially I ignored, thinking it was a curious incident, maybe a super convincing spam email, I’d check it out properly tomorrow in front of a laptop. Fast forward an hour later though and I check my phone again and see these emails have been coming in continuously. Something is amiss!
What transactions?
Sudden influx of customers called 'donation'... pretty suspicious!
I do have a Stripe account from some long ago projects. I log in and take a closer look at the transaction log. Now I’m sure something is not right. Transactions are:
- All in US Dollars. The Stripe account I used was attached to a site that only sold things in GBP.
- All for very low amounts ranging from $1 to $3.50. I’ve never offered a product or subscription for that amount.
- The customer names are all different but look auto generated. All lowercase and similar length.
- The payment is named as ‘donation’. I don’t have a site asking for donations.
- Most payments are either declined or marked as refunded.
Part 2: Cutting It Off
At this point I’m not sure what the deal is but I’m very sure these aren’t actual donations for me to put towards my coffee funds.
My Stripe account is using a long random LastPass password and MFA, so I don’t think that login has been compromised. The API keys for my account are another matter though.
I revoke my Stripe API keys and the transactions immediately stop appearing.
The SaaS Background
Before carrying on let’s pause so I can explain why I have a Stripe account hooked up to a website if I am not expecting any sales.
In 2018 I set up some regular WordPress maintenance scripts as-a-service. I planned to charge a monthly fee for businesses to make use of them. Once I’d got a working proof of concept I spun up a brochure site where anyone could sign up and buy the service. It wasn’t actually fully automated, but the site account and Stripe subscription creation parts were. The site was a PHP Laravel site with the Cashier module that integrates with Stripe for subscription management.
The service wasn’t a runaway success, but I did sign up a few businesses that I’d previously worked with doing bits of web development. I then moved on to other projects and left that brochure site running with zero maintenance. That means for years the site has run with all its old libraries and quickly thrown together code, just how I left it in 2018.
Finding The Vulnerability
Back to the issue at hand, just stopping the transactions by revoking the API key isn’t good enough. We need to find out how these bad payments were being made. I know that I’m very behind on software updates for Laravel and Cashier so decided to start there. A little library update to fix would be nice and simple.
The Cashier version used on the site is 9.3. They are now on version 14, so I was a good number of major versions behind. There didn’t appear to be any reported vulnerabilities with v9 and scanning the changelog since doesn’t suggest there have been any fixes regarding a flaw in the library that would allow completely rogue transactions.
With that in mind my starting hypothesis became that the mini subscription site I wrote has a bug in it. After all, thinking back to Pragmatic Programmer tips, “select” Isn’t Broken.
I crawled the error and access logs for Apache. The error log didn’t have anything useful. The access log didn’t either but the lack of entries was significant. I had expected to see requests at the time of the bad transactions, thinking these would be hitting some API endpoint that was vulnerable. Instead all I discovered were Stripe webhooks calls.
These webhooks are just HTTP requests form Stripe calling back into my site to tell me a payment has been made. Cashier then handles these to mark a subscription in my own app logic as ‘active’ (or whatever is appropriate if it knows about the customer).
Seeing only the webhook entries tells me that the attacker knew the Stripe API keys and was executing requests from elsewhere (later confirmed via Stripe dashboard logs). That means the vulnerability is in revealing the API keys, not in having Stripe API calls maliciously executed via my own API.
False Starts
Initially I thought I’d cracked it when I found some debug views still in the production site. One of these was a test HTML template that dumped the entire Stripe session information. However a closer look showed the view was never called or included by an active route so feels very unlikely to be the root cause.
I also had a mini shock when I thought I’d dumped out the Stripe API private
key in to the client side JavaScript but this turned out to be fine. The
environment variable is called STRIPE_PK, but that’s the publishable key not
private key when I chased back to the site settings. Phew.
An aside point - there is some programming wisdom buried in there. Don’t use a short name when you can use the full name and avoid any ambiguity!
When is a stack trace and debug output not useful? Answer: In production.
Still not finding an obvious error in my code meant I resorted to some crude web crawling tactics. I went through each HTTP route one-by-one, firing in valid and invalid requests. To my surprise this worked, I found a leaky page.
I had created some web routes that were designed to allow a user to download an invoice. I don’t think these were ever used, but they were live. A simplified version of the code looks a little like this:
Route::get('user/invoice/{invoice}', function (Request $request, $invoiceId) {
return $request->user()->downloadInvoice($invoiceId, [...]);
});
Note the URL is a very generic and guessable route, user/invoice/$anything.
Here’s the kicker. user() might not return a user object if no one is logged in.
In that case we throw an exception, Call to a member function downloadInvoice() on null.
Anyone could just call this URL directly (as a normal GET request, no login or
session required) and trigger the error case.
When the exception throws we get a nice error page which shows all our request variables
and also the server environment variables!.
Not what you want in production!
This is a complete disaster for a production site.
Where does this come from?
Laravel has the idea of running in different environments. You can use the env
files to tweak things between development and production (or whatever environments
you want to define).
The environment
configured on the live site was actually set to production, but the configuration also has
APP_DEBUG=true set which is why a 500 server error response was giving such neat and useful output. With that set to false
and a request made against the same URL we get something much saner (and
safer!).
The docs for APP_DEBUG even have a big warning about this. Important to read the manual
when using someone else’s framework.
Nothing to see here!
The access logs show that lots of web scrapers hit the server all the time. I
assume one of these tried an endpoint that gave a 500 with interesting content
(not exactly a difficult find, /user/invoice/foo will do it!), then searched
the returned response body for anything that looks sensitive like API keys or
passwords.
Part 3: Finishing Up
The Transactions
The Stripe developer dashboard does provide a full web log allowing you to see all the API requests within a date range. Unfortunately there doesn’t seem to be an easy way to export, so you’re stuck poking around via the web interface without full pagination. This is fine except in my case there were thousands of requests and the web UI will only filter by date, not time.
My investigation shows that the bad requests all came from some Linode IP.
The attacker’s script would create a Stripe source, a credit card. This would
be a different card each time an, doesn’t appear to be limited to just one
brand (I’ve seen Visa, Mastercard, American Express and Discovery in the logs).
It would supply the card number, expiry and name. It doesn’t appear to have
filled in the CVC. The name is always given as one word which looks
weird (e.g. name: johndoe).
A lot more chargeable cards were added than those where a payment was attempted. I’m not sure whether there is reason behind that.
All payments either failed or succeeded but were then refunded. Stripe marked a large proportion as fraud, interestingly not all of them, which feels weird when to me they all look so obviously bogus.
I think the idea behind this type of attack is to find out if these credit card details are valid. Stripe even have a section on this activity on their fraud page under Card Testing. I assume they were all sourced from some leak or hack elsewhere on the internet.
I did notify Stripe but that wasn’t super helpful. I guess in the big scheme of things the numbers here are tiny and their automated processes mostly helped keep things under control. The support person I chatted with did come up with a great quote though:
"...API keys, they are generally not considered secure; they are typically accessible to clients, making it easy for someone to steal an API key."
Stripe Support
My guess is that they are used to answering requests about the publishable part of the key. Either that or I’ve completely misunderstood how the ‘secret key’ is supposed to be used! The text from me that led up to that answer included the phrase ‘I believe my secret key has been leaked’.
Lessons
I guess the main lesson is not to leave any leaky debug pages or settings in production. You’re welcome for the top tip. Beyond that, here are the things I’d do differently when setting up such a website again:
- Restrict API keys just for the actions that matter. Possible since late 2017.
- Sites that aren’t static either need to be retired or kept up to date. I guess there is a mid-point where you might consider it somewhat safe to kick in to the long grass. A site that interacts with a payment processor is not the type of thing to just leave though. Whilst I wouldn’t have fixed the root cause here, Cashier have a policy of only applying security updates to the last major version so it’s obviously an increasingly high risk to not upgrade.
- Lock down your Stripe account. I didn’t know about this prior to writing the blog but via Stripe’s Radar (basically anti-fraud) features you can create rules to auto block payments that aren’t in the currency or country you expect. Assuming this has some alerting I could have found out about the leaked API credentials but without actually having to process any payments through my account.
An Eventful Evening
All things considered I think I got off pretty lightly. Aside from some stress whilst stopping the transactions then finding the likely source of the issue this whole episode hasn’t come at too high a cost. It feels close though, when I was researching some bits for this post I came across another fraud experience here. Their story is with much larger (read scarier) amounts and bigger consequences. Fortunately my brush with Stripe fraud was not quite so dramatic!