AWS’s faulty UX cost our business $7k and they refused to do anything about it
tl;dr: be very careful when using AWS. Their UX can be highly inaccurate and it might cost you money. If it does, don’t expect any support or sympathy from them.
After this experience, I cannot honestly recommend using AWS unless you have a professional sysadmin operate it for you, even if you think you know what you’re doing.
We trusted the UX. Big mistake
We believed it would be an accurate representation of what was happening under the hood. We were wrong.
In November of 2019, I set about using AWS to back up 24TB of raw video files from our feature film shoot + some personal data. Amazon had recently released its Deep Glacier storage tier, which charges $1/TB/mo, meaning we’d be paying around $40/mo to store 40TB of data.
I ordered the Amazon Snowball, a giant drive in a box they ship to you to transfer data. Once the data was in the cloud, I performed the following operations to my S3 bucket on the web interface:
1. Open bucket
2. Hit the “select all” box at the top to select all objects in the bucket
3. Actions -> Edit Storage Class
4. Select “Glacier Deep Archive”
5. Hit “Save Changes”
Happy with how simple Amazon had made this process, I forgot about it and turned towards other tasks: scaling our business from 3 to 10 and then 100 people, raising money, editing our film, and planning our release. Oh, and a virus shut down the world, I moved two times, and our business went completely digital.
I admit to negligence in not scanning our bank statements more carefully. It wasn’t until July of 2020 that I saw that instead of charging us $30–50/mo, Amazon was charging us $850/mo. For some reason, storage was being billed at the standard S3 rate.
In the end, we were charged $6963 more than we should have been.
I panicked. I was the most technical person on our executive team, and amid 18 hour workdays, I opened a support ticket on June 12, 2020, asking why we were being charged so much more.
The resulting exchange spanned:
65 messages
11,200 words
3 separate accountings of charges
and over 8 months to resolve.
Messages often took a full month to get a response. And in the end, nothing. No change whatsoever. Amazon wouldn’t budge.
Stupid lessons
It turns out that selecting all objects and changing storage class doesn’t change storage class on all objects. Instead, you have to create a lifecycle rule to transition objects, and tell the lifecycle rule to clear multipart uploads. Otherwise, many objects (in our case, most of them) won’t transition.
The Edit Storage Class box gives a warning that this operation will copy objects — my interpretation of that warning is that it creates a copy and deletes the original object, which should have done what I wanted to do.
It may also be that selecting a top-level object and changing its storage class is not a recursive operation. I’m still not sure.
As soon as I had a spare hour to read through support’s suggestions, educate myself on lifecycle rules and multipart uploads, and implement a lifecycle rule, our monthly bill dropped to $50.
Amazon will not take responsibility for their bad UX
It’s a valid crticism that I should have read the manual and audited our bank statements more carefully. But the reality of keeping a small business afloat during the pandemic was that I didn’t have the time to read the manual. And I expected a thing to do what it says it does. I trusted that it would, and if it doesn’t, that the company would own up to it.
If an interface doesn’t quite do what it appears to, it should give a warning. (Gmail, for example, warns you that you haven’t selected all emails when you hit select all.)
If the interface messes up and you are charged $7k for it, and the company is a giant behemoth and you are a tiny startup which under slightly different circumstances could have been bankrupted by this, that sucks.
Amazon needs to recognize that not all of its S3 users are huge corporations. If they want to get and keep small business customers, they need to have better UX, better support, and better remediation. Otherwise, small businesses are going to go somewhere else. I used to recommend that my film friends back up their media on S3, and cannot do so any more in good conscience.
Did something similar happen to you? I can be reached at michael@definitelyreal.com
