Back when I was Director of Engineering at a previous company, my boss came to me looking for answers, and he wasn't happy. That was Erik Loyd, our COO at the time. Today he's my co-founder and Sentasity's CEO, which makes this story a little funnier now than it was then.
Someone had spun up an entire reference architecture of YugabyteDB, a distributed SQL database built to run at very high scale. It's the kind of thing you deploy when you need serious throughput across multiple availability zones, and it's priced accordingly. It was sitting in one of our accounts, running around the clock, and nobody could tell him why.
His question was the obvious one: whose is this?
I didn't know. So I started asking around.
"I think he was working on something like that," someone told me, and pointed me at another engineer. He was in a meeting, so I waited. When he got out, it wasn't his, but he had a guess about who it might be. That person had a guess too. At one point I posted in a couple of big Slack channels, which got me a few "not mine" replies and interrupted a lot of people who had nothing to do with it.
Eventually I found the engineer who'd launched it. It wasn't a secret project or something production depended on. He'd set it up to try something, moved on to other work, and forgotten it was there.
"Sorry about that!"
That stung a little. The cluster had cost the company thousands of dollars and earned me a reprimand from Erik, and the whole thing ended with a cheerful apology. I don't remember the exact number, but a production-sized YugabyteDB cluster on AWS runs about $2,000 a month at on-demand prices, and this one had been running for a while.
It wasn't really his fault
The easy version of this story ends with a lesson about one careless engineer. I don't think that's the right lesson.
The way those experiments get left behind is completely ordinary, too. You need an instance for an hour or two, so when the console asks which instance type you want, you think "what the heck" and pick something with the newest, most expensive GPU on the list. Then a production fire starts, or the CEO has an urgent request, or there's an emergency at home, and that instance is the last thing on your mind. Or you plan to pick the work back up tomorrow, so you leave it running overnight. Tomorrow never comes, and it keeps billing.
Nobody in either of those stories did anything unreasonable. Multiply them by every engineer and every experiment, though, and you get a steady layer of things nobody is using and nobody remembers starting.
The bill tells you what, never who
There weren't any guardrails, either. Anyone with access could launch anything, at any size, and nothing recorded who it belonged to or how long it was supposed to live.
So every forgotten resource turns into a manhunt. Someone notices the cost, someone gets asked about it, and that person starts the same detective work I did: asking around, waiting on meetings, getting redirected, posting in channels and hoping. And it's rarely just one resource.
The money is the part everyone notices. The part nobody budgets for is the time, and it usually comes from the person with the least of it to spare. I had very little time to give back then, and I spent a real chunk of it on a scavenger hunt.
What I wish I'd had
Looking back, what I wanted wasn't a better way to hunt. It was not needing to hunt at all. If that cluster had carried an owner tag and a project tag from the moment it was created, Erik's question would have taken thirty seconds to answer. If it had also carried a "needed until" date, it might never have become a question, because someone would have noticed it was past its date.
That's a tagging policy: a short list of tags that everything has to carry, like owner and project. The idea is simple. Making it hold is a different story. It has to work whether people deploy with CloudFormation, Terraform or the AWS console. The values have to be consistent enough for a machine to read. And enforcing it strictly enough to matter can break the very deploys you're trying to tag. That deserves its own post, and I plan to write it.
For now, the question I'd ask is simpler. Find the most expensive thing in your account that you can't immediately explain. How long would it take you to find out who owns it?
