AWS

`3:18:25`

or 3.4 million at its peak. That's kind of a lot of money. 3.4 million, I mean we have a ton of users and customers, but still that just struck me as unreasonable. And the reason why it was so unreasonable was because I had the pitch for the cloud ringing in my ears. Hey, this is gonna be faster. This is gonna be easier. This is gonna be cheaper. Why are you trying to produce your own power, right? Like, do you have your own power plant? Why would you do that? Leave the computers to the hyperscalers. They're much better at it anyway. I actually thought that was a compelling pitch. I bought in on that pitch for several years and thought, do you know what? I'm done ever owning a server again. We are just gonna render our capacity and Amazon is gonna be able to offer us services much cheaper than we could buy 'em themselves because they're gonna have these economies of scale. And I was thinking Jeff's word ringing. My competitors margin is my opportunity. That was something he used to drive amazon.com with.

`3:19:26`

That if he could just make 2% when the other guy was trying to make 4%, he would end up with all the money. And on volume, he would still win. So I thought that was the operating ethos for AWS. It turns out that's not true at all. AWS by the way operates its almost 40% margin. So just in that, there's a clue that competitors are not able to do the competitive thing we like about capitalism, which is to lower costs and so forth. So the cloud pitch in my optics, it's fundamentally false. It did not get easier first of all. I dunno if you've used AWS recently, it is hella complicated. If you think Linux is hard, you've never tried to set up IAM rules or access parameters or whatever for AWS. — AWS is always difficult. It was always complicated. — Well, I think it's gotten even more difficult. But yes, now some of that is it's difficult because it's very capable and you have a bunch of capacity on tap. And there are reasons I don't think they're good enough to justify how complicated

`3:20:26`

the whole jing-a-ma-jing has become. But what's certainly true is that it's no longer easier. It's not easier to use AWS than it is to run your own machines, which we learned when we pulled out the cloud and didn't hire a single extra person. Even though we operate all our own hardware, the team stayed exactly the same. So you have this three-way pitch, right? It's gonna be easier, it's gonna be cheaper. Certainly wasn't cheaper. We've just proved that by cutting our spend on infrastructure by half to two thirds. And it's gonna be faster. The last bit was true, but way too many people overestimated the value of that speed. If you need a thousand computers online in the next 15 minutes, nothing beats the cloud. How would you even procure that? If we just need another 20 servers, it's gonna take a week or two to get boxes shipped on pallets delivered to a data center and unwrapped and racked, and all that stuff, right? But how often do we need to do that? And how often do we need to do that

`3:21:27`

if buying those servers is way, way cheaper so we get vastly more compute for the same amount of money. Could we just buy more servers and not even care about the fact that we're not hyper optimized on the compute utility? That we don't have to use things like automatic scaling to figure things out because we have to reduce costs? Yes, we can. So we went through this journey over a realization in early 2023 when I had finally had enough with our bills. I wanted to get rid of them. I wanted to spend less money. I wanted to keep more of the money ourselves. And in just over six months, we moved seven major applications out of the cloud in terms of compute caching databases to works onto our own servers. A glorious, beautiful new fleet bought from the king of servers, Michael Dell, who really, by the way, is another icon on my, I saw he just celebrated 41 years in business. 41 years this man has been selling awesome servers

`3:22:28`

that we've been using for our entire existence. But anyway, these pallets arrive in a couple of weeks and we rack 'em up and get everything going. And we were out. At least with the compute part. We then had a long multi-year commitment to S3 because the only way to get decent pricing in the cloud, by the way, is not to buy on a day-to-day basis, not to rent on a database basis, but to bind yourself up to multi-year contracts with computes often a year. That was in our case and with storage is was four years. We signed a four-year contract to store our petabytes of customer files in the cloud, to be able to get something just halfway decent affordable. So all of these projects came together to the sense that we're now saving literally millions of dollars, projected about 10 million over five years. It's always hard. How do you do the accounting exactly, and TOC this, that and the other thing. But it's millions of dollars. But it's not just that. It's also the fact that getting out of the cloud

`3:23:30`

meant returning to more of an original idea of the internet. That the internet was not the sign such that three computers should run everything. It was a distributed network such that the individual nodes could disappear and the whole thing would still carry on. DARPA had designed this such that the Russians could take out Washington and they could still fight back from New York, that the entire communication infrastructure wouldn't disappear because there was no hub and spoke. It was a network. I always found that an immensely beautiful vision. That you could have this glorious internet and no single node was in control of everything. And we've returned to much more of a single node controlling everything idea with these hyperscalers. When US-East-1, the main and original region for AWS goes offline, which has happened more than a few times over the years. Seemingly a third of the internet is offline. Like that in itself is just an insult to DARPA's design. It just detract from the fact

`3:24:30`

that what AWS built was marvelous. I think the cloud has moved so many things so far forward, especially around virtualization, automation setup. It's all those giant forward for system administration that's allowing us now to be able to run things on-prem in a way that smells and feels much like the cloud just at half the cost or less, and with the autonomy and the satisfaction of owning hardware. I don't know what the last time you looked at like an actual server and took it apart and looked inside of it. These things are gorgeous. I mean, I posted a couple of pictures of our racks out in the data center and people always go crazy for 'em because we've gotten so abstracted from what the underlying metal looks like in this cloud age that most people have no idea. They've no idea how powerful a modern CPU is. They have no idea how much RAM you can fit into a one U rack. Progress in computing has been really exciting,

`3:25:32`

especially I'd say in the last four to five years after TSMC with Apple's help really pushed the envelope. I mean, we kind of sat still there for a while while Intel was spinning their wheels going nowhere. And then TSMC with Apple propelling them really moved things forward. And now servers are exciting again. Like you're getting jumps year over year and the 15, 20% rather than the single digit we were stuck with for a while. And that all means that owning your own hardware is a more feasible proposition than it's ever been. That you need fewer machines to run ever more and that more people should do it. Because as much as I love Jeff and Amazon, like he doesn't need another whatever, 40% margin on all the tech stuff that I buy to run our business. And this is just something I've been focused on. Both because of the ideology around honoring DARPA's original design, the practicality of running our own hardware, seeing how fast we can push things

`3:26:34`

with the latest machines and then saving the money. And that has all been so enjoyable to do, but also so counterintuitive for a lot of people because it seemed, I think for a lot of people in the industry that like we'd all decided that we were done buying computers, that that was something we would just delegate to AWS and Azure and Google Cloud, that we didn't have to own these things anymore. So I think there's a little bit of whiplash for some people that, "Oh, I thought we agreed. We were done with that." And then along come us and say, "Ah, do you know what? Maybe you should have a computer." — Is there some pain points to running your own servers? — Oh, plenty. There's pain points to operating computers of all kind. Have you tried just like using a personal computer these days? Half the time when my kids or my wife have a problem, I go like, have you tried turning it just off and on again. Computers are inherently painful to humans. Owning your own computer though kind of makes some of that pain worth it. There's a responsibility that comes with actually owning the hardware that to me, at least make the burden of operating