On June 3rd, our public API took 489 million requests in 24 hours (~5,660 requests per second), which is more than 200x what we serve on a normal day, sustained from one minute past midnight until the same minute the next morning. Somebody pointed real volume at the platform and held the trigger down for a full day, which is the kind of event that most early-stage projects find out about by watching their dashboards turn red, their error rates climb, and their users start filing tickets (or in our case blowing up our Discord). None of that happened to us. Miners kept submitting agents through the CLI the way they always do, the leaderboard kept refreshing for everyone watching the standings, the race kept running on schedule, and the dashboards kept loading the same data they load every other morning. If you were building on ORO that day, the only way you would have known any of this was happening is if we told you, and we didn't tell you, because there was nothing for you to do about it.
That last sentence is, in a sense, the entire post. But it's worth taking a few paragraphs to talk about why "nothing happened" was the only acceptable outcome for us, and why we'd been quietly working toward it for months before there was anything to defend against.
When you run a public competition with real prize money on the line, the people you're building for have exactly one expectation of you, which is that the platform works. They do not care, and should not have to care, about the infrastructure decisions we make underneath. They care that when they push an agent at two in the morning it gets picked up, evaluated, and placed on the board by the time they wake up. They care that when the qualifying window opens, it opens on time, and when the race fires, it fires on time. The moment any of those things stutter, the trust we've spent months building with the miner community starts to stutter with it, and trust in a competition is not a thing that recovers quickly once it's been spent.
So in the early months of ORO, while most of the team's visible work was on the things you'd expect (the race format, the cheating defenses, the static analysis, the scoring math), we were also spending a substantial amount of time on the parts of the platform that nobody would ever see. The competition logic gets all the headlines, and rightly so, because it's the thing that makes ORO so compelling in the first place. But underneath the competition logic is a public API, and underneath the public API is our backend system. It doesn't matter how good our incentive mechanism and race system is if the front door falls over the first time something pushes on it. We made the call early on to invest in the boring layer that holds everything else up, because the alternative is to spend your first real incident scrambling, and we did not want our first real incident to be the moment we learned what we should have built months prior.
The way we found out about this particular event was, frankly, anticlimactic. On June 1st, a Sunday, we served just over two million requests across the API, which is a perfectly normal day for us. Leaderboard polls from the website, dashboard loads from miners checking their agents, the API traffic behind the submission CLI, and the background hum of validators and integrators reading public data. On June 2nd, that number climbed by roughly a factor of ten, which is the kind of jump that gets flagged on a dashboard but doesn't yet rise to the level of pulling anybody off other work. We watched the curve, talked about it internally, and decided to keep watching. By the end of June 3rd, the same dashboard was showing 489 million requests for the day, which is when it stopped being a curiosity and started being an actual sustained attempt to put pressure on the platform.
What happened on our side during that window is the thing that's hardest to write a compelling blog post about, because the answer is essentially that nothing happened. The backend CPU graph for June 3rd is, to the eye, indistinguishable from the graph for June 2nd or the graph for June 4th. Our error rate did not move. No queues backed up, no retries cascaded, no alarms paged anybody. The same small set of operational metrics we look at every morning, like request latency, error rate, queue depth, and database load, all sat exactly where they normally sit. There was no heroic intervention because there was no intervention to make. The systems we had built months earlier to handle a day exactly like this one did what they were supposed to do, which was absorb the event without letting it bleed into the experience of anyone using the platform.
This is, I think, the genuinely interesting part of the story, and it's also the part that almost never gets written down. Building a platform that quietly does its job during a hostile event is not glamorous work. There is no demo to record, no feature to ship, no screenshot worth posting. The deliverable is a flat graph and a Discord channel that stays quiet, and you only find out whether the deliverable shipped when something actually tries to break the platform. We had done the work. The work paid off. And then the day ended, the traffic dropped back to baseline on June 4th, and we went back to whatever we were doing before.
The reason we're writing about it now is that I think there's something useful to say to other people building in this space. A lot of subnets, and a lot of platforms more broadly, are still in the phase where the happy path works in development and not much more is true than that. That is genuinely fine in the early days, when you have no users, no prize money, and no public surface area worth attacking. Every minute spent on resilience in that phase is a minute not spent on the product you're actually trying to find product-market fit for, and we made the same trade-off at the start of ORO that everybody else does. The trade-off, however, flips fast. The day you have real builders depending on the platform, real prize money flowing through it, and a real public surface area that's worth pointing volume at, the cost of a single bad incident is enormous and asymmetric. It is not just a lost day. It is a chip taken out of the foundational thing the whole competition depends on, which is the quiet assumption from every miner that the platform is fair, available, and not somewhere they have to babysit. Once that assumption is broken, every future submission carries a tiny extra cost. The cost of checking, of refreshing, of wondering whether this is the day the platform eats their entry.
Building ORO to handle a 489-million-request day is, in our view, the same kind of investment as building the race format, the cheating defenses, and the static analysis. The end goal is identical in every case: a competition where the only thing miners have to think about is the agent they're shipping. The race format means they don't have to worry about benchmarks being gamed. The cheating defenses mean they don't have to worry about being beaten by hardcoders. The platform work, the unglamorous, never-demo'd, mostly-invisible platform work, means they don't have to worry about whether the system will be there when they push the submit button. That is the platform we want to run, and on June 3rd, while a substantial fraction of a billion requests bounced off the front door, it is the platform we did run.
This post isn't supposed to be a brag. While we are incredibly proud that the systems we built worked as expected, this is also supposed to be a call to action for anyone else building in public, especially on Bittensor. The incentive design of Bittensor means miners will do the thing that is most profitable to them. If that thing is to top the leaderboard and then do everything in their power to stop others from doing the same, that is exactly what they will do. So if you're building a subnet, or any public competition with real incentives on the line, assume that day is coming for you too, and build the boring layer that absorbs it before the trigger gets pulled, not after. The day somebody decides to point real volume at your platform, you don't get to ask for an extension.