Pretty much all tech demos are fake. Not our words but the words of Gizmodo, who argue that tech demos are rehearsed and optimised to within an inch of their lives, with a pinch of dramatic licence from PR and marketing sprinkled to make the end result look sexier than it actually is.
Sort of true. We’ll let Gizmodo off in this instance because they said ‘pretty much all’. Our Aether Engine tech demo at GDC, where we aimed for 10,000 players joining the same online battle in EVE: Aether Wars, was as real as it gets. It was real players getting their hands on real technology in a real environment that we simply couldn’t have rehearsed for, even if we wanted to.
After all, we wanted to see what would happen with thousands of players turning up to push our Aether Engine tech in new and unexpected ways. The whole point of the stress test was that it was unique and not something we could rehearse or practice in advance.
The advice we heard again and again was that we shouldn’t do it. Not live. Not 10,000 players. You’re mad! Don’t do it. But we had confidence in our Aether Engine tech and EVE: Aether Wars was a huge success.
We thought it would be interesting to revisit the five things we learned from doing a tech demo live… and the reason most tech companies won’t do it!
This is the biggest reason most tech companies won’t run with ‘real’ tech demos in a live environment. No matter how much you prepare and rehearse and optimise and mitigate risk, there’s still a small element of the unknown that could strike at any moment, in any place. In our case, thankfully, it was something that didn’t cause any harm but it’s an example of why companies shy away from these large scale, live tech demos.
We ran a series of smaller EVE: Aether Wars tests in preparation for the 10,000-ship extravaganza at GDC. There were several internal tests at our office, with the number of players topping out around 25 to 30, and there was a successful 1,000-ship battle just one month away from The Actual Real Test. We also gave some insight into how we tested a 10,000-player deathmatch before running the stress test live at GDC.
We had prepared as thoroughly, as carefully and as diligently as we could. But nothing prepares you for the moment you have to go live in front of the world. In this case, we were going live with thousands of eager players ready to jump in and push our tech. Aether Engine worked exactly as we knew it would and it was a real thrill watching the player number slowly creep up as the stress test progressed.
But when we sent the EVE: Aether Wars keys out to those who signed up, a small percentage of the playerbase found an issue with binding keys a certain way, something that didn’t come up in our many tests. Fortunately, our brilliant community discovered a workaround the issue almost as quickly as they discovered the problem, but it just goes to show how the unpredictable element of live tech demos can flare up when and where you least expect it.
We thought we had a good handle on the interest in our EVE: Aether Wars stress test because we could measure that interest in raw sign-up numbers. As our behind the scenes blog post illustrated, we had over 15,000 players signing up to take part in our ambitious stress test. So our job was easy - focus on those who signed up and make sure we catered to them. Where were the majority of our sign-ups from? What time for the stress test was our community asking for? What sort of set-ups do they have? And so on.
The one element we didn’t account for, which sounds obvious in hindsight, was how many people would visit our booth to see the stress test when it started. In short, our booth was slammed as soon as the tech demo started, as word of our ambitious tech demo quickly spread around GDC. And what do crowds of people attract? More people, wondering why there’s a crowd. As that crowd grows bigger, more and more people drop in, curious to see what’s going on.
GDC has fairly strict rules in place to keep its aisles clear so attendees can navigate its halls safely. One rule, and we’re paraphrasing slightly here, is ‘make sure you don’t have lots of people standing around and blocking the aisles’. That’s something we didn’t account for and we spent an hour awkwardly shuffling people around our booth to try and keep the aisles clear, while making sure they could still see the action, until the live stress test ended.
In short, we apologise to Insomniac Games, who had a stand opposite us and experienced a sudden, unexplained surge of people overspilling from our booth at 11am on the first day of GDC.
Perhaps our biggest learning is that it’s not just the live tech demo itself that presents challenges. As the previous point illustrates, it’s dangerous to focus just on the tech demo itself, and one factor we had overlooked was the rush of delegates to our booth when the stress test took place, adding a few local logistical challenges on the day that we simply didn’t expect.
Another challenge was finding the right time zone to satisfy our broad playerbase who signed up to take part in the challenge from countries all over the world. This live stress test project had been in the works for almost six months and our first decision was picking GDC as the perfect platform to stress test our technology live, with thousands of players taking part. What better way to demo your technology that to use the biggest games development conference in the world as your platform?
It was only as GDC approached that we realised that this approach limited the amount of players who could actually take part. GDC runs from Tuesday to Thursday from 10:00 to 18:00 US time. We had players signing up from all over the world and understandably, some of them found those hours incompatible with their working or personal lives. We wouldn’t have found the right timeslot to satisfy every player, of course, but it was still a limiting factor we hadn’t considered from the outset. Another valuable lesson for us.
Also, here’s a little behind-the-scenes story for you. Shortly after the EVE: Aether Wars stress test concluded, we were scheduled to host a lightning talk on our stand from CCP’s CEO Hilmar V. Petursson. Right on schedule, we fired up his slides, got Hilmar on the podium and… no sound. The microphone wasn’t working. What? How? We had tested everything that morning and everything was fine. The microphone was plugged in. The speakers were on. There was nothing wrong with the audio mixer. The crowd started to impatiently fidget. Panic was setting in. Why isn’t it working?
Turns out, the microphone wasn’t switched on. Sometimes, it’s the little things you overlook...
In many ways, a live tech demo is a product launch. You’re scrambling against a tight deadline to finalise the product, the messaging, the assets, the logistics, and so on. When everything is in place, you go ahead with the live tech demo, revealing your product to the world in the biggest and boldest way possible.
The easy trap to fall into here is thinking that when the live tech demo concludes, the hard work is over. It’s the opposite. The hard work is just beginning. It’s important to remember that when the live tech demo concludes, the riskiest part is over, and you need to get ready to take advantage of the momentum and interest you’ve just gained.
Fortunately, it was something we had prepared for, as we knew the EVE: Aether Wars stress test would be a launchpad to huge interest in Hadean. We were ready for the questions, the interviews, the probing, the visits, the emails, the tweets... everything that followed, we were prepared for.
There’s a danger that having made it through your live tech demo, you switch off a little, mentally and emotionally. But you need to keep going because the time immediately following a successful live tech demo is arguably even more important than the tech demo itself, because that’s when you need to take advantage of and build upon the tech demo’s success.
Good food and industry conferences are not happy bedfellows and neither are breakfast burritos and avoiding indigestion. If there’s one thing we would do over again from GDC, it’s that we would avoid breakfast burritos, no matter how convenient they are. Our bodies are still recovering from the eggy onslaught of beige and beans...