180,000 Child Programmers Versus 44 Ogres

Nick Winter -

In one week last week, Code.org’s Hour of Code reached more than 15 million students in 170 countries. Every major tech company promoted it, celebrities talked about it, and even the US President helped get the word out in their kickoff video. And shooting past Code.org’s crazy target of ten million players, kids are still continuing to play this week, with 600 million lines of code written and one in five US schoolchildren participating (with six times as many girls playing last week than have ever taken a computer science class in the US). It spread to more students in seven days than the first seven months of Facebook, Twitter, Tumblr, Pinterest, Instagram combined.

As one of Code.org’s partners, we at CodeCombat were both excited and hilariously unprepared to help teach such a sizable swarm of students to defeat the 44 ogres in our beginner campaign. Read on for what we learned from the onslaught of child programmers, including how obsessed kids are with games, how American students are the best trolls and the worst programmers, just how badly a user experience test can go, and the unfortunate difference between reddit traffic and school traffic.

Teachers Want Lessons, but Kids Just Want Games

Code.org lists 24 one-hour coding tutorials from partners across seven categories. These are ranked by teachers, and the higher the rank, the higher the clicks, roughly following a Zipf distribution. Probably because CodeCombat joined Hour of Code at the last minute, we’re relatively new, and the site still had some bugs to iron out when we were evaluated, we were placed in the 18th-most-desirable slot. That’s so far into the long tail, we expected to get around 0.1% of the total students going to the third-party tutorials.

But the thing is that many classrooms let the students pick which tutorial to do, and it seems students love games so much that every tutorial for playing or making games got far more traffic than it should have based on its ranking. So CodeCombat, the very last tutorial in the JavaScript section, ended up with 180,000 players--20 times what we expected based on our ranking. So although most teachers emphasize lessons, if they want to reach kids, it helps to do so through games.

This tweet, for instance, shows just how engaging a learning game can be:

COD is Call of Duty, the latest ultra-violent first person shooter which kids are playing at a rate of more than a millennium per day. Wouldn’t it be great if that kind of time was put to programming!

American Students Are the Best Trolls and the Worst Programmers

We could have predicted it, but we weren’t ready for the trolling. CodeCombat uses a live Olark chat box that lets players reach out to us. By Tuesday, we were disabling it for all but an hour a day because the nonsensical trolling by US students became so abusive. The most representative exchanges are too obscene to reproduce here, but here is one we can (mostly) print:

  • Student: heyy
  • Nick: hi there
  • Student: is your real name nick
  • Student: where you work
  • Student: wHERE DO YOU WORK
  • Nick: I'm in San Francisco
  • Nick: and yes
  • Student: yes what? where do you work
  • Nick: in my house
  • Student: I love you b*#%@
  • Nick: :(
  • Student: stfu0
  • Student: ugly ahh rappewt
  • Student: rappest
  • Student: lk;ldsjflkdsjfljsa

When the US school day ended and the Hour of Code went international, the conversations became human again. For example, Nick was chatting with an enthusiastic class of South Koreans for fifteen minutes, trying to guess whether they were college students or mature high schoolers, when this happened:

  • Student: If you need help with Korean translation just ask my friend, Alice she is 11 and has a interest in coding
  • Nick: how old are the students in your class?
  • Student: 10-12
  • Nick: Whooaaa, I thought you guys were in college! Korean students are way more mature than American students.

To be fair, we got a lot of very positive, intelligent responses from American students too, but they were overshadowed by the persistent trolling of their peers. Apart from those nonsensical dialogues, the most common topic of conversation was proving that we were in fact real humans. Despite our best attempts to pass their Turing tests, we failed to convince several students.

Curious about the differences between countries, we started collecting statistics at the end of the week. For 18 countries with more than 40 players beating the first level, students from ten of those countries were significantly more successful than the US and UK students, and students from six of them were much faster. Only three countries were either slower or less successful--and this being despite the game only being partially localized!

Incidentally, thanks to our volunteer Diplomats for translating the parts we were able localize. You helped tens of thousands of non-English-speaking kids learn to code last week.

Our Worst User Experience Test of All Time

We run user experience tests frequently on CodeCombat, and although we’d had things stable for a while, as we rushed to finish new versions of our beginner levels before the Hour of Code, we decided to run another round of UX tests just to double-check things. The in-person tests went fine, but two of the online tests from usertesting.com were disastrous. We watched for 15 minutes as a well-intentioned player went from naive optimism to rage quitting. The heartbreaking moment was when she reached out for help on our Olark chat, and Nick answered ready to help out, but then a bug with auto-focus on the code editor made it impossible for her to type out her reply in the chat box, instead redirecting her typing to the code!

It was ominous to see this intelligent, compassionate user so quickly discouraged by our creation. We fixed those bugs, and as the students poured in, they managed to break most of our levels in ways we couldn’t have even predicted. Here are a few of the highlights:

Thank you to everyone that pushed through our levels, gave us the benefit of the doubt, and helped us improve for future players.

Reddit Traffic Versus School Traffic

When we launched our beta in October and melted our server with the onslaught of reddit, Hacker News, Startup School, and Brazil, we were only able to serve about 250 concurrent players at peak. So when Hour of Code hit and we saw 643 concurrent players, we looked nervously over to our server dashboards. 1% CPU usage on all application nodes? Was it working? Was the multiplayer even turned on?

Yes, the new AWS cluster architecture that Michael has set up is just that efficient, and Firebase hummed along nicely as well. So sorry, reddit, but the days of us cowering before your hugs of death are over. You’re going to have to hit us harder than that!

But we weren’t prepared for one thing. Redditors use modern browsers like Chrome (67%), Firefox (21%), and the latest Safari (5%). Only 1.5% were on Internet Explorer. Schools, on the other hand, have about 14% Internet Explorer usage, and run far more out-of-date browsers of every kind. So when Hour of Code linked directly to the first level, bypassing our polite “your browser may not work” warning on the homepage, there rose a collective wail from ancient browsers across the globe. It didn’t help that our hard-won IE 9 compatibility had stopped working the day before.

We’re sorry to those students who tried it on Monday before we did what everyone says not to do and implemented a server-side browser-sniffing redirect to a “Sorry, you can’t play in this old browser” page, which instantly fixed almost all of our problems. You can still play from home!

Conclusion

We’re extremely proud to have been part of the Hour of Code, with CodeCombat players contributing another two million lines of code last week, and want to extend a sincere thank you to the Code.org team both for letting us participate and for doing such a phenomenal job of promoting the event and content.

We can’t wait to participate again next year and help push programming adoption forward in thousands of schools across the world.

Next up: watch this blog for the announcement of our upcoming open source launch, wherein we will put every last line of CodeCombat's code on GitHub.