GDC 2013 Slides


I uploaded the slides from my GDC talk, “Network Serialization and Routing in World of Warcraft”.  You can download them in several formats:

  • Keynote What I used to build and present them
  • PowerPoint Auto-exported from Keynote, haven’t checked it AT ALL.  Likely has a few broken things
  • PDF – maybe your best bet to just read them

Also, Simon Koo (@sm9kr) seems to have written a summary of the talk in Korean along with pictures of all or most of the slides.  I’m taking his word for it that the text is actually about my talk!


Comments (14)

Voice Controlled Garage Door

As seen previously (more), I’ve got our garage door hooked up to the internet. Now that we’ve both got Siri though, it seemed like I ought to be able to ask her to open the thing. Apple doesn’t yet provide any kind of API into or out of Siri, but you can get her to send texts.

Which brings us to Twilio, a fantastically easy to use SMS and voice menu API. It’s about as simple as I can imagine it being to set up a number on Twilio that, when texted, forwards the text along to a web URL of your choosing. So all I needed to write was a tiny web app to bridge SMS messages from Twilio to Indigo‘s web interface.

Since I’ve been working on Buzz, a tiny little web framework written in Lua, I decided to use that for my bridge.

So given all the various pieces were already in place, this is all the new code I had to write for this entire project. As a bonus, I can turn the lamp behind me in my office on and off too (mostly because it was a lot more comfortable sitting at my desk writing this than sitting on the garage floor!)

With this app running and Twilio set up to send texts to it, I made a contact on my phone named “House” with my new Twilio number, and can now tell Siri “Text house open the garage”, and the door will open. Technically it will open or close, it’s just a toggle, but you can see how to modify this code to do more stuff, or be smarter about it, pretty easily. Until Apple opens up the Siri API, this is a pretty good poor man’s substitute.

local ffi=require('ffi')
ffi.cdef [[
	int	system(const char *command);
function homecontrol(request)
  request.headers['Content-Type'] = 'text/xml'
  body = body:lower()
  if body:find('lamp') then
	ffi.C.system('curl -X PUT -d toggle=1 http://sideshowbob.local:8176/devices/OfficeLamp')
	response='Turned the office lamp on or off'
  elseif body:find('garage') then
	ffi.C.system('curl -X PUT -d outputBinaryStates="1,0,0,0,0,0,0,0" http://sideshowbob.local:8176/devices/GarageDoor')
	response='Activated the garage door'
				'<?xml version="1.0" encoding="UTF-8"?>\n'..
buzz.get('^/homecontrol/\?.*$', homecontrol)

Some issues with the above code:

  • There’s no security at all, other than the phone number being a secret. Anyone could open the garage if they knew the right number to send a text to, or the right URL to send a Twilio-like request to. There are several really simple things that could be done to greatly increase security, if one were worried about it.
  • It should really be set up as a PUT or POST request, as something is OBVIOUSLY changing state as a result of the request! But Buzz only does GET requests right now. Luckily Twilio can deal with that.
  • It requires luajit, rather than working with plain old lua, since I’m using ffi to make a system() call to curl. There are better ways to do that, but ffi makes things like that really easy for experienced C programmers. Buzz itself requires luajit for the socket layer anyway, though I don’t think it will forever.

Comments (5)

Thanks, Steve

Steve Jobs’ passing has caused me to reflect on just how much of my life is directly or indirectly influenced by his work. Although the first computer we had in the house growing up was an Imsai 8080 (you may not recognize the name, but you’ve seen one in War Games with Matthew Broderick), the first computer that was “mine” was an Apple ][. My dad bought a used one for me for Christmas when I was 11 or 12, meaning it was already a couple of years old. But at the the time, a computer could be two years old and still be state of the art. That computer stayed on my desk through high school, and it’s still in a box in my mom’s garage. I got online, on BBSes, for the first time on that computer, using a modem and an I/O card that I think cost my dad about $20, and which were so generic we had to write our own “drivers” (I’m not sure that’s what we called drivers back then, but that’s what they were) to get them to work at all.

At the time, Wozniak was more of an idol to me than Jobs, being the geeky hacker who made it all possible. I didn’t have anything against Jobs of course, but to a kid like me, the Woz was the awesome one. As an adult, I see how important Jobs really was in actually getting that computer onto my desk. I have no doubt that Woz would have made the same computer with or without jobs, and then sold a few dozen as kits to people around Silicon Valley.

I jumped ship to Amiga when that came out. I never had an early Mac. But when I got to college, there was a room full of NeXT computers, so once again I was using Jobs’ creations. Although we had Sun workstations too, the NeXTs were my favorite. I even used them remotely for homework when dialing in by modem from home during the hours the lab was closed (and occasionally when it was open, probably annoying whoever was sitting at the console, most likely wondering why things were so slow that day!), because no one said I couldn’t, and I got a whole machine to myself that way rather than sharing time on the Sun server we were “supposed” to use.

Out of college, my first professional game industry job involved writing the network layer for Close Combat, which ran on both Macs and PCs. That was the first time I really used a Mac, and I actually hated it, but that had a lot to do with how non-standard the network stack on pre-OS X Macs was, since that’s what I was mainly dealing with. Also, the machine I had was a prototype on “loan” from Apple, which I believe my boss eventually told Apple they lost because they didn’t want to spend money on a new one. It had a flaky video connector I had to prop up with a book, and I seem to remember I had the case off of it due to heat problems too. But I never held that against Apple, just my boss, it was not a production machine after all.

My next experience with Apple was inheriting the iBook my dad bought not long before he died. It was his first Apple product since he’d bought me that Apple ][, and I think his favorite computer he’d had in a long time. That was right after OS X had come out, and like me, he’d spent a lot of years on Unix systems. Although I didn’t use that computer as “intended” very much, I did use it as a little low power mail server for a few years, and the few times I did use it like a Mac made me realize Apple was definitely on to something. It was just a little underpowered for actually running OS X. But it was only a few more years before I had a PowerBook of my own. Plus an iPod a couple of years after those came out. I’ve had at least one Apple product in daily use ever since. Right now, there are four within arm’s reach (MacBook, iPhone, iPad, AppleTV remote!). There is almost never a moment I’m without my iPhone. Even if I was using Android phones, I’d still have a lot to thank Steve for.

So although there are significant gaps in there, I think it’s safe to say that I’ve used Jobs’ various products daily for the majority of time I’ve been using computers, and I’ve spent a lot of time on computers. There are other tech industry figures I might admire more than Jobs, but none that have had anywhere near the impact on my life that he did, nor will there ever be anyone else who can match his influence.

Here’s to the crazy ones.

Comments (4)

Mouse Potato

Mouse Potato lets you track what TV shows you’ve watched, and tells you what episodes to watch next.”

I spent a couple of days making a website for tracking what TV episodes you’ve watched. It’s live at, free for anyone to use. It’ll stay that way unless it somehow gets popular and I have to figure out better hosting. Not that there’s anything wrong with Dreamhost, I’m happy with them, but it’s running on the same VM as this blog and @lnorigb’s blog and other stuff.

From the about page:

Other sites concentrate on telling you what episode is going to air next. At Mouse Potato, we don’t care. We watch shows on DVRs, on Blu-Ray, and on the internet. We just want to remember where WE left off, not where the network left off.

Mouse Potato remembers what episodes you’ve watched, and tells you what the next one is. That’s all there is to it. Sign up and get started today!


Down the NoSQL Rabbit Hole

Although I’m going back to work in a little over a week, I’ve been working on an iPhone game in the meantime. More on that some other time (soon!) This game has an online turn-based component, kind of like Scrabble (Also Words With Friends), Carcassone, Disc Drivin’ and other great iPhone games. So besides the iPhone front end, I’m writing a Django back-end since I’m already familiar with Django and it’s nice for getting things done cleanly and quickly.

Of course, the back end needs a back end too, so rather than just shoving everything in MySQL, I decided I’d try out newer, more scalable, stuff just in case I happen to write the next Angry Birds. Am I likely to need more than a single server MySQL instance? Nah. But it’s a good learning experience even if I never grow beyond one virtual machine serving everything. And if I need to, I’ll be ready to in a hurry.

At first, I was using Amazon’s SimpleDB and the boto package for Python. It was working great! Early on I’d made a brief attempt to verify that it could hold enough data per row to do what I was doing. But I failed to Google the right thing and so just put it off until later. I wrote everything such that I could replace it with a different storage engine pretty easily.

Then a few days ago I implemented chat. That worked great too! Until the first time I added a chat line that pushed the total chat for one game over 1K bytes. BOOM, that’s an end of the world event for SimpleDB. It turns out it has a limit of 1K bytes for any individual piece of data, and also a limit of 256 keys per row. I am also saving the complete game histories and moves for each player, three keys per turn, though it could be squished down to one key per turn as each turn is completed. I had a plan to work around the 256 keys limit, because I thought a limit like that might come up, but the 1K limit was harder. I could’ve still come up with something, either offloading the chat to Amazon S3 (that’s how they really want you to do things anyway), or breaking it up into chunks somehow, but none of that was very appealing.

So I googled something like “distributed database software” and found Cassandra at the first hit. Oh yeah, Cassandra, I’d looked at it before but never used it for anything. It doesn’t have any meaningful limits on size for my purposes (there are limits, but they’re measured in billions). It has a Python binding. This will be fun! I downloaded it, pretty easily got it up and running, and started rewriting my storage class. At the point where I was saving games, I realized I had a race condition with both players trying to save at the same time. That would be quite common, the turns in this game are simultaneously executed and if both players are online at the same time, they will try to submit a save within seconds of each other, depending how I have the polling set. To be fair, I had the same race condition with SimpleDB, and was aware of it, but I hadn’t even gotten around to looking at how to fix it.

Next, I Google “Cassandra row locking”. There isn’t any. Nice as it would be, it’s understandably beyond the Cassandra project’s scope. But I did find someone talking about using something called ZooKeeper to implement a distributed locking system on top of Cassandra. Look around, see there’s a python binding for ZooKeeper, ok, I should be in business, let’s get it going

A little while later, I’ve got a ZooKeeper instance up and running. Go back to that link, figure out he was actually talking about using something called Cages, which uses ZooKeeper itself, but isn’t part of ZooKeeper. Cages is a Java library. There is not a Python equivalent. The python binding for ZooKeeper just wraps the C API, and while it’s possible to implement locking using ZooKeeper, it doesn’t provide the simple mutex/critical section type lock I’m locking for out of the box.

At this point I’m starting to question whether I really wanted to try out all this stuff or whether I should just hang up the towel and put everything in MySQL after all. But no, I’ve come this far, I’m not stopping now!

Poking around Cages a little bit, I find it’s quite large, very Java-y, and implements a whole bunch of stuff I don’t really need (at least not right now). After some more googling, I found some simpler ZooKeeper lock implementations, still all in Java, but managed to muddle through them enough to understand how it can be used to implement my simple little lock. This ClusterMutex was especially helpful.

So finally, I wound up implementing my own ZooKeeper lock class for Python. Which also caused me to create a github account to have somewhere public to put it, because if it’s useful to me then hopefully it will be useful to someone else too. And also a PyPI account so I could upload it as a Python pip (or easy_install) package. So now anyone can get the zklock package and start using it as easily as “pip install zklock” (Well, sort of, it needs the zkpython package also, which in turns the ZooKeeper C library installed first. And of course, you need a ZooKeeper instance somewhere to use any of that.) Which is all great, I haven’t contributed any open source software anywhere for a long time. I’m happy to have an excuse to set up those accounts.

Now inserting chat messages is this simple:

def append_text(game_id, key, text):
    # Lock the game
    l = zklock.Lock(item_name(game_id))
        # See if the text key exists already
        cols = game_fam.get(item_name(game_id), [key])
        existing = cols[key]
    except pycassa.NotFoundException:
        # Nope, start with a blank one
        existing = ''
    # Create the entry
    new = {key:existing + text}
    game_fam.insert(item_name(game_id), new)
    # Release the game lock
    # Return the new column
    return new

All of this is basically so I can have chat in my silly iPhone game. It’s completely overkill (though it would be nice if it turned out not to be!), and there are a thousand other ways I could’ve gone about implementing this, many of them much simpler, but I learned a bunch of new stuff along the way, and hopefully created something someone else will find useful. A worthwhile day and a half’s work for sure.

Comments (4)

Road Trip: Day 6: Light Show

Today was the last day in the mountains. I drove up through Taos, and on to the Great Plains. My first real stop was a long dormant volcano whose name I already forgot because I left the brochure in the car. It’ll come to me as soon as I click publish on this. I got there too late to hike around the rim. It would’ve taken maybe half an hour, but I didn’t get to the parking lot at the top until ten minutes before the park closed for the night. Oh well, at least I got to go up. On the way down, I could see lightning off in the distance. I took some video and did capture one strike. Click on any of these for full size versions.

Later while driving, there were some strikes off to my left, so I put the camera on burst mode and pointed it out my window for a while. I got this one that way.

Finally, when I got to my hotel room in Guymon, OK, the light show was in full swing, and went on for close to an hour. I got several good shots, here are the best couple.

Comments (9)

Road Trip Day 4 & 5: Grand Canyon, Page, Santa Fe

Before leaving Zion, I visited the Mean Bean coffee house for coffee and a waffle, both of which were excellent. So far I’m keeping my don’t eat at the same place twice guideline. From Zion, I drove to Page, Arizona. The notable stop along the way was the Grand Canyon’s north rim. It’s an amazing view, of course, and all the more so when you find out that reaching the other side, which is clearly visible, by car, is a 200 mile drive.

I opted not to do the 200 miles, and instead headed to Page, which sits on Lake Powell. I tried to go out and see Horseshoe Bend on my way out this morning, but the road was temporarily closed for something, the workman said 15 minutes, but when I came back 15 minutes later after getting gas it was still closed and traffic was backing up. So I gave up on that idea. That was disappointing. There was a picture of it on the wall of my hotel room which I thought was a picture of the canyon taken with a fisheye lens. It wasn’t until I was looking at stuff to see online later that I realized it actually looks that way. Oh well, some other trip!

The picture I would have taken if I’d had more patience.

Today’s main attraction was Four Corners, where Utah, Colorado, Nevada and Arizona meet. It’s the only place in the U.S. where four states meet. The site itself is operated by the Navajo Nation, and surrounded by jewelry sellers and other touristy souvenir stuff. I walked to Colorado and bought a magnet for our door at home. You should all bug @lnorigb to post some pictures of our magnet door before all the magnets pull the space station down on top of us. It’s going to happen someday, especially after all the magnets I’m adding from this trip. Walking around the monument to Colorado “the long way”, I walked through as many states as I did on my AT hike. Today was slightly easier.

Also today was the Best Buy in Farmington where I got a replacement hard drive for my ailing laptop. It’s being restored from my external drive right now. Bombich Software, makers of the excellent and free Carbon Copy Cloner are getting a donation out of this, it definitely “saved my bacon” as they put it.

Farmington has this interesting builiding in its old downtown section.

It might be hard to see it in the picture even if you click for the full sized version, but there are swastikas all around the top of it. It’s not subtle at all when you’re standing across the street from it. It’s clearly an old building, and I know swastikas were sometimes used that way prior to WW2, but it’s really surprising it’s been left that way all these years! If you can read the sign, you’ll see that Tony Bennett took up photography and set up shop in the building.

I had planned on going over the mountains and staying in Los Alamos tonight, but when I got to the road I would’ve taken, there were signs indicating it was closed due to fires 30 miles ahead. I had seen the smoke before that, but hadn’t put it together that that was where Los Alamos was, I wasn’t really headed towards it at that point. Checking the Internet, it was immediately obvious I wasn’t going to Los Alamos. The whole town of 12,000 has been evacuated. So I wound up taking the interstate (drat!) to Santa Fe instead, where I found a hotel room on my second call only because I was willing to take the room with a broken TV. But I feel guilty about being here when there are probably evacuees who could’ve used it. There are a lot of hotels here, but I doubt they’re enough to absorb the entire town of Los Alamos! If I’d thought about it a little more before heading out here, I would’ve either gone to Albuquerque or headed out past Santa Fe to Taos or somewhere. I just hope everyone from Los Alamos stays ok.

Location:Fox Rd,Santa Fe,United States

Comments (4)

Road Trip Day 3: Zion

Arrived at the Driftwood Lodge in the afternoon. The town of Springdale sits just outside the entrance to Zion Canyon National Park and seems to have a thriving tourist industry. The park runs free shuttles up and down the main street and into the canyon (a separate shuttle) every 10 minutes. Given how many people were visiting, traffic must have been a nightmare before they started the shuttle service in 2000.

I barely scratched the surface of Zion riding the shuttle and doing a barely two mile hike up to Emerald Pools. It’s definitely somewhere one could spend several days.

Emerald Pools.

Looking down the canyon.

Waiting at the tunnel on the way out of the park.


Road Trip: Days 1-2.5

I’ve always wanted to watch a Space Shuttle launch. There’s only one more, ever. So I’m on my way to Florida, the long way. I don’t have a schedule except the family will join me at Disney World on July 4th.

I do have some guidelines though.

  • No Interstates – try to take the old highways when I can.
  • Never eat at the same place twice. For example, McDonald’s is a place, not one specific McDonald’s. So I can eat at chains, but only once per. I’ll have to try the local places. I’m writing this from a 50s Diner in Mesquite, Nevada. It feels authentic to me in that the one adjective that comes to mind when I think of food from the 50s is “boring”. If I decide this rule applies to Starbucks too, I might be in big trouble.
  • If I see any roadside attractions proclaiming “World’s Adjectivest Noun”, I have to stop there. Like the World’s Biggest Ball of Twine. Or the World’s Tiniest Violin.

    They’re guidelines, not rules. The no interstates rule is pretty hard to follow completely in the southwest, but I think other than the stretch of I15 I’m on now from Vegas to Zion in Utah, and a little bit of I40, I can do it.

    So far I’ve stopped at the Palm Springs tram and then spent a couple of nights in Vegas. Tonight I’ll be at Zion National Park, and then the Grand Canyon tomorrow. Not really enough time for either, probably, just enough for some quick pictures and then back on the road.

    I was going to add pictures from the tram to this post before posting, but I’d already gotten them off my phone onto my laptop. And then my laptop’s hard drive died. Completely. I have a spare bootable external drive that was cloned just a few days ago. That works fine. The only thing I’ll lose at all besides time fixing it are those pictures. So it’s really just annoying. But boy do I feel smart for having that extra drive! I’ve never travelled with one before. Now I always will.

    Location:N Sandhill Blvd,Mesquite,United States

  • Comments

    Off the Trail

    Yes, for real. I made the decision to get off the trail a few days ago. I’m at tge airport now waiting for my flight home. I’ve been hanging around to make sure I got a proper goodbye with my good friends Shadow, Country Mouse and Flute Walker. I picked them up near Waynesboro yesterday and we all went out for one last AYCE hiker meal (last for me anyway, they’re all still going!) and ice cream. They are friends for life at this point and I hope they make it to the end. If you’re looking for trail updates, you can follow their continuing journey at I have promised to leave them snarky comments too. If the timing works out, I’d love to come back and hike up Katahdin with them in the fall too. But other than that my hike is over.

    As for why, it just reached the point where I felt like I wasn’t getting anything out of it any more. On top of that, it’s getting hotter, and buggier. The bugs seemed to like me an awful lot; there wasn’t going to be a day I didn’t itch until it got cool again up north. I’ll miss my friends a lot, but I’m not going to miss the heat, the bugs, the sore feet, the weight on my back, or a whole bunch of other little annoyances.

    I hiked 800 miles and I’m glad I did it. I would do it again if I had it to do over. Maybe I’ll even make it back someday and do the rest. Maybe I won’t. Either way it was a great experience.

    What next? I don’t know! I’m still planning on taking the summer off of work. But I’ll spend it with Lnorigb at home or somewhere else nice instead of with the mosquitos out in the woods.

    Thanks for reading the blog. I’ll miss writing daily updates here. Writing here was one of my favorite parts of the trip.

    Location:Sayward Blvd,Herndon,United States

    Comments (4)