Down the NoSQL Rabbit Hole

Although I’m going back to work in a little over a week, I’ve been working on an iPhone game in the meantime. More on that some other time (soon!) This game has an online turn-based component, kind of like Scrabble (Also Words With Friends), Carcassone, Disc Drivin’ and other great iPhone games. So besides the iPhone front end, I’m writing a Django back-end since I’m already familiar with Django and it’s nice for getting things done cleanly and quickly.

Of course, the back end needs a back end too, so rather than just shoving everything in MySQL, I decided I’d try out newer, more scalable, stuff just in case I happen to write the next Angry Birds. Am I likely to need more than a single server MySQL instance? Nah. But it’s a good learning experience even if I never grow beyond one virtual machine serving everything. And if I need to, I’ll be ready to in a hurry.

At first, I was using Amazon’s SimpleDB and the boto package for Python. It was working great! Early on I’d made a brief attempt to verify that it could hold enough data per row to do what I was doing. But I failed to Google the right thing and so just put it off until later. I wrote everything such that I could replace it with a different storage engine pretty easily.

Then a few days ago I implemented chat. That worked great too! Until the first time I added a chat line that pushed the total chat for one game over 1K bytes. BOOM, that’s an end of the world event for SimpleDB. It turns out it has a limit of 1K bytes for any individual piece of data, and also a limit of 256 keys per row. I am also saving the complete game histories and moves for each player, three keys per turn, though it could be squished down to one key per turn as each turn is completed. I had a plan to work around the 256 keys limit, because I thought a limit like that might come up, but the 1K limit was harder. I could’ve still come up with something, either offloading the chat to Amazon S3 (that’s how they really want you to do things anyway), or breaking it up into chunks somehow, but none of that was very appealing.

So I googled something like “distributed database software” and found Cassandra at the first hit. Oh yeah, Cassandra, I’d looked at it before but never used it for anything. It doesn’t have any meaningful limits on size for my purposes (there are limits, but they’re measured in billions). It has a Python binding. This will be fun! I downloaded it, pretty easily got it up and running, and started rewriting my storage class. At the point where I was saving games, I realized I had a race condition with both players trying to save at the same time. That would be quite common, the turns in this game are simultaneously executed and if both players are online at the same time, they will try to submit a save within seconds of each other, depending how I have the polling set. To be fair, I had the same race condition with SimpleDB, and was aware of it, but I hadn’t even gotten around to looking at how to fix it.

Next, I Google “Cassandra row locking”. There isn’t any. Nice as it would be, it’s understandably beyond the Cassandra project’s scope. But I did find someone talking about using something called ZooKeeper to implement a distributed locking system on top of Cassandra. Look around, see there’s a python binding for ZooKeeper, ok, I should be in business, let’s get it going

A little while later, I’ve got a ZooKeeper instance up and running. Go back to that link, figure out he was actually talking about using something called Cages, which uses ZooKeeper itself, but isn’t part of ZooKeeper. Cages is a Java library. There is not a Python equivalent. The python binding for ZooKeeper just wraps the C API, and while it’s possible to implement locking using ZooKeeper, it doesn’t provide the simple mutex/critical section type lock I’m locking for out of the box.

At this point I’m starting to question whether I really wanted to try out all this stuff or whether I should just hang up the towel and put everything in MySQL after all. But no, I’ve come this far, I’m not stopping now!

Poking around Cages a little bit, I find it’s quite large, very Java-y, and implements a whole bunch of stuff I don’t really need (at least not right now). After some more googling, I found some simpler ZooKeeper lock implementations, still all in Java, but managed to muddle through them enough to understand how it can be used to implement my simple little lock. This ClusterMutex was especially helpful.

So finally, I wound up implementing my own ZooKeeper lock class for Python. Which also caused me to create a github account to have somewhere public to put it, because if it’s useful to me then hopefully it will be useful to someone else too. And also a PyPI account so I could upload it as a Python pip (or easy_install) package. So now anyone can get the zklock package and start using it as easily as “pip install zklock” (Well, sort of, it needs the zkpython package also, which in turns the ZooKeeper C library installed first. And of course, you need a ZooKeeper instance somewhere to use any of that.) Which is all great, I haven’t contributed any open source software anywhere for a long time. I’m happy to have an excuse to set up those accounts.

Now inserting chat messages is this simple:

def append_text(game_id, key, text):
    # Lock the game
    l = zklock.Lock(item_name(game_id))
        # See if the text key exists already
        cols = game_fam.get(item_name(game_id), [key])
        existing = cols[key]
    except pycassa.NotFoundException:
        # Nope, start with a blank one
        existing = ''
    # Create the entry
    new = {key:existing + text}
    game_fam.insert(item_name(game_id), new)
    # Release the game lock
    # Return the new column
    return new

All of this is basically so I can have chat in my silly iPhone game. It’s completely overkill (though it would be nice if it turned out not to be!), and there are a thousand other ways I could’ve gone about implementing this, many of them much simpler, but I learned a bunch of new stuff along the way, and hopefully created something someone else will find useful. A worthwhile day and a half’s work for sure.

Tags: , ,

Thursday, September 8th, 2011 Uncategorized 1 Comment

I Made Asteroids

AsteroidI stayed home sick today, and made a little browser asteroids game on my laptop.  The cool part is, it doesn’t need Flash or any other plugin.  It’s all JavaScript.  The even cooler part is, I am still blissfully ignorant about JavaScript, I wrote the whole thing in Python, using Pyjamas.  I’ve tested it on Firefox and Safari, works great on both, and runs super smooth even with lots of asteroids.

It’s not much of a game, really, just classic Asteroids without even scores or keeping track of ships or much of anything but the basics.  But it’s only about 300 lines of Python, and it runs in any off the shelf browser.

I’ll try to add some polish to it and get the source up somewhere soon for anyone interested.  I’m pretty happy I managed to do all that in a day, plus a few hours on previous days getting Pyjamas set up.

Update (11/19): I uploaded the source code. It’s not the best code I’ve ever written, and it’s not commented at all, but now it’s a real project!  Sort of.

Update (11/20): Version 0.2 is now live, with sound effects, a score, and a different Canvas object that allows it to run in IE6.  It’s too slow to be playable in IE6, but at least it runs.

Tags: , , ,

Wednesday, November 18th, 2009 Uncategorized 14 Comments

There’s no sense crying over every mistake

You just keep on trying till you run out of cake.

Tags: , , ,

Tuesday, October 23rd, 2007 Uncategorized 1 Comment

The rules are different here

Little known fact: in most parts of Orange County, fire is harmless. You can sit right in the middle of a roaring bonfire on the beach and not feel a thing. But the rules are different in Santiago Canyon. When traveling through that zone, fire can and will hurt you. Luckily, the server displays the following message when this rule change takes effect.


Fire is hazardous from that point on.

Tags: , , ,

Sunday, August 19th, 2007 Uncategorized No Comments

Speaking at GDC

I’ll be on a round table discussion on “Taming Online Scaling Issues” at the Austin Game Developer’s Conference, September 5th through the 7th. My session is on the 6th. Since it’s just a round table, I’m only talking for a few minutes, as are the other participants, and then it’s an open discussion for anyone that shows up.

Stop by and join in, or just say hi, if you’re around.

Taming online scaling issues is closely related to taming lions. Whips and wodden chairs are involved, and every once in a while someone suffers a disfiguring accident.

Tags: , ,

Monday, July 30th, 2007 Uncategorized No Comments

Fun with language filters

All MMOs have profanity filters. Usually optional, but always available.

This week I had to look at a ticket about a particular player who was forced to change his name for “no apparent reason.” It turns out his name, his whole name, not a substring, had been added to the profanity filter in the most recent patch, which will cause an automatic rename. The closest translation of his name (it was German) appears to be “rascal”, but obviously more derogatory if someone put it in the profanity list. I’d link you the google translation of the German wikipedia page, but I don’t want to cause the player any grief by posting his (old) name.

Adding actual substring matching to profanity filters is usually a bad idea. You might think there are certain words that are always bad even if they only match substrings. You might even go and add them to the list.

Until your game winds up filtering out “Brightwater Lake”

That happened in beta. Look closely.

Also, the forums used to filter out cockroach.

Tags: ,

Friday, May 25th, 2007 Uncategorized 1 Comment

Guitar Hero 2

The XBox 360 version of Guitar Hero 2 is pure evil. I never went for high scores much on the PS2 version. The leaderboard has completely changed all that, and is sucking up way too much of my time. I am ranked in the top 5000 on career score as of this moment. My tag is Ogre36, feel free to send a friend request.

I only have a single song where my high score is on expert too. The rest are all medium or hard (mostly hard until the last couple of sets). So lots of room for improvement, but going for score on expert is a LOT harder than hard. Just passing the songs isn’t, but keeping streaks going is way more difficult.


Friday, April 20th, 2007 Uncategorized No Comments

Puzzle Quest Solver

I figure it’s not cheating when I use this program to solve Puzzle Quest capture puzzles. But if you use it, you’re totally cheating. I wrote the code, that means I solved the puzzle. You, you’re scum.
pqsolver.exe (Get this if you just want to run it)

pqsolver.cpp (Source code, if you want to see how it works or don’t trust random executables off the internet. Builds on Mac, Linux, Windows, probably anything else with a C++ compiler)

Sample input files:

test2.pq (not a real puzzle, just an early test)

OrcLord.pq (Real)

Sandworm.pq (Real)

To run it:

pqsolver < file.pq

The magic values, straight from the code:
case red: c = 'r'; break;
case yellow: c = 'y'; break;
case green: c = 'g'; break;
case blue: c = 'b'; break;
case skull: c = 's'; break;
case skullx5: c = '5'; break;
case xp: c = 'x'; break;
case money: c = 'm'; break;
case empty: c = '.'; break;

The output has the first step at the bottom, last step at the top. Backwards, in other words. Harder to do it the right way around the way the algorithm works. The bottommost board displayed is the initial input, the topmost is the next to last state before winning. Above each board are the coordinates to move to get to the next step.

Updated 11/09/2007 : Thanks to Nige in the comments, the bug with skulls is fixed.  You Rock, Nige!

Tags: , ,

Wednesday, April 4th, 2007 Uncategorized 24 Comments