Programming

GDC 2013 Slides

 

I uploaded the slides from my GDC talk, “Network Serialization and Routing in World of Warcraft”.  You can download them in several formats:

  • Keynote What I used to build and present them
  • PowerPoint Auto-exported from Keynote, haven’t checked it AT ALL.  Likely has a few broken things
  • PDF – maybe your best bet to just read them

Also, Simon Koo (@sm9kr) seems to have written a summary of the talk in Korean along with pictures of all or most of the slides.  I’m taking his word for it that the text is actually about my talk!

FlashForwardSlide.001

Tags: , , , ,

Friday, March 29th, 2013 Uncategorized 1 Comment

Down the NoSQL Rabbit Hole

Although I’m going back to work in a little over a week, I’ve been working on an iPhone game in the meantime. More on that some other time (soon!) This game has an online turn-based component, kind of like Scrabble (Also Words With Friends), Carcassone, Disc Drivin’ and other great iPhone games. So besides the iPhone front end, I’m writing a Django back-end since I’m already familiar with Django and it’s nice for getting things done cleanly and quickly.

Of course, the back end needs a back end too, so rather than just shoving everything in MySQL, I decided I’d try out newer, more scalable, stuff just in case I happen to write the next Angry Birds. Am I likely to need more than a single server MySQL instance? Nah. But it’s a good learning experience even if I never grow beyond one virtual machine serving everything. And if I need to, I’ll be ready to in a hurry.

At first, I was using Amazon’s SimpleDB and the boto package for Python. It was working great! Early on I’d made a brief attempt to verify that it could hold enough data per row to do what I was doing. But I failed to Google the right thing and so just put it off until later. I wrote everything such that I could replace it with a different storage engine pretty easily.

Then a few days ago I implemented chat. That worked great too! Until the first time I added a chat line that pushed the total chat for one game over 1K bytes. BOOM, that’s an end of the world event for SimpleDB. It turns out it has a limit of 1K bytes for any individual piece of data, and also a limit of 256 keys per row. I am also saving the complete game histories and moves for each player, three keys per turn, though it could be squished down to one key per turn as each turn is completed. I had a plan to work around the 256 keys limit, because I thought a limit like that might come up, but the 1K limit was harder. I could’ve still come up with something, either offloading the chat to Amazon S3 (that’s how they really want you to do things anyway), or breaking it up into chunks somehow, but none of that was very appealing.

So I googled something like “distributed database software” and found Cassandra at the first hit. Oh yeah, Cassandra, I’d looked at it before but never used it for anything. It doesn’t have any meaningful limits on size for my purposes (there are limits, but they’re measured in billions). It has a Python binding. This will be fun! I downloaded it, pretty easily got it up and running, and started rewriting my storage class. At the point where I was saving games, I realized I had a race condition with both players trying to save at the same time. That would be quite common, the turns in this game are simultaneously executed and if both players are online at the same time, they will try to submit a save within seconds of each other, depending how I have the polling set. To be fair, I had the same race condition with SimpleDB, and was aware of it, but I hadn’t even gotten around to looking at how to fix it.

Next, I Google “Cassandra row locking”. There isn’t any. Nice as it would be, it’s understandably beyond the Cassandra project’s scope. But I did find someone talking about using something called ZooKeeper to implement a distributed locking system on top of Cassandra. Look around, see there’s a python binding for ZooKeeper, ok, I should be in business, let’s get it going

A little while later, I’ve got a ZooKeeper instance up and running. Go back to that link, figure out he was actually talking about using something called Cages, which uses ZooKeeper itself, but isn’t part of ZooKeeper. Cages is a Java library. There is not a Python equivalent. The python binding for ZooKeeper just wraps the C API, and while it’s possible to implement locking using ZooKeeper, it doesn’t provide the simple mutex/critical section type lock I’m locking for out of the box.

At this point I’m starting to question whether I really wanted to try out all this stuff or whether I should just hang up the towel and put everything in MySQL after all. But no, I’ve come this far, I’m not stopping now!

Poking around Cages a little bit, I find it’s quite large, very Java-y, and implements a whole bunch of stuff I don’t really need (at least not right now). After some more googling, I found some simpler ZooKeeper lock implementations, still all in Java, but managed to muddle through them enough to understand how it can be used to implement my simple little lock. This ClusterMutex was especially helpful.

So finally, I wound up implementing my own ZooKeeper lock class for Python. Which also caused me to create a github account to have somewhere public to put it, because if it’s useful to me then hopefully it will be useful to someone else too. And also a PyPI account so I could upload it as a Python pip (or easy_install) package. So now anyone can get the zklock package and start using it as easily as “pip install zklock” (Well, sort of, it needs the zkpython package also, which in turns the ZooKeeper C library installed first. And of course, you need a ZooKeeper instance somewhere to use any of that.) Which is all great, I haven’t contributed any open source software anywhere for a long time. I’m happy to have an excuse to set up those accounts.

Now inserting chat messages is this simple:

def append_text(game_id, key, text):
    # Lock the game
    l = zklock.Lock(item_name(game_id))
    l.acquire()
    try:
        # See if the text key exists already
        cols = game_fam.get(item_name(game_id), [key])
        existing = cols[key]
    except pycassa.NotFoundException:
        # Nope, start with a blank one
        existing = ''
 
    # Create the entry
    new = {key:existing + text}
    game_fam.insert(item_name(game_id), new)
 
    # Release the game lock
    l.release()
 
    # Return the new column
    return new

All of this is basically so I can have chat in my silly iPhone game. It’s completely overkill (though it would be nice if it turned out not to be!), and there are a thousand other ways I could’ve gone about implementing this, many of them much simpler, but I learned a bunch of new stuff along the way, and hopefully created something someone else will find useful. A worthwhile day and a half’s work for sure.

Tags: , ,

Thursday, September 8th, 2011 Uncategorized 1 Comment

iTunes Expressions

iTunes smart playlists are actually fairly complex boolean expressions (since iTunes 9, previous versions were not as robust). This page describing iTunes library management contains more text than the entire Lua 3.0 manual! (I’m cheating here, Lua is on v5.1, and the manual is more than twice as large these days, but the point still stands). Via dropdowns and text boxes you’re practically writing SQL WHERE clauses when creating a smart playlist.

Yet for all that complexity, there is STILL no way built into iTunes to make a Smart Playlist that can precisely select between High-def and standard-def TV Shows and movies.

My compromise is making a playlist that selects shows between 20 and 31 minutes that are greater than 500MB, and shows 39 minutes or longer that are greater than 900MB. Those numbers aren’t very precisely chosen, but they seemed to work properly for all but one episode in my library, there was one hour long episode that says it’s “HD” but is only 600MB, I ignored that one, but it illustrates why this is such a bad solution. There are standard def shows that are nearly 600MB in my library too.

The compromise some people come to is manually adding an HD Tag to the composer, description, comments, or some other field, but the whole point here is that I’m trying to avoid manually doing anything to differentiate them. I could just as easily add all the HD videos to a dumb playlist, and then select on that playlist in a smart playlist.

I’ve had this frustration for at least a year, since we first got an Apple TV. When, when, when will Apple fix it? Or if they have, then when, when, when someone on the internet figure out how to do it?

Since the thing I wanted this playlist for is already an Applescript that’s iterating TV Shows from a playlist, I suppose the “solution” in this case is to make the Applescript able to differentiate. The filename actually has an “(HD)” in it for HD episodes, and Applescript does have access to filenames, but iTunes proper can’t select on filename contents.

On the other hand, the horror of having to actually write more than a few lines of Applescript caused me to write just enough to be able to call out to a python script instead. So there’s a whole ‘nother rant about crappy Apple “programming languages” right there. I do realize that Python can use Applescript interfaces directly, but then it’s harder to just Download a script that almost does what I want and fix the parts that don’t.

Tags: , , , , ,

Sunday, May 16th, 2010 Uncategorized No Comments

Movie encoder “daemon”

I sometimes have incoming high-def video in formats that are not directly compatible with AppleTV.  I had been using iFlicks and an El Gato Turbo.264 HD encoder (a USB dongle with H.264 hardware innit), but lately that started producing bad output – I think the hardware was going bad as even the official software had the same results.  It was great while it lasted, it allowed the poor overworked Mac Mini that was doing the encoding to do 720p transcodes faster than real-time.  I should note that iFlicks’ author has actually recommended not using the El Gato thing, he mentions audio sync problems, but I never saw that, just frequent badly encoded video after mine started to go bad.

I’ve now switched back to software encoding using Handbrake.  To that end, I wrote a tiny little daemon that watches for files in one directory, encodes them, then moves them to the folder that iFlicks is watching and trashes the original.  iFlicks will then add meta-data (it searches thetvdb.com based on the file name), and finally send the file off to iTunes (which will then sync it to AppleTV.  WHEW that’s a lot of steps, you see why I need to write myself things to help?)

iFlicks can do software encoding itself, but it just uses Quicktime, which is a whole hell of a lot slower than Handbrake. Handbrake still doesn’t do 720p at real-time on a Mac Mini, but it’s only about half real-time, as opposed to Quicktime, which was taking 4-8 hours to encode an hour of video.

The script also gives me Growl notifications, and I have the Prowl app on my phone, so I get push notifications when encodings start and finish too.

Code after the jump.
› Continue reading

Tags: , , , , ,

Sunday, February 28th, 2010 Uncategorized No Comments

Kindle DRM Removal Using SCons

I have a Kindle. I like it a lot, but it has some issues. The recent Macmillan vs. Amazon fight is one. Another is the DRM on every file purchased from Amazon. If I buy a book, I want to know that I can read it forever. Luckily, smart people figured out how to break the DRM on most Kindle books a long time ago. A few books are in another format called Topaz that until recently hadn’t been broken. I just found a book I wanted that was in that format, so I went and looked again, and it turns out someone finally cracked it about a month ago.

I’m not going to link to any of the actual tools here, but they shouldn’t be too hard to find. My contribution is an SConstruct file for the SCons build system that automatically copies all my books to my computer and removes the DRM all in one step.  It’s not your typical use of a build system, but when I thought about what I was trying to do, I realized it was really just a simple dependency graph, something SCons is perfect for.

The complete process is:

  1. Plug Kindle into computer via USB
  2. In a shell, cd <path/where/this/file/lives>
  3. scons

This will automatically get any new books, and remove the DRM (and in Topaz’s case, convert them into svg files viewable in a browser)

Installation is not difficult either:

  1. Install SCons
  2. Put this SConstruct file in the directory where you want to save your books
  3. Get your Kindle’s PID and put it in the SConstruct file (the DRM tools will tell you how to do that)
  4. Edit the location your Kindle gets mounted to. The one I have in there is for a Mac, but except for that path I don’t see any reason this won’t all work on any OS
  5. Put the DRM tools in a tools subdirectory

SConstruct file after the jump
› Continue reading

Tags: , , , , , ,

Sunday, February 21st, 2010 Uncategorized 3 Comments

How do I love thee? Let me count the days.

Lnorigb‘s been bugging me to give her a thing in her calendar that tells her what day number each of her projects is on, so that her blog entries can be accurate. I finally wrote up this crappy little python script to do it. Requires the icalendar python package (easy_install icalendar). I just set up a file like:

Towner
90
2009/11/18
2009/12/25
2010/01/01

Which tells my script to create an iCal event every day for 90 work days named “Towner, Day ##”, and don’t count Christmas or New Year’s day as work days.

Not shown is the bit at the end that uploads the resulting .ics file to a web server, which then allows Lnorigb to simply subscribe to it in iCal. So if I make any changes in the output, like skipped days, increasing or decreasing the total count, or just general improvements, her calendar will automatically reflect the changes.

This is isn’t the best code I ever wrote, but it gave me an excuse to put a syntax highlighting plugin on the blog. And it’s not like I’m expecting to have to do a lot of maintenance work on it. Of course now that I’ve said that, it’s obviously going to cause me grief for many years. Eventually I suppose it will need a full fledged scheduling application, complete with payment calculators for her workers based on facial recogonition of the posted pictures on her blog, auto-blog posting, twitter updates, a related facebook application, and RSS feed generators. All of the above will be driven by the nine million state version of the stupid little state machine parser at the top of the script.

Yeah, don’t write code you’re not willing to maintain.

(Code follows after the break.)

› Continue reading

Tags: , , , ,

Monday, February 8th, 2010 Uncategorized No Comments

I Made Asteroids

AsteroidI stayed home sick today, and made a little browser asteroids game on my laptop.  The cool part is, it doesn’t need Flash or any other plugin.  It’s all JavaScript.  The even cooler part is, I am still blissfully ignorant about JavaScript, I wrote the whole thing in Python, using Pyjamas.  I’ve tested it on Firefox and Safari, works great on both, and runs super smooth even with lots of asteroids.

It’s not much of a game, really, just classic Asteroids without even scores or keeping track of ships or much of anything but the basics.  But it’s only about 300 lines of Python, and it runs in any off the shelf browser.

I’ll try to add some polish to it and get the source up somewhere soon for anyone interested.  I’m pretty happy I managed to do all that in a day, plus a few hours on previous days getting Pyjamas set up.

Update (11/19): I uploaded the source code. It’s not the best code I’ve ever written, and it’s not commented at all, but now it’s a real project!  Sort of.

Update (11/20): Version 0.2 is now live, with sound effects, a score, and a different Canvas object that allows it to run in IE6.  It’s too slow to be playable in IE6, but at least it runs.

Tags: , , ,

Wednesday, November 18th, 2009 Uncategorized 13 Comments

Language Benchmarks: Why Ruby Won’t Win

Benchmarking programming language performance is an imprecise exercise, but the Computer Language Benchmarks Game makes a pretty good go at it.  Ruby is next to last on the overall scores, and loses on pretty much every test to Python and Lua, the current kings of game scripting in general.  I still think Ruby is a fantastic language and wholly appropriate for a lot of tasks.  But general performance is a major consideration for use as an embedded game scripting language.  Until they get Ruby into the same performance class as Python and Lua, it’s not going to be a viable option for what I’ve been looking at it for.

Tags: ,

Thursday, June 21st, 2007 Uncategorized No Comments

Late to the party

I wrote a little program in Ruby (and I do mean little, Ruby continues to impress me with ease of getting things done) to get a semi-random picture from Flickr (checks for anything new from Stuck in Customs first, then gets a random picture tagged with “landscape” if there aren’t any), then set it as my desktop.  I set it to run every half hour as a cron job.  Then I set up GeekTool to get the current picture displayed (the ruby program writes it to a log file, I have GeekTool get the last line of that file) and display the title and user at the lower right of my desktop.

I figured there would already be several other programs way cooler than mine, but if there weren’t I figured I’d package it up with a few more options and release it.  Turns out, you can already do most of that without downloading any software at all.  You can subscribe to photo RSS feeds in iPhoto (news to me!) and you can then use photos from those feeds as desktop backgrounds directly from the Desktop preferences panel.  OS X and its packaged applications are just really impressive sometimes.

On the other hand, the randomly select a picture every X minutes feature doesn’t work with photo RSS feeds, so it doesn’t actually do everything my little program does.  I bet Leopard will.  But I’ve got at least a narrow window where I can release an app that does it and maybe someone will actually want it 🙂   Plus I can be a little more flexible – I don’t think you can do the trick I’m doing where a particular user’s pictures always go up when there’s a new one, but that depends more on how sophisticated you can get with Flickr’s RSS feeds.  I haven’t really looked at that.

It’s all mainly for my own education anyway, and I think I’m going to use it as an excuse to learn about RubyCocoa  and then maybe release something when I have a “real” standalone Mac app that normal Mac users can use.  (IE Requiring crontab editing is RIGHT OUT.) But if anyone reads this and wants the ruby script sooner than that, send me a note.  It was fun to write and fun to get all these great pictures on my desktop all the time!

Tags: , , ,

Monday, June 4th, 2007 Uncategorized 2 Comments

LOLCODE

Since I’ve been looking at languages for embedding/scripting purposes lately, I naturally came across LOLCODE. I’d been leaning towards Ruby as the overall most elegant, cleanest, robust language I could find, but how can you beat this?

HAI
CAN HAS STDIO?
I HAS A VAR
GIMMEH VAR
IZ VAR BIGGER THAN 10 O RLY?
	YA RLY
		BTW this is true
		VISIBLE "BIG NUMBER!"
	NO WAI
		BTW this is false
		VISIBLE "LITTLE NUMBER!"
	KTHX
KTHXBYE

				
				

Tags: ,

Wednesday, May 30th, 2007 Uncategorized No Comments