Archive for category trivia

The 2014 MIT Mystery Hunt – Alice Shrugged

Winning the hunt in 2013

Winning the hunt in 2013

Calling the winning team in 2014

Calling the winning team in 2014

So we ran the MIT Mystery Hunt this year (our dubious award for winning it last year). The experience is pretty well bookended by the above two pictures: one of Laura answering our phone in 2013 to hear that we had answered the final meta successfully and won, and another one of Laura calling the winning team in 2014 (One fish, two fish, random fish, blue fish) to congratulate them on answering the final meta successfully. I have no idea how or where to begin describing what it was like to do this this year. My team ran the hunt in 2004, but I was out of town in Champaign, IL at the time and played no part in that little misadventure. This time around, I was on the leadership committee, in charge of hunt systems, IT and infrastructure.

Thank You

First I’d like to thank the other members of the systems team. James Clark just about single-handedly wrote a new django app and framework (based loosely on techniques and code from the 2012 codex hunt) which we will be putting up on github soonish and we hope that other teams can make use of it in the future — we have dubbed it “spoilr”. It worked remarkably well, and has several innovative features that I think will serve hunt well for years to come. Joel Miller and Alejandro Sedeño were my on-site server admin helpers and helped keep things running and further adjusted code (although only slightly) during the hunt. Josh Randall was our veteran team member on call in England (which helped because he was available during shifted hours for us). And Matt Goldstein set up our HQ call center and auto-dial app with VoIP phones provided by IS&T.

With the exception of only a few issues (which I’ll try to address below), from the systems side of things the hunt ran extremely well. We were the first hunt in a while to actually start on time, and we were also the first hunt in a while to actually have the solutions and hunt solve statistics up and posted by wrap-up on Monday. This hunt had a record number of participants, and a record number of teams, (both higher than we planned when designing and testing the system) making our job all the more difficult. And of course, I’d like to join everyone else in thanking the entire team of Alice Shrugged that made this hunt possible. It was great working with you all year and pulling off what many feel was a fantastic hunt.

Hunt Archive and Favorites

To look at the actual hunt, including all puzzles and solutions, and some team and hunt statistics, go to the 2014 Hunt Archive. My favorite puzzles (since everyone seems to be asking) were: Callooh Callay World and Crow Facts. Okay, I guess Stalk Us Maybe was pretty neat too.


First of all, I’d like to apologize on behalf of the systems team for the issues with Walk Across Some Dungeons. This performed artificially well during our load test, but load test clients are far more well behaved than actual real people on a real network. There were several socket locking and connection starvation issues with the puzzle even after we spent all day (and night) Friday parallelizing it onto as many as 7 virtual servers. Eventually we patched the code to allow for better handling of dropped connections and to be more multi-threaded within each app instance and by late Saturday night it was working much better. The author has since ported it to javascript, and it should work fine well into the future. Lesson to future teams: don’t try to write your own socket-handling code or any server-side code for that matter that has to interact with the hunters. The issues surrounding the puzzle (lag, and the frequent reset back to stage 1 requiring many clicks to get back to your position) affected every team equally for the first day and were not fixed until at least the top 5 teams had already solved the puzzle in its “difficult” state. So luckily this was not a fairness issue, it just made a puzzle a LOT harder than it should have been.

Second, there was an interesting issue with our hunt republishing code. At times during the hunt there were errata (remarkably few, actually) and some point-value and unlocking-system changes (will mention more about this below) that required a full republish of the hunt for all teams. This is not unusual. However, with the number of teams and hunters and the pace our call handlers (particularly Zoz the queue-handling machine) were progressing teams through the hunt on Friday in particular, this created a race condition. If any puzzle unlocks happened during one of these republishes, they would be put back to the state they were when the publish started. Since a republish takes a looooooong time for all of these teams and puzzles, a number of teams noticed “disappearing” puzzles and unlocks on Friday while we were updating our first erratas in puzzles and then later Friday night when we changed the value of Wonderland train tickets to slow the hunt down a bit. We alleviated this slightly in the spoilr code by making the republish iterate team by team rather than take its state of the whole hunt at the beginning and then apply it to everyone. By later on Friday though, teams had enough puzzles unlocked that even just republishing for a team had a risk of coinciding with a puzzle unlock, so we simply froze the handling of the call-in queue while we were doing these. As a note for future teams, this could probably be fixed by making the republish work more transactional in the code.

Release Rate, Fairness, and Fun

On this subject I can not pretend to speak for the whole team (nor can anyone probably), but I will share what I experienced and what I think about it. Many medium and small-sized teams have written to congratulate us on running a hunt that was fun for them and that encouraged teams to keep hunting in some cases over 24 hours after the coin was found. On the flip side some medium and large-sized teams were a bit disappointed in the later stages of the hunt when puzzles unlocked at a slower rate (particularly once all rounds were unlocked) leaving them with less puzzles to work on and creating bottlenecks to finishing the hunt. One of the overriding principles of us writing this hunt was to make it fun for small teams, and fair for large teams. The puzzle release mechanism in the MIT round(s) was fast, furious and fun. Something like 30 teams actually solved the MIT Meta and got to go on the MIT runaround and get the “mid-hunt” reward. From the beginning of our design, the puzzle release mechanism for the wonderland rounds (particularly the outer ones) was constrained to release puzzles in an already-opened round based only on correct answers in that round. The rate of how many answers in a round it took to open up the next set of puzzles in that round, and the order in which puzzles were released in a given round was designed to require focused effort on a smaller number of opened puzzles in order to progress to a point where those metas were solvable. This rate was, incidentally, tweaked to be somewhat lower on Friday night (but only for the two rounds no team had opened yet) in a concerted effort to make sure the coin wasn’t found as early as 6-8pm on Saturday. Coming from a large team myself, I have seen the effect of the explosion of team size on the dynamics of Mystery Hunt. This is an issue that teams will face for years to come, and everyone may choose to solve it a different way. But once again, our overriding goal was to make the hunt fun for small teams, and fair for large teams, and I think we did just that.

Architecture Overview

For the curious, and to those running the hunt next year, our server setup was fairly simple. We had one backend server which ran a database and all of the queue-handling and hunt HQ parts of the spoilr software (in django via mod_wsgi and apache). There were two frontend servers which shared a common filesystem mount with the backend server so all teams saw the consistent view of unlocks. Each team gets its own login and home directory which controls their view of the hunt when the spoilr software updates symlinks or changes the HTML files there. The spoilr software on the frontends handled answer submissions and contact HQ requests among some other things, but they were mostly just static web servers. We didn’t need two for load reasons, we just had both running for redundancy in case one pooped out over the weekend. However, splitting the dynamic queue operations and Hunt HQ dashboards off from the web servers that 1500+ hunters were hitting for the hunt was a necessity. Each of the front ends also acted as a full streaming replica of the database on the backend server, and we had a failover script ready so the hunt could continue even if the backend server and database failed somehow. There was also a streaming database replica and hunt server in another colocation facility in Chicago in case somehow both datacenters that the hunt servers were in failed or lost internet connectivity. I’d like to thank Axcelx Technologies for providing us with hosting and support, and would recommend them to anyone looking for a reasonably priced virtual server provider or collocation provider.

As far as writing the hunt goes, we used the now-standard “puzzletron” software and made a lot of improvements to that and hope to get that pushed back up to gitweb for the next team to start writing with. We had dev and test instances of puzzletron running all year so we could deploy our new features quickly and safely as our team came up with neat new things to track with it. Beyond that, we set up a mediawiki wiki, and a phpbb bulletin board, as well as several mailman mailing lists and a jabber chat server (which nobody really used). As a large team, collaboration tools have always been very important for us in trying to win the hunt, and were even more important in writing it. In retrospect, we probably should have taken more time to develop an actual electronic ticketing system (or find one to use) for the run-time operations of the hunt. Instead we ended up using paper tickets which passed back and forth between characters, queue handlers, and the run-time people. Since this hunt had so many interactions and so many teams which needed to get through them, this got clumsy and some were dropped or not checked off early in the hunt (I’m very sorry if this happened to any teams and delayed unlocks of puzzles/rounds early on).

In Closing

In closing, I had a great time working on the hunt. I can’t say how great it would have been to go on it, since sadly I did not get to. But, hearing the generally positive comments from everyone thus far, I’m glad we didn’t screw it up :) The mailing list will continue to work into the future, and I look forward to getting some of our code and documentation posted up for random to perhaps use and further improve upon next year, and for other teams to carry on the tradition for many years to come.


MIT Mystery Hunt 2013 (a.k.a. The Misery Hunt)

Happy June everyone. Back in January, I had the privilege of being on the winning team of the 2013 MIT IAP Mystery Hunt (pretty sure I already mentioned that a couple of posts ago). For those unaware, we were a huge team (~100+ people), and the name of our team was the full text of Ayn Rand’s Atlas Shrugged. Whenever we’d communicate with hunt HQ, we’d continue reading the text until they made us stop (or let us stop). Among others thought this would be a neat, clever idea — maybe even “cute.”  In reality, however, it turned out to just add to the pain and misery of what turned out to be an already painful and misery-drenched (but also somewhat fun) hunt. Tired-sounding reading of the rambling Randian prose quickly became the leitmotif for the weekend.

Other people have already shared their opinions and experiences about the hunt (google “2013: the year the mystery hunt broke” for an example).  The organizers (Manic Sages) have, in my opinion, already gotten more than enough criticism dumped on them for putting together the longest weekend in hunt history that almost ended in hunt ending by decree or draw — which would have been disastrous for the 2013 hunt, as well as the concept and tradition of the mystery hunt moving forward (in my opinion).

The word “grueling” was the one word I used most when people asked me what the hunt was like this year.  I’ve participated in the hunts with this team consistently for the past 5 years and on and off going back to 2004 (the last year our team won) and earlier.  I spent four years at MIT, with all of the all-nighters, failing grades, and frustration that that entails.  But I’ll be damned if the 2013 mystery hunt wasn’t one of the most intellectually demoralizing experiences of my life. Is that such a bad thing? In retrospect,  I’m not so sure. Challenging experiences and “rolling up one’s sleeves and getting to work” (sometimes by doing insane statistical analysis on endless streams of random numbers) are ways in which we attain personal growth — right?  Maybe if it was just as difficult, but shorter.  Maybe if it had some more fun and games mixed in.  Maybe if we didn’t decide to do that stupid Atlas Shrugged thing.  Maybe then the hunt would have been FUN as well as just grueling — and wouldn’t have left me with hunt PTSD.  Seriously, I’m not alone on my team in having had nightmares up to a week after hunt about still doing the hunt, or still needing to solve a meta.

I’m not going to bore everyone with detailed stories of extremely difficult puzzles with perhaps one-too-many “a-ha!” moments necessary to solve, or the detailed methods our team uses to keep fresh shifts of solvers moving in and out of the room, taking naps, and ultimately winning the battle of attrition that the last 24 hours of the weekend became.  But I will recount my tale of how the hunt ended (from my perspective).

The beginning of the end was 8pm Sunday night (already over 16 hours after the point the 2012 mystery hunt had ended in its weekend). We already knew by that point that this was going to be a hunt for the ages.  My team had that glazed-over deer-in-the-headlights look that comes from being up for 20-30+ hours in some cases doing extreme mental gymnastics. An email came in from HQ reading: “Our honest estimate of hunt’s end is Monday at 9AM given what we’ve seen of solving rates on our puzzles so far.”  At this time, I was on my way out to a room to sleep for a quick 4 hours (or until hunt ended).  It turns out that not only was hunt nowhere near ending, but with an end-time of 9AM, they were predicting it to surpass the 2004 hunt for the all-time duration record.  And who had written the 2004 hunt?  Our team — then known as “French Armada” (because the team wanted to wear funny hats — from what I hear).  When I woke up to my alarm 4 hours later, my disillusionment with the mystery hunt had turned into a sort of prideful anger.  How dare they assume that their hunt will be even more un-defeatable than the one we’d written (somewhat poorly) a decade earlier?  At around 2am, the late-night shift of fresh puzzlers dug in and I, for one, was hoping to prove the Manic Sages’ assumption wrong and to keep our dubious record of “longest hunt ever.”

But it was not to be. At 6AM, or so, I went back in for another brief nap. 9AM came and went, and another shift of freshly-napped hunters came in. The 2004 French Armada’s hunt length record had fallen.  Free answers to puzzles were getting handed out every 20 minutes now to help draw things to a close.  The requirements for finishing hunt were changed so that one full meta-puzzle (out of 5 total) could be skipped entirely. For those unfamiliar with the standard mechanics of a mystery hunt, there are generally puzzles in “rounds”, and then for each round (or group of rounds), the answers of the puzzles plug into a “meta” puzzle.  Once all meta-puzzles are complete, the team is eligible to go on a final “runaround” (involving, literally, running around and solving more puzzles) and ultimately win the hunt.  So, eliminating an entire meta-puzzle requirement was a big deal — and up until this year, unheard of (at least by me).

At some point on Monday morning, one of our freshmen (Lauren Herring) had been sitting in the same spot working on one of the metas (“The Enigma”) for what seemed like 12 hours.  She’d be sitting there when I left for a nap, and she’d be in the same spot, wild-eyed and turning those same infernal rolls of paper when I got back to the room hours later.  And so eventually, that meta got solved.  And that left us needing exactly one more meta to get to the runaround.  One of them (“Rubik”) seemed totally impossible and we had made little progress on it at all from what I could see.  The other one (“Indiana Jones”) was getting churned on slowly at a table by puzzlers including our “old guard” — the bleary-eyed Mark Feldmeier and Zoz Brooks — with the whole team cheering them on.  Actually it was more nervous pacing, drinking coffee, and watching vs. audible “cheering.”  We were getting close by 10-11am, and calling HQ regularly for hints and clues.  We had even called in to verify a partial answer for it — “hey are the first 8 letters of the answer this?” (this is also unheard of in a mystery hunt) — only to get rebuked.

And then something weird happened.  Our team phone rang, and Manic Sages’ HQ was on the other end.  The inimitable Laura Royden was “manning” the phone at the time and dealing with team-wide organization (we call it “puzzle bitch”ing).  It turns out that an offer of settlement/surrender had been made and was being brokered by the Manic Sages in the interest of ending hunt.  The terms were to stop hunting now and whatever team was deemed the “furthest ahead” by the Sages would be declared the winner.  After being at it for over 70 hours at that point, it was a tempting prospect.  The team huddled together, and Laura told HQ that we’d call them back in “a few minutes”.  How far could any other team possibly be if they were willing to make this offer? We heard rumors of other competitive teams giving up and packing in to go home by this point, leaving us as one of the few teams insane and stubborn enough to still be trying to win.  We knew we only needed one more meta but were, for the moment, scratching our heads on what we were doing wrong with “Indiana Jones.” A couple of the more senior puzzlers on our team (Dan Katz and Erin Rhode at least from what I remember) immediately leaned towards rejecting the offer.  Then we got another call from HQ, and they told us that they’d made a terrible mistake and our partial answer check from earlier was actually on the right track. At that point the choice was clear. Not only did we know that we were the farthest ahead, but we also knew we were potentially only minutes away from winning.  To accept the other team’s surrender at that point would have perhaps been merciful, but wouldn’t have been a good thing for the 2013 hunt, or hunt as a tradition and concept (in my opinion at least).  Laura shouted clearly into the phone: “no we will not accept your offer!” And, sure enough, about 10 minutes later, she called back with the complete correct answer to our final required meta-puzzle and accepted congratulations that we had, for all intents and purposes (with the exception of the runaround), won the hunt.  Below is a picture commemorating that very moment.


Once that was done, Manic Sages actually asked if we wanted to do the full runaround, or just be handed the coin and declared the winners at that point.  Staying true to tradition, even though it was almost noon on Monday at that point, we elected to make them put on the entire runaround for us.  At 3:30pm on Monday, a full 75 hours after the hunt began, the coin was found by the small subgroup of our team that was still awake (this did not include me, as I collapsed shortly after the final answer was called in and I knew we had won).

So now what?  Now our team is writing and running the 2014 IAP MIT Mystery Hunt, that’s what.  The experience of last year (and echoes of our 2004 hunt) sort of lend a feeling of “there but for the grace of god, go I” to the whole thing.  Each and every one of us knows (or should know) that it is very much possible, with the best of intentions and the smartest and most experienced people, to write a hunt that turns out to be “bad” or even maybe “a disaster.”  That’s kind of a lot of responsibility, isn’t it?  But alas, we will do our best.  Without further ado, I’ll wrap up here and introduce the board of directors of the 2014 IAP MIT Mystery Hunt.  For continued missives from our team, and guest writers talking about hunt, please visit our blog at  And oh yeah, good luck in 2014 everyone!

DSC_0967 copy

Galen Pickard (Executive Producer)

Anand Sarwate (Finance)

Anand Sarwate (Finance)

Erin Rhode (Director)

Erin Rhode (Director)

DSC_0992 copy

Benjamin O’Connor (IT and Infrastructure)


Pranjal Vachaspati (Operations and Logistics)

DSC_0970 copy

Laura Royden (Theme)

DSC_0971 copy

Harvey Jones (Quality Control)


Roger Boisjoly Poll

Roger Boisjoly
So, what do you think?

Leave a comment

The Play’s The Thing

Tomorrow, and tomorrow, and tomorrow
Creeps in this petty pace from day to day,
To the last syllable of recorded time;
And all our yesterdays have lighted fools
The way to dusty death. Out, out brief candle!
Life’s but a walking shadow, a poor player
That struts and frets his hour upon the stage
And then is heard no more: it is a tale
Told by an idiot, full of sound and fury,
Signifying nothing.

Macbeth, Act 5, Scene 5

So, just curious. What are your favorite words from the bard? The above piece has struck my fancy lately, although it’s neck and neck with “what dreams may come.” Discuss!


Clip Show

In honor and observance of the Writer’s Strike currently going on, here is my version of a “clip show.” This blog has over 175 entries now, some good, some bad, some awesome. I know there are several new readers here, and it’s a pain to sift through the entire blog to find the good nuggets. Here, in my opinion, are the 15 most worthy entries if you’re flipping through this place I like to call “Into the Weeds”:

  • yuck: Cold Stone Creamery Rant
  • The Soul of the Commuter: Commuting sucks. In very many ways. Here I get back to my urban studies roots.
  • UIUC Mail Server Disaster: A tale of sysadmin sorrow from 2002
  • Trivia: See if you can identify who all of these footnotes to history are.
  • The Cult of Scientology: Mini-rant and link to a freaking awesome article on scientolgy.
  • Longwood Towers: Rant about why the Longwood Towers Condominiums are some place you should not be thinking about living in.
  • : Classic tale of dot-bomb woe. Ever wonder what that “kozmo” box is that’s sitting in the kitchen lounge on second west?
  • Florence Co. ADMAX: Crazy terrorist prison in Colorado with all sorts of nasties inside.
  • New Apartment: Check out the awesome view from my living room.
  • Driving in Boston: A primer, with scary crazy maps of intersections.
  • Captain Al Haynes: The amazing story of Capt. Haynes and United flight 232.
  • Firefighting Mode: Some systems philosophy, and the condition of what things were like back when I worked at MIT IS&T.
  • The Switch: Mac vs. PC
  • Unofficial!!!!: The crazyness at UIUC known as “Unofficial St. Patrick’s Day” and why it is awesome and awesomely stupid at the same time.
  • Datacenter Transformer: A tale of woe from some crazy-ass electrical problem and superbad preparation and handling at MIT IS&T.

1 Comment

Balls to the Wall

And now for a brief piece of trivia that some/most of you may already be aware of, considering one of my favorite colorful phrases in the english language, “balls to the wall.” My apologies in advance to those readers who are already aware of the etymology.

I told the staff … that I expected them to cut every piece of red tape, do everything they could, that it was balls to the wall” -Michael Brown, former director of FEMA, in testimony to U.S. Senate 2/10/06 [emphasis mine]

Contrary to common belief and misconception, the phrase has nothing whatsoever to do with potentially uncomfortable proximity of genitalia and walls. It’s origins apparently date back to use in the US Air Force during the Vietnam War, and possibly as far back as the Korean War or earlier. You see, “balls” refers to the balls on top of the throttle control and flight control of an airplane, and the “wall” is the cockpit firewall. Thus, “balls to the wall” would indicate the state where the plane is either pushed into a nose dive and summoned to 100% throttle, for example in a situation where extreme evasive maneuvers are required.

Similar to “pedal to the metal,” “balls to the wall” indicates a condition of maximum effort and intensity. But, of course the connotations and nuances of the term are so much more than that. Like all phrases of this kind, it really isn’t possible to succinctly and effectively rephrase it in other language — that’s the reason it exists and is in use in the first place.


The Ides of March

It seems that I’m out of interesting things to write about for now, so let’s all stop and take a moment to remember 2000; specifically March 10th 2000. You see, that’s the date that is numerically recognized as the burst of the Dot Com bubble. Seven years ago this past weekend, the Nasdaq peaked at 5,132.52. Yahoo closed at 89 points. Akamai closed at 296. Even Sun Microsystems got in on the action, and closed at 47 points. Less than a month later, the news was as such on April 4th at 1:42pm eastern time on CNN:

PATRICIA SABGA, CNN ANCHOR: Our market coverage continues on CNNfn. We would also like to welcome our CNN viewers. Right now, the Nasdaq composite is off 332 at 3890. That is, however, well off the lows of the session. The Nasdaq composite had lost as much as 13 percent so far today.

BILL TUCKER, CNN ANCHOR: In fact, it is now well into bear territory, completely more than 20 percent off its highs, those highs just hit back in early March, on March 10th at 5048. The Dow also selling off very strongly today, had been down over 500 point, we have seen a big comeback in the Dow 30 stocks.

It’s actually kind of entertaining to read this old transcript. The market correspondent then comes on and says stuff like: “…However, the bottom, he sees it, as in sight. Now the level he had picked as a bottom for this market was around 3500; 3590 would represent a 62 percent retracement of the market’s movement upward since early October….” Of course, a year later, in April of 2001, the market was closing at 1638. How’s that for a “retracement?”

So yes, there it was. 7 years ago: the Nasdaq peaked at 5,132.52. Yahoo closed at 89 points. Akamai closed at 296. Even Sun Microsystems got in on the action, and closed at 47 points. Today, the Nasdaq is at 2378, Yahoo is struggling to maintain 30, Akamai’s hanging out respectably in the low 50s, and beleaguered Sun Microsystems is at 6.22.



%d bloggers like this: