Archive for category life

MIT Mystery Hunt 2013 (a.k.a. The Misery Hunt)


Happy June everyone. Back in January, I had the privilege of being on the winning team of the 2013 MIT IAP Mystery Hunt (pretty sure I already mentioned that a couple of posts ago). For those unaware, we were a huge team (~100+ people), and the name of our team was the full text of Ayn Rand’s Atlas Shrugged. Whenever we’d communicate with hunt HQ, we’d continue reading the text until they made us stop (or let us stop). Among others thought this would be a neat, clever idea — maybe even “cute.”  In reality, however, it turned out to just add to the pain and misery of what turned out to be an already painful and misery-drenched (but also somewhat fun) hunt. Tired-sounding reading of the rambling Randian prose quickly became the leitmotif for the weekend.

Other people have already shared their opinions and experiences about the hunt (google “2013: the year the mystery hunt broke” for an example).  The organizers (Manic Sages) have, in my opinion, already gotten more than enough criticism dumped on them for putting together the longest weekend in hunt history that almost ended in hunt ending by decree or draw — which would have been disastrous for the 2013 hunt, as well as the concept and tradition of the mystery hunt moving forward (in my opinion).

The word “grueling” was the one word I used most when people asked me what the hunt was like this year.  I’ve participated in the hunts with this team consistently for the past 5 years and on and off going back to 2004 (the last year our team won) and earlier.  I spent four years at MIT, with all of the all-nighters, failing grades, and frustration that that entails.  But I’ll be damned if the 2013 mystery hunt wasn’t one of the most intellectually demoralizing experiences of my life. Is that such a bad thing? In retrospect,  I’m not so sure. Challenging experiences and “rolling up one’s sleeves and getting to work” (sometimes by doing insane statistical analysis on endless streams of random numbers) are ways in which we attain personal growth — right?  Maybe if it was just as difficult, but shorter.  Maybe if it had some more fun and games mixed in.  Maybe if we didn’t decide to do that stupid Atlas Shrugged thing.  Maybe then the hunt would have been FUN as well as just grueling — and wouldn’t have left me with hunt PTSD.  Seriously, I’m not alone on my team in having had nightmares up to a week after hunt about still doing the hunt, or still needing to solve a meta.

I’m not going to bore everyone with detailed stories of extremely difficult puzzles with perhaps one-too-many “a-ha!” moments necessary to solve, or the detailed methods our team uses to keep fresh shifts of solvers moving in and out of the room, taking naps, and ultimately winning the battle of attrition that the last 24 hours of the weekend became.  But I will recount my tale of how the hunt ended (from my perspective).

The beginning of the end was 8pm Sunday night (already over 16 hours after the point the 2012 mystery hunt had ended in its weekend). We already knew by that point that this was going to be a hunt for the ages.  My team had that glazed-over deer-in-the-headlights look that comes from being up for 20-30+ hours in some cases doing extreme mental gymnastics. An email came in from HQ reading: “Our honest estimate of hunt’s end is Monday at 9AM given what we’ve seen of solving rates on our puzzles so far.”  At this time, I was on my way out to a room to sleep for a quick 4 hours (or until hunt ended).  It turns out that not only was hunt nowhere near ending, but with an end-time of 9AM, they were predicting it to surpass the 2004 hunt for the all-time duration record.  And who had written the 2004 hunt?  Our team — then known as “French Armada” (because the team wanted to wear funny hats — from what I hear).  When I woke up to my alarm 4 hours later, my disillusionment with the mystery hunt had turned into a sort of prideful anger.  How dare they assume that their hunt will be even more un-defeatable than the one we’d written (somewhat poorly) a decade earlier?  At around 2am, the late-night shift of fresh puzzlers dug in and I, for one, was hoping to prove the Manic Sages’ assumption wrong and to keep our dubious record of “longest hunt ever.”

But it was not to be. At 6AM, or so, I went back in for another brief nap. 9AM came and went, and another shift of freshly-napped hunters came in. The 2004 French Armada’s hunt length record had fallen.  Free answers to puzzles were getting handed out every 20 minutes now to help draw things to a close.  The requirements for finishing hunt were changed so that one full meta-puzzle (out of 5 total) could be skipped entirely. For those unfamiliar with the standard mechanics of a mystery hunt, there are generally puzzles in “rounds”, and then for each round (or group of rounds), the answers of the puzzles plug into a “meta” puzzle.  Once all meta-puzzles are complete, the team is eligible to go on a final “runaround” (involving, literally, running around and solving more puzzles) and ultimately win the hunt.  So, eliminating an entire meta-puzzle requirement was a big deal — and up until this year, unheard of (at least by me).

At some point on Monday morning, one of our freshmen (Lauren Herring) had been sitting in the same spot working on one of the metas (“The Enigma”) for what seemed like 12 hours.  She’d be sitting there when I left for a nap, and she’d be in the same spot, wild-eyed and turning those same infernal rolls of paper when I got back to the room hours later.  And so eventually, that meta got solved.  And that left us needing exactly one more meta to get to the runaround.  One of them (“Rubik”) seemed totally impossible and we had made little progress on it at all from what I could see.  The other one (“Indiana Jones”) was getting churned on slowly at a table by puzzlers including our “old guard” — the bleary-eyed Mark Feldmeier and Zoz Brooks — with the whole team cheering them on.  Actually it was more nervous pacing, drinking coffee, and watching vs. audible “cheering.”  We were getting close by 10-11am, and calling HQ regularly for hints and clues.  We had even called in to verify a partial answer for it — “hey are the first 8 letters of the answer this?” (this is also unheard of in a mystery hunt) — only to get rebuked.

And then something weird happened.  Our team phone rang, and Manic Sages’ HQ was on the other end.  The inimitable Laura Royden was “manning” the phone at the time and dealing with team-wide organization (we call it “puzzle bitch”ing).  It turns out that an offer of settlement/surrender had been made and was being brokered by the Manic Sages in the interest of ending hunt.  The terms were to stop hunting now and whatever team was deemed the “furthest ahead” by the Sages would be declared the winner.  After being at it for over 70 hours at that point, it was a tempting prospect.  The team huddled together, and Laura told HQ that we’d call them back in “a few minutes”.  How far could any other team possibly be if they were willing to make this offer? We heard rumors of other competitive teams giving up and packing in to go home by this point, leaving us as one of the few teams insane and stubborn enough to still be trying to win.  We knew we only needed one more meta but were, for the moment, scratching our heads on what we were doing wrong with “Indiana Jones.” A couple of the more senior puzzlers on our team (Dan Katz and Erin Rhode at least from what I remember) immediately leaned towards rejecting the offer.  Then we got another call from HQ, and they told us that they’d made a terrible mistake and our partial answer check from earlier was actually on the right track. At that point the choice was clear. Not only did we know that we were the farthest ahead, but we also knew we were potentially only minutes away from winning.  To accept the other team’s surrender at that point would have perhaps been merciful, but wouldn’t have been a good thing for the 2013 hunt, or hunt as a tradition and concept (in my opinion at least).  Laura shouted clearly into the phone: “no we will not accept your offer!” And, sure enough, about 10 minutes later, she called back with the complete correct answer to our final required meta-puzzle and accepted congratulations that we had, for all intents and purposes (with the exception of the runaround), won the hunt.  Below is a picture commemorating that very moment.

424244_910363591908_1978574625_n

Once that was done, Manic Sages actually asked if we wanted to do the full runaround, or just be handed the coin and declared the winners at that point.  Staying true to tradition, even though it was almost noon on Monday at that point, we elected to make them put on the entire runaround for us.  At 3:30pm on Monday, a full 75 hours after the hunt began, the coin was found by the small subgroup of our team that was still awake (this did not include me, as I collapsed shortly after the final answer was called in and I knew we had won).

So now what?  Now our team is writing and running the 2014 IAP MIT Mystery Hunt, that’s what.  The experience of last year (and echoes of our 2004 hunt) sort of lend a feeling of “there but for the grace of god, go I” to the whole thing.  Each and every one of us knows (or should know) that it is very much possible, with the best of intentions and the smartest and most experienced people, to write a hunt that turns out to be “bad” or even maybe “a disaster.”  That’s kind of a lot of responsibility, isn’t it?  But alas, we will do our best.  Without further ado, I’ll wrap up here and introduce the board of directors of the 2014 IAP MIT Mystery Hunt.  For continued missives from our team, and guest writers talking about hunt, please visit our blog at http://mysteryhunt.wordpress.com/.  And oh yeah, good luck in 2014 everyone!

DSC_0967 copy

Galen Pickard (Executive Producer)

Anand Sarwate (Finance)

Anand Sarwate (Finance)

Erin Rhode (Director)

Erin Rhode (Director)

DSC_0992 copy

Benjamin O’Connor (IT and Infrastructure)

MysteryHunt_6

Pranjal Vachaspati (Operations and Logistics)

DSC_0970 copy

Laura Royden (Theme)

DSC_0971 copy

Harvey Jones (Quality Control)

3 Comments

New Job Observations: Farewell Harmonix, Hello TripAdvisor.


I know it’s April already, but happy new year everyone!

For those not in the know, I got a somewhat unexpected new job prospect (and offer, which I accepted) at the end of 2012. Since then, I’ve been a senior member of the technical operations team at TripAdvisor.

TripAdvisor is the world’s largest travel site, with over 100 million reviews, and over 100 million unique users per month. For people keeping track, this is the third company I worked for during the year 2012, and all three have been mentioned on The Office (Linden Lab [Second Life], Harmonix [Guitar Hero / Rock Band], and now Tripadvisor [check out the Schrute Farms episode]). However, it’s not just popularity or “hipness” that led me to shift around.

I like to tell people (and recruiters) that I have four rules for picking a place to work:

  • Must not be generally evil or tending towards evil (in my opinion) — This rules out most banks or the pharmaceutical industry, any petrochemical comany, and probably currently most of Google and Facebook.
  • Must be a profitable venture — I’m too old to play the startup risk game.
  • Must be accessible to my apartment in Boston via public transportation commute of <30 minutes — I don’t own a car, don’t want one, and I’m not moving anywhere.
  • Operations and Systems must be critical to the core business and of the highest priority — My job is best executed when it has the highest respect and attention of the company and management (immediate as well as upper).

It was that last one that I forgot about when I ended up at Harmonix. After being at Linden for 4+ years, I could feel myself falling into the crotchety grizzled BOFH sysadmin role. Come to think of it, that probably happens to anyone in my field after a few years in an organization. Spending a year at Harmonix was a great chance to broaden my horizons, relax, and experience new perspectives on things. As I stated in an earlier blog post, I loved working there, and I do miss the place, people, and incredibly fun things happening in their awesome Central Square office. At TripAdvisor, we’re still in the business of providing joy to people. Rather than by selling some of the best video games, this time it’s by helping folks plan and take vacations.

Very similar to my time at Linden Lab, when I told people that I worked at Harmonix (makers of Rock Band and Dance Central franchises) the first response was usually “wow that’s really cool.” However, the second response was more often than not, “are they still relevant? What are they working on now?” While it’s true that the heyday of plastic instruments (and maybe console gaming in general — according to some naysayers) has passed, I’m still rooting for the folks over there, and I happen to know that they are still a vital, awesome independent studio with the best people and some blow-your-mind projects in the pipeline. If I was still there, I’d be hustling along side them doing my best to keep up and push forward the state of game network interaction and back ends. That being said, the effort that game developers (particularly independents) put into network features, operations, and backends is decreasing over time. And it should be. Great games are great because of the focus on art, gameplay, story, and other intangibles. Console manufacturers and third-party contractors can be brought on to do the job now of multi-player matchmaking and scoreboard databases, letting game makers stick to making awesome games and fostering and maintaining player communities — both things that Harmonix has done and will continue to do very well.

What drew me out to TripAdvisor (other than the folks I already know who work there — hi Laura and Drew!) was the scale. Honestly, I missed the excitement and challenges of running a huge infrastructure. At its peak, Second Life consisted of three data centers, 12,000+ servers, and received a new rack of 40 servers or so every couple of weeks. TripAdvisor isn’t quite that big infrastructure-wise (although we have 5 times as many employees), but we serve 2 billion ads a year, and are peaking at 600k web requests per minute (and growing tremendously still year-over-year). The company has a weekly release cycle, an innovative and freewheeling engineering culture, and an unofficial motto of “speed wins.”

At first, being a somewhat methodical systems engineer, the concept of putting velocity in front of “correctness” scared me a little bit. I’ve focused on things like proper cabling, thorough documentation, long planning cycles, enforcing automation prior to production, eliminating waste, etc. Here, though, I quickly learned that it’s important to keep moving and to cut a little slack to the folks that came before me for bad cabling, some missing documentation, or leaving a half dozen underutilized or unused servers around (sometimes literally powered-off in the racks or on the floor) while buying new ones in a hurry. If everyone takes the extra time (myself included) to do things the absolute correct way, we’ll lose our competitive advantage and then I’d be out of a job. So yeah, speed does win.

At this point, I’d be remiss if I didn’t offer you all potential jobs here. So, visit TripAdvisor Careers, find something you want to do, and drop me a line if I know you — I’d love to give a few hiring referrals, and yes we are hiring like crazy as the company expands!

1 Comment

MIT.edu and IS&T Fail


So, my team won the 2013 MIT IAP Mystery Hunt this past weekend.  More on that in another post though. 

On my way into work today after sending some emails about mystery hunt infrastructure, I started thinking on something and it seriously pissed me off.  Our team, and other teams running mystery hunt in recent years have been unable or unwilling to use MIT network and systems infrastructure to run the activity, and have instead needed to use private and other funding sources to host our hunting and collaboration tools with other internet hosting providers. This morning, I realized that our team should be able to have this event hosted at MIT next year, and that’s the angle I want to take when I start politely talking to the new leadership at IS&T. I just can’t find out who any of them are right now or their email addresses since their website is down. Not that email would necessarily get to them anyway. Even before the hunt, I was working on an email to send and post to open forums about these things, but now that the weekend is over and I’m facing the daunting task of running the system next year for our team, here goes:

An open letter to the Acting Director of MIT IS&T (if any person exists), and the MIT Administration:

Why does the MIT Mystery Hunt need to be hosted at EC2 or get sponsorship and infrastructure from VMware, Google, or Rackspace in the first place? MIT is still, in my opinion, the world’s preeminent engineering institution, yet its inability to host something as relatively mundane as a student-run puzzle hunt activity (yes the largest in the world, but still it’s just a puzzle hunt on a web site) in 2014 would be an absolute embarrassment.

Currently IS&T’s web site itself is down, email delivery between MIT and the rest of the world is spotty, the tech’s web site is inaccessible, 3down (the institute’s site-outage notification site) itself is down, and most other MIT-related and hosted web sites other than the front page are also inaccessible. The director of IS&T has resigned (apparently not as a result of these issues, The Tech reports — I’d link to the article but, well, you know, the site’s down).

The administration and IS&T will surely blame the DDOS (distributed denial of service) attack and anonymous (the amorphous organization out there on the web organizing these attacks) for all of this. Yet a site like WBC (Westboro Bible Church) has been the target of attacks for weeks now and none of their hateful websites are down or have needed to be mangled, or have been compromised or vandalized as badly as MIT’s have. Several other websites and companies (some of which I have worked for or currently work for) are also regularly targets of DDOS attacks and yet remain generally accessible and organized in the face of even the worst full-frontal internet assaults. IS&T’s response to the DDOS attack has been, from external appearances and my experiences on campus this weekend at least, worse than the attack itself. Continued vast service outages, intermittent detachment of MIT from email systems around the world, and zero effective communication with customers, departments, and students as far as I can tell. DDOS attacks are a fact of life on the internet. They should, like anonymous itself, be respected but above all expected.

At MIT, departments like the Broad Institute, Media Lab, CSAIL, etc. all have split off from MIT’s network and computing infrastructure because of IS&T’s apparent perennial failure as a service organization worthy of MIT and the people that work and study there.  Here’s an anecdotal example of the kind of failures that these departments and organizations expect:

When I was a Systems Administrator at IS&T, installing what passed as a small-sized supercomputer into an IS&T server room (hosted for the department of Biology if my memory serves correctly) caused a fire, power outage, and a rushed redesign of the power infrastructure, followed by several more power outages.

Universities like UIUC, the state systems at California and Florida, and Universities of Wisconsin, and Ohio all have well known, fully operational technology incubators and “startup factories” on their campuses connected and serviced through their network services, infrastructure, and hosting and IT departments. As the birthplace of so many startups, ideas, and technologies, it’s shameful that something like this can apparently not exist on the MIT network under the umbrella of IS&T in its current form. Is this because of current administration and management’s short-sightedness, entrenchments, technological incompetence, or a combination of all of the above?

With my team winning the 2013 MIT Mystery Hunt, we are already starting to look towards the 2014 Hunt and the network and computing services it will require. By engaging with the new leadership at IS&T, our team should be able to use the actual MIT infrastructure to give us what we need and want for a successful activity that showcases MIT to all of the world. We want to be able to have this event hosted on MIT’s network. Anything else should be an embarrassment to whoever is in charge of IS&T as well as the rest of the Institute’s administration.

-Benjamin O’Connor

  • Senior Systems Engineer, Tripadvisor (Formerly at Harmonix, Linden Lab, UIUC, NSA, and MIT IS&T)
  • Systems Infrastructure Manager, MIT Mystery Hunt Team <full text of Atlas Shrugged>
  • MIT Class of 2000

1 Comment

RIP Aaron Swartz


Aaron Swartz, best known as a co-author of the RSS specification, a co-owner of pupular social news site Reddit, and a lead campaigner against internet censorship and corruption committed suicide yesterday at the age of 26.

It’s sad to see such a brilliant, creative mind and potentially positive, young, influence for good in the world and community gone. I didn’t personally know him, and I can’t say I don’t have mixed feelings about the JSTOR incident (although it has been instrumental in bringing publicity to the issue of journal access and the heavy-handedness of computer-crime law in this country).

Some may say he went too far, pushed the envelope, and was unreasonable. George Bernard Shaw said the following:

“The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.”

Leave a comment

A Look Back in Time: “Golden Age of the Internet?”


In May of 1999, my hall at MIT’s East Campus Dormitory, known as “Second West”, “putz”, “PTZ”, “Pi Tau Zeta” (and probably many others), was featured in Yahoo! Internet Life Magazine because of the venerable putz webcam, which had been operating in some form since 1997. Today, of course, that webcam is still operating (assuming Pranjal or some current resident remembers to reboot it every once in a while) at http://www.mitwebcam.com.

So I kept the magazine. It turns out that the single paragraph and picture on the putz webcam wasn’t the only gem in this thing. I took the liberty of scanning the full magazine a while ago and a full pdf is linked below. Aside from the mention of the webcam, there’s an introduction to this new “stark” search engine known as “Google!” (not sure what the exclamation point is about), and an interview with Richard Branson about the future of the internet and his businesses. The centerpiece of the issue is the list of America’s top 100 wired colleges (MIT got 2nd place to Case Western, of all things, and I have no idea why).

Article mentioning “Google!”, and PTZ Webcam mention Clippings:

Full PDF of magazine issue (right click and “save as”):

1 Comment

Trip down memory lane. Who is this “Bin Ladin”???


Ahh, 1998. A much “simpler” time. Got this from a friend at the agency back then. Digging through old papers and I figured I’d post it without comment.

20121014-224052.jpg

Also, this doozy from 1993 is cute:

20121014-224413.jpg

1 Comment

Late News: I have a new job and it makes me happy.


Most friends of mine will already know that this January, after being a Senior Systems Engineer for over 4 years at Linden Lab (the makers of Second Life), I embarked on a journey to find something new, different, fun, and challenging (no, I wasn’t looking for — nor did I receive — additional financial compensation for switching jobs). This blog entry is a bit late, I know, but I figured I couldn’t readily sleep tonight, and just got back from an amazing showing at E3 in Los Angeles, so it’s appropriate to put some things down in virtual ink.

After looking around for about two weeks and working with an awesome recruiter at Hollister in Boston, I found what I was looking for. Since January, I’ve been a Senior Systems Administrator in the LiveOps group at Harmonix Music Systems.

As the video game industry grows more internet-connected, social, and network-dependent both in synchronous and asynchronous multiplayer capabilities, the rhythm, dance, music, and beat-matching games that we make at Harmonix need to as well. And all of that requires server infrastructure. Not just a hodgepodge of a few dozen boxes in a closet in the back of the office run by one guy, but an actual redundant, reliable, and well architected infrastructure that can hopefully serve the needs of all of our customers, contribute to the joy they get from playing video games, and grow and adapt for future titles with minimal financial and human investment.

As we saw from the piss-poor launch situation of Diablo III, this can sometimes be a challenging and daunting task, and I look forward to being on the team that makes sure all of our multi-player and backend functionality continues working without a hitch — even during the hopefully huge launches of new games coming up later this year in the Rock Band and Dance Central franchise that will both rely heavily on a newly architected and constructed backend infrastructure.

No, it’s not as huge or as technically challenging as running a virtual world with 16,000 servers and over 150 well-organized and configuration-managed high-load mysql servers spread over 3 remote datacenters, but it’s somehow more “fun.” At least so far it has been, and I hope it remains that way. In short, I love my new job and I love the company (we’re hiring, by the way). The organizational, personal and corporate-level challenges of designing, maintaining, and growing a smaller infrastructure with a smaller staff in a smaller company in many ways trump the far-out optimizing of an infrastructure with tens of thousands of servers and crazy-over-optimized solid-state-disk mysql clusters with 20+ slaving nodes each running an entire 3d virtual world the size of denmark and with the economy the size of Brazil or whatever country it is equal to these days.

The stuff we do, and will do, at Harmonix might not be the basis of any papers I’d be able to present at ATC or LISA, but it’s actually more reproducible and applicable to the vast majority of systems operations groups out there that we rely upon in our daily connected lives. And I look forward to sharing some of it with you, my faithful readers, as I hope my time here at Harmonix draws on through several awesome upcoming projects.

I used to think that working on SL was cool, and would occasionally see news stories or emails from residents saying how they met their mate there, or learned to escape a debilitating mental or physical deficiency by existing in the virtual world and could easily grasp the impact of what we were doing. At Harmonix, In addition to my work at the office in beautiful Central Square in Cambridge, Massachusetts, I get a chance to accompany our amazing community team to demos, conferences, and expos (to set up on-site backends for the games given lack of internet connectivity) and see just how much the once-derided (by my parents at least) video game industry has matured and risen to prominence in American culture. Harmonix’ games have brought fun and happiness to millions, and no doubt introduced (or re-introduced) many to the joy of music itself (and now, with the Dance Central series, dance). Remember the first time you played Rock Band with a few of your friends, and experienced that rush of pleasure and appreciation playing through a good tune properly and getting in the “groove”? Talking with some of our customers and fans at PAX-EAST and more recently E3, it really does kind of hit home.

And besides, I’ve said it before, but I get great satisfaction from working for a company that actually makes something real. We ship code on discs (ok, it’s downloadable content too these days) for people to buy, and play, and get enjoyment out of. This is not a shady business model of exploiting our customers by gathering up their personal information to spread around to the highest bidder (*cough* Facebook *cough*), or leveraging internet search (which, call me naive, is kind of a solved problem) and email account provisioning (also a solved and uninteresting problem) to also gather up personal information, track users, bubble them into predetermined categories and force feed them advertisements all the while violating their expressed wishes for privacy in many cases (*cough* Google *cough*).

So, what if my new job is less intense, less technically challenging or “awesome” in a geeky unix tech way? In many ways it is more rewarding, and I feel good about what I do when the day is done. I don’t think I could say that if I worked for any of those aforementioned silicon valley behemoths (despite being hounded by their recruiters regularly). But even more importantly on a personal level, and my main reason for switching jobs, is that it is quite a bit of a shift out of my comfort zone and more challenging in other ways. I’m now working with a smaller group and company of diverse talents and far different attitudes, personalities, and skill sets than I got used to at the mostly-all-computer-geek IT departments of universities and Linden Lab where I previously made my living.

So let’s lift a glass to change, sometimes even if it’s just for change’s sake. And also to all of the different types of people that make the video game industry, and our lives, work — the artists, musicians, talkers, writers, dreamers and thinkers, along with us nerdy engineers. And most of all I propose a toast to fun and joy, both of which I hope to be contributing to for many millions of players during my time at Harmonix.

Leave a comment

%d bloggers like this: