MIT.edu and IS&T Fail


So, my team won the 2013 MIT IAP Mystery Hunt this past weekend.  More on that in another post though. 

On my way into work today after sending some emails about mystery hunt infrastructure, I started thinking on something and it seriously pissed me off.  Our team, and other teams running mystery hunt in recent years have been unable or unwilling to use MIT network and systems infrastructure to run the activity, and have instead needed to use private and other funding sources to host our hunting and collaboration tools with other internet hosting providers. This morning, I realized that our team should be able to have this event hosted at MIT next year, and that’s the angle I want to take when I start politely talking to the new leadership at IS&T. I just can’t find out who any of them are right now or their email addresses since their website is down. Not that email would necessarily get to them anyway. Even before the hunt, I was working on an email to send and post to open forums about these things, but now that the weekend is over and I’m facing the daunting task of running the system next year for our team, here goes:

An open letter to the Acting Director of MIT IS&T (if any person exists), and the MIT Administration:

Why does the MIT Mystery Hunt need to be hosted at EC2 or get sponsorship and infrastructure from VMware, Google, or Rackspace in the first place? MIT is still, in my opinion, the world’s preeminent engineering institution, yet its inability to host something as relatively mundane as a student-run puzzle hunt activity (yes the largest in the world, but still it’s just a puzzle hunt on a web site) in 2014 would be an absolute embarrassment.

Currently IS&T’s web site itself is down, email delivery between MIT and the rest of the world is spotty, the tech’s web site is inaccessible, 3down (the institute’s site-outage notification site) itself is down, and most other MIT-related and hosted web sites other than the front page are also inaccessible. The director of IS&T has resigned (apparently not as a result of these issues, The Tech reports — I’d link to the article but, well, you know, the site’s down).

The administration and IS&T will surely blame the DDOS (distributed denial of service) attack and anonymous (the amorphous organization out there on the web organizing these attacks) for all of this. Yet a site like WBC (Westboro Bible Church) has been the target of attacks for weeks now and none of their hateful websites are down or have needed to be mangled, or have been compromised or vandalized as badly as MIT’s have. Several other websites and companies (some of which I have worked for or currently work for) are also regularly targets of DDOS attacks and yet remain generally accessible and organized in the face of even the worst full-frontal internet assaults. IS&T’s response to the DDOS attack has been, from external appearances and my experiences on campus this weekend at least, worse than the attack itself. Continued vast service outages, intermittent detachment of MIT from email systems around the world, and zero effective communication with customers, departments, and students as far as I can tell. DDOS attacks are a fact of life on the internet. They should, like anonymous itself, be respected but above all expected.

At MIT, departments like the Broad Institute, Media Lab, CSAIL, etc. all have split off from MIT’s network and computing infrastructure because of IS&T’s apparent perennial failure as a service organization worthy of MIT and the people that work and study there.  Here’s an anecdotal example of the kind of failures that these departments and organizations expect:

When I was a Systems Administrator at IS&T, installing what passed as a small-sized supercomputer into an IS&T server room (hosted for the department of Biology if my memory serves correctly) caused a fire, power outage, and a rushed redesign of the power infrastructure, followed by several more power outages.

Universities like UIUC, the state systems at California and Florida, and Universities of Wisconsin, and Ohio all have well known, fully operational technology incubators and “startup factories” on their campuses connected and serviced through their network services, infrastructure, and hosting and IT departments. As the birthplace of so many startups, ideas, and technologies, it’s shameful that something like this can apparently not exist on the MIT network under the umbrella of IS&T in its current form. Is this because of current administration and management’s short-sightedness, entrenchments, technological incompetence, or a combination of all of the above?

With my team winning the 2013 MIT Mystery Hunt, we are already starting to look towards the 2014 Hunt and the network and computing services it will require. By engaging with the new leadership at IS&T, our team should be able to use the actual MIT infrastructure to give us what we need and want for a successful activity that showcases MIT to all of the world. We want to be able to have this event hosted on MIT’s network. Anything else should be an embarrassment to whoever is in charge of IS&T as well as the rest of the Institute’s administration.

-Benjamin O’Connor

  • Senior Systems Engineer, Tripadvisor (Formerly at Harmonix, Linden Lab, UIUC, NSA, and MIT IS&T)
  • Systems Infrastructure Manager, MIT Mystery Hunt Team <full text of Atlas Shrugged>
  • MIT Class of 2000

1 Comment

RIP Aaron Swartz


Aaron Swartz, best known as a co-author of the RSS specification, a co-owner of pupular social news site Reddit, and a lead campaigner against internet censorship and corruption committed suicide yesterday at the age of 26.

It’s sad to see such a brilliant, creative mind and potentially positive, young, influence for good in the world and community gone. I didn’t personally know him, and I can’t say I don’t have mixed feelings about the JSTOR incident (although it has been instrumental in bringing publicity to the issue of journal access and the heavy-handedness of computer-crime law in this country).

Some may say he went too far, pushed the envelope, and was unreasonable. George Bernard Shaw said the following:

“The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.”

Leave a comment


Substitute the word “Communist” with “Socialist” and I could see this happening today all the same….

Under the Mountain Bunker

Yesterday vs. today:

1947 FBI Memo Re: “It’s A Wonderful Life”

With regard to the picture “It’s a Wonderful Life”, [redacted] stated in substance that the film represented rather obvious attempts to discredit bankers by casting Lionel Barrymore as a ‘scrooge-type’ so that he would be the most hated man in the picture. This, according to these sources, is a common trick used by Communists. [In] addition, [redacted] stated that, in his opinion, this picture deliberately maligned the upper class, attempting to show the people who had money were mean and despicable characters.”

Related: 

View original post

Leave a comment

Slugtalk Presentation: Systems Engineering in the 21st Century


For those of you that don’t know, the hall I used to live on at MIT, putz (a.k.a. East Campus Second West) has a relatively recent tradition of “slugtalks” (warning, page is terribly out of date).

People have been asking me these questions lately:

  • What does benoc do?
  • Why is it one of the fastest growing employment opportunities in the world, and why do we need more actually smart, well-educated people doing it?
  • Why aren’t we really called “sysadmins” anymore?

So last month I gave a pretty good slugtalk describing the modern state of my profession, what I do, and why you would want to consider joining this cabal of folks who keep the civilized world’s bits safe, flowing freely, and functional.

Topics Covered:

  • History of the field
  • Impact of the dotcom boom and scale explosion
  • Systems management theory
  • The “stack” and where we are in it — The rise of “Devops”
  • Why would someone with an MIT degree do this?
  • Admin lore: swearing and scotch

Full Presentation (PDF)

2 Comments

Roger Boisjoly Poll


Roger Boisjoly
So, what do you think?

Leave a comment

A Look Back in Time: “Golden Age of the Internet?”


In May of 1999, my hall at MIT’s East Campus Dormitory, known as “Second West”, “putz”, “PTZ”, “Pi Tau Zeta” (and probably many others), was featured in Yahoo! Internet Life Magazine because of the venerable putz webcam, which had been operating in some form since 1997. Today, of course, that webcam is still operating (assuming Pranjal or some current resident remembers to reboot it every once in a while) at http://www.mitwebcam.com.

So I kept the magazine. It turns out that the single paragraph and picture on the putz webcam wasn’t the only gem in this thing. I took the liberty of scanning the full magazine a while ago and a full pdf is linked below. Aside from the mention of the webcam, there’s an introduction to this new “stark” search engine known as “Google!” (not sure what the exclamation point is about), and an interview with Richard Branson about the future of the internet and his businesses. The centerpiece of the issue is the list of America’s top 100 wired colleges (MIT got 2nd place to Case Western, of all things, and I have no idea why).

Article mentioning “Google!”, and PTZ Webcam mention Clippings:

Full PDF of magazine issue (right click and “save as”):

1 Comment

Trip down memory lane. Who is this “Bin Ladin”???


Ahh, 1998. A much “simpler” time. Got this from a friend at the agency back then. Digging through old papers and I figured I’d post it without comment.

20121014-224052.jpg

Also, this doozy from 1993 is cute:

20121014-224413.jpg

1 Comment

%d bloggers like this: