Ahh, 1998. A much “simpler” time. Got this from a friend at the agency back then. Digging through old papers and I figured I’d post it without comment.
Also, this doozy from 1993 is cute:
Most friends of mine will already know that this January, after being a Senior Systems Engineer for over 4 years at Linden Lab (the makers of Second Life), I embarked on a journey to find something new, different, fun, and challenging (no, I wasn’t looking for — nor did I receive — additional financial compensation for switching jobs). This blog entry is a bit late, I know, but I figured I couldn’t readily sleep tonight, and just got back from an amazing showing at E3 in Los Angeles, so it’s appropriate to put some things down in virtual ink.
After looking around for about two weeks and working with an awesome recruiter at Hollister in Boston, I found what I was looking for. Since January, I’ve been a Senior Systems Administrator in the LiveOps group at Harmonix Music Systems.
As the video game industry grows more internet-connected, social, and network-dependent both in synchronous and asynchronous multiplayer capabilities, the rhythm, dance, music, and beat-matching games that we make at Harmonix need to as well. And all of that requires server infrastructure. Not just a hodgepodge of a few dozen boxes in a closet in the back of the office run by one guy, but an actual redundant, reliable, and well architected infrastructure that can hopefully serve the needs of all of our customers, contribute to the joy they get from playing video games, and grow and adapt for future titles with minimal financial and human investment.
As we saw from the piss-poor launch situation of Diablo III, this can sometimes be a challenging and daunting task, and I look forward to being on the team that makes sure all of our multi-player and backend functionality continues working without a hitch — even during the hopefully huge launches of new games coming up later this year in the Rock Band and Dance Central franchise that will both rely heavily on a newly architected and constructed backend infrastructure.
No, it’s not as huge or as technically challenging as running a virtual world with 16,000 servers and over 150 well-organized and configuration-managed high-load mysql servers spread over 3 remote datacenters, but it’s somehow more “fun.” At least so far it has been, and I hope it remains that way. In short, I love my new job and I love the company (we’re hiring, by the way). The organizational, personal and corporate-level challenges of designing, maintaining, and growing a smaller infrastructure with a smaller staff in a smaller company in many ways trump the far-out optimizing of an infrastructure with tens of thousands of servers and crazy-over-optimized solid-state-disk mysql clusters with 20+ slaving nodes each running an entire 3d virtual world the size of denmark and with the economy the size of Brazil or whatever country it is equal to these days.
The stuff we do, and will do, at Harmonix might not be the basis of any papers I’d be able to present at ATC or LISA, but it’s actually more reproducible and applicable to the vast majority of systems operations groups out there that we rely upon in our daily connected lives. And I look forward to sharing some of it with you, my faithful readers, as I hope my time here at Harmonix draws on through several awesome upcoming projects.
I used to think that working on SL was cool, and would occasionally see news stories or emails from residents saying how they met their mate there, or learned to escape a debilitating mental or physical deficiency by existing in the virtual world and could easily grasp the impact of what we were doing. At Harmonix, In addition to my work at the office in beautiful Central Square in Cambridge, Massachusetts, I get a chance to accompany our amazing community team to demos, conferences, and expos (to set up on-site backends for the games given lack of internet connectivity) and see just how much the once-derided (by my parents at least) video game industry has matured and risen to prominence in American culture. Harmonix’ games have brought fun and happiness to millions, and no doubt introduced (or re-introduced) many to the joy of music itself (and now, with the Dance Central series, dance). Remember the first time you played Rock Band with a few of your friends, and experienced that rush of pleasure and appreciation playing through a good tune properly and getting in the “groove”? Talking with some of our customers and fans at PAX-EAST and more recently E3, it really does kind of hit home.
And besides, I’ve said it before, but I get great satisfaction from working for a company that actually makes something real. We ship code on discs (ok, it’s downloadable content too these days) for people to buy, and play, and get enjoyment out of. This is not a shady business model of exploiting our customers by gathering up their personal information to spread around to the highest bidder (*cough* Facebook *cough*), or leveraging internet search (which, call me naive, is kind of a solved problem) and email account provisioning (also a solved and uninteresting problem) to also gather up personal information, track users, bubble them into predetermined categories and force feed them advertisements all the while violating their expressed wishes for privacy in many cases (*cough* Google *cough*).
So, what if my new job is less intense, less technically challenging or “awesome” in a geeky unix tech way? In many ways it is more rewarding, and I feel good about what I do when the day is done. I don’t think I could say that if I worked for any of those aforementioned silicon valley behemoths (despite being hounded by their recruiters regularly). But even more importantly on a personal level, and my main reason for switching jobs, is that it is quite a bit of a shift out of my comfort zone and more challenging in other ways. I’m now working with a smaller group and company of diverse talents and far different attitudes, personalities, and skill sets than I got used to at the mostly-all-computer-geek IT departments of universities and Linden Lab where I previously made my living.
So let’s lift a glass to change, sometimes even if it’s just for change’s sake. And also to all of the different types of people that make the video game industry, and our lives, work — the artists, musicians, talkers, writers, dreamers and thinkers, along with us nerdy engineers. And most of all I propose a toast to fun and joy, both of which I hope to be contributing to for many millions of players during my time at Harmonix.
In the midst of an epic (although, sucky so far) weekend of baseball right outside my window here at Fenway, I came across this blog entry. Reblogging here so that my readers (baseball fans and otherwise) might appreciate.
Happy tax-time, and Thanks for the link, @Pranjlolz:
Created by: OnlineCriminalJusticeDegree.com
Gee, It’s been a while since this old thing has been updated, eh? There are lots of new things in my life since that last post down there. New job, new year, new haircut to name a few
As you may have already noticed, the blog looks a little bit different today. I went ahead and migrated from an old Movable Type v3.3 instance I was hosting myself onto the up-to-date platform at wordpress.com. It wasn’t without a few bumps and hiccups, but I’m pretty sure everything is here and future updates will go smoothly. And I do have some updates in the queue.
Posted in work on May 1, 2011
If you’re the least bit involved with the business-end of internet services (as I certainly am), you’ll have already heard that a few weeks ago, Amazon Web Services (AWS)[aws.amazon.com] suffered a major outage of an east-coast region of their platform. This outage caused serious issues and downtime for many other internet systems (including, but not limited to: reddit, foursquare, heroku, quora, and my employer Linden Lab / Second Life) and services that have come to rely on AWS over the past few years as a reliable provider of what have become known as “cloud computing services.” [wikipedia] This is their offical post-mortem of the incident.
What is particularly interesting and notable about this outage, in my opinion, is the set of lessons we in the industry can learn about putting our eggs in such a basket, what “high-availability” really means, the dangers of “sorcerer’s apprentice synrome” and “auto-immune” vulnerabilities in redundancy engineering, and how to maintain a high level of service in this age of “cloud computing.” People are still arguing and wanking about where to place the blame for all of the havok that this incident wreaked upon the internet, but the plain truth is that there’s more than enough blame to go around for everyone — the web sites and service providers, as well as Amazon itself. On one side of things, it’s true that engineers and administrators should have spread deployments across multiple AWS regions (not just availability zones). On the other side of things, AWS has made it difficult to use multiple AWS regions, had indeed maintained that spreading deployments across availability zones would provide adequate insurance against an outage — and it turned out that in this case they were very very wrong, and pushed a new cloud storage service (EBS) that proved to be even more unreliable and in many cases incompatible with the possibility of using multiple AWS sites.
Here’s a quick rundown:
- Amazon outage and the auto-immune vulnerabilities of resiliency
- Amazon EC2 outage: summary and lessons learned
- Heroku status and post-mortem from the AWS outage — an illustration of how NOT to do things if you want reliability, but an admirable case of owning up to and being honest about your faults.
- How SmugMug survived the Amazonpocalypse — an illustration of how you SHOULD do things if you want reliability from a company that used AWS but was able to stay up.
No, in the end it turns out that it wasn’t Skynet’s fault after all. Just some over exuberance about new hottness, and a distinct deficit in reliability-engineering and availability paranoia.
Posted in sports on February 16, 2011
Looks like they’re assembling the new scoreboard at Fenway Park. It appears to be roughly the same size as the entire old structure, but consumed entirely by the digital board rather than surrounded by advertisements (for now at least). Nifty.
Posted in reality on January 11, 2011
So I have some thoughts about the new CDMA iPhone (compatible with Verizon’s network) released today that folks have been frothing about. Everybody knew that the AT&T exclusivity deal with Apple was ending this year, so it’s inevitable that other carriers — particularly Verizon, would eventually end up with an iPhone to sell to their customers.
The issue with Verizon, however, is that their CDMA network is incompatible with the electronics hardware in the iPhone for AT&T. This means that making an iPhone available to Verizon customers was never just a matter of flipping a switch somewhere, or installing some code in firmware. New electronics had to be designed to be integrated with and fit into the quite constraining iPhone form factor, and of course tested and approved by various regulatory officials.
CDMA, however, is on its way out the door. Verizon, AT&T, and other carriers, are already rolling out so-called 4G networks that will be wholly incompatible with already-odd-duck CDMA (the network, for example, lacks the ability to transmit voice and data at the same time — no web surfing while talking).
My question is then, is it really worth it for Verizon (or Apple for that matter) to go through the effort and expense of designing and rolling out this new phone on an old, deprecated network? Apple COO Tim Cook reportedly said that building it as a 4G LTE phone would “force some design compromises” and “customers have told us they want the iPhone now.” So for those still on Verizon’s network, wanting an iPhone, but unwilling/unable to switch to AT&T, my advice would still be to wait. I’d hope that this little side detour into CDMA 3G-land doesn’t delay the eventual release of a 4G LTE iPhone for both AT&T and Verizon in the near future.
The weather service has issued a Blizzard warning for Boston (among other places) this afternoon. Looks like things are starting to pick up out there. It’s not officially a blizzard until the Citgo sign (<1/4 mile away) is no longer visible from my window though. (second picture added at 4pm)