The gist of Trevor’s post is that there’s almost certainly going to be price war start in the IaaS (Infrastructure as a Service), or “Cloud Computing” space. The players (Amazon, Google, IBM, Microsoft) are massive, have huge war chests, and have the means (money) to destroy everybody else in order to establish themselves as dominant providers. There will probably end up only being a handful of providers (not unlike the server market today). That has some pretty big implications. The first casualties are likely going to be the traditional “hosting” companies that most web developers use today to deploy their web apps. They won’t be able to compete on cost. They won’t be able to compete on provisioning agility / scale, either. And scale is important (see below). So yeah, it’s gonna suck to be a ServerBeach or EV1 type provider sitting on datacenters full of aging, generic, power sucking, under-utilized PCs trying to sell them one-by-one while competing against an Amazon or Microsoft or Google who just built a $500M datacenter floating in the freakin’ ocean equipped with the latest in purpose built, power saving, virtualization nodes..  Apocalypse? Probably, yes. Another immediate casualty is going to be “control panel” software, like Ensim or Plesk and the shared hosting that it caters to. Shard is gonna be squeezed out of the market, attacked on the bottom end by these IaaS giants, and rendered irrelevant on the top end by smarter services. Things like Google App Engine, Reasonably Smart, GitHub, Campaign Monitor, Mosso, and Heroku are all examples the different kinds of “higher level hosting services” that are replacing the shared and dedicated hosts we used to use to do this kind of stuff. But control panel software is for pussies anyways, right, so why would a sysadmin care if they all eat it? Because the other casualty of the upcoming cloud era is going to be the traditional sysadmin. That’s right, as an industry we’re truly about to automate ourselves out of our jobs. Which, of course, is what we’ve been nobly aspiring to do for 30 years, but, folks, this time we’re actually going to do it and it’s going to happen at a massive scale as the whole industry shifts to Iaas/PaaS. This, I think, is a sea change event for sysadmins. Our roles as System Integrators will totally change. Got your attention now?

You already see it on every site that talks about cloud computing: Cost justifications showing how having to pay your own sysadmins cost soooo much money. There’s a lot of small shop sysadmins out there right now, working as employees or contractors, and the balance of power is being shifted to the hands of developers more than ever as they can now fire up VMs with a few lines of code. They don’t need sysadmins for basic server provisioning. What does it mean for us folks out there that have traditionally bought and/or managed “do it all” shared hosting or dedicated physical servers? It means we won’t be able to buy them, for one thing. And it means nobody will want us to, either. I can hear you now: “But they’re so flexible in the right hands!! I’m a scripting wizard and my dev peers and boss love me for setting up shit they can’t/don’t want to understand!! How do I continue to be valuable and solve problems?!?!”. So yes, a login on a dedicated box or typical shared host is flexible. Yes, you got postfix working with SASL and that was a cause for celebration and adoration. But let’s be real here. Things are changing. And with that in mind, there’s three things I think all good sysadmin types need to work on ASAP to continue to be valuable in the upcoming cloud computing era:

1. We have to accept and plan for a future where getting access to bare metal will be a luxury. Clients won’t want to pay for it, and frankly we won’t have the patience or time to deal with waiting more than 5 minutes for *any* infrastructure soon anyways. We have to assume “shared hosting” is going to die. Plan your exit strategy now! The good news is that we’ll still have access to “bare” virtual machines. I definitely don’t see that going away; VMs are the building blocks for everything else these brave new PaaS/SaaS providers are cooking up for the millions of web devs out there to consume. So our sysadmin skills and OS knowledge will still be required, but not by as many people/shops. Some of us will be building VM images. Most of us won’t be. If we don’t know virtualization inside and out, we’ll be out of work. Disposable VM images and better tools is going to mean the vast majority of sysadmins will be totally out of the loop when it comes to spinning sites/apps up and down.
2. We need to learn to program better, with richer languages. Why should the developers have all the fun? When you think about it sysadmins have a tremendous advantage: We know how the OS and computer work!. Most web devs don’t!  :)  But let’s not get cocky. The fact that we know how to run 6 different apps/services on the same machine or can tweak the UNIX scheduler to do our bidding or can swap cron implementations won’t matter. Instead of scripting processes and files on a box in bash, we’ll need to script integration between processes and data in different VMs. We’ll just as importantly need to script integration with services like storage (think S3), asynchronous messaging gateways (XMPP, AMQP), and talk to lots and lots of REST and SOAP APIs. Why? Because we sysadmins and systems engineers are going to be inventing all sorts of neat services that us and those pesky web devs (and ultimately enterprises) are going to eat up. Being able to integrate these new services and deployment options into existing solutions will very, very valuable to everyone. In short, the things we need to do SI on have programmable interfaces, so we’d better learn to talk to them directly.
3. Perhaps the biggest challenge we’re going to have is one we share with developers: How to deal with a world where everything is distributed, latency is wildly variable, where we have to scale for *throughput* more than anything else. I believe we’re only just starting to see the need for “scale” on the web and in systems work. We’ve gotten by on big centralized solutions for a long time as a profession and industry because it was simpler and in turn more reliable, but just like with processors “the free ride is over”. Us sysadmins are going to have to deal with gobs of VMs, gobs of remote services, gobs of “things” that need to run somewhere, somehow. We’re often going to have to give up some of the flexibility luxuries we’ve enjoyed (like the POSIX filesystem model) to reap the rewards of distributed systems. We’ll increasingly depend on messaging even for the simplest of scripting tasks, I think, as just one example. We need to start thinking differently about how we architect and implement the solutions our customers and peers are going to need.

So what can you do, in practical terms, today, to get ready for all this stuff? Start playing with a language like Ruby or Python and try doing some simple scripting in it instead of, say, bash. Play with stdin/stdout/stderr in the language so you can continue to build scripts The UNIX Way(tm). Add a REST or XMLRPC interface to that simple script so you can talk to it via a UNIX pipe and over the web. Fun! Try talking to EC2 and S3, via their cli tools and via a language library. Wrap your head around Git. Setup an XMPP server and write a little bot, oops, I mean “agent”, that lets you see that status of all your VMs/machines via that XMPP server. Cool, huh? Play around with Xen, KVM, VMware, xVM, whatever virtualization stuff you can. Figure out how to move a Xen image to/from EC2 and your local XenSource or xVM. Not so easy, huh? Finally, spend some time learning about HPC technologies like batch schedulers and Globus and distributed file systems like Lustre. There’s a LOT of tech and lessons from the academic HPC world that are incredibly relevant yet under-used and under-appreciated outside of that niche. Install Grid Engine or Condor across a few nodes and write a script to inject jobs for queued processing to get a taste. Instead of storing all that logging data for the new billing project on a centralized SAN, could you use kfs and hypertable? How many tens of thousands might that save the project? Try a test run on a cloud provider to proof-of-concept it. Now you’re talkin’. 

In short, all sysadmins are going to need to broaden our skill set to reach out across the network to speak to distributed systems and services. Centralized anything is dead. Successful UNIX sysadmins have always been good at scripting, we just need to take that scripting to the next level, and yes we’ll be doing a lot more of it. We’re going to have to take our HPC friends out for beers whenever possible and hear their battle stories of the dangers of latency and TCP. We’re going to need to learn to think differently and dive into distributed, event-driven architectures. We’re gonna have to play with some new stuff, we’re still going to have to be creative, and we’re going to have a lot of fun doing it!!

And that’s the ultimate take-away for me. I rarely get to see the physical servers anymore. I seldom setup or install software in an OS. I treat VMs like disposable containers. Verbs like “cloning” are part of my regular vocabulary. I now routinely work from within irb instead of a bash shell. I work with different constructs and ultimately my job is very different than it was just a few years ago. I’ve had to learn a lot of new stuff, and I still have far more to learn. Yet I’m having more fun now than I’ve had in years in this profession!! So don’t be shy, embrace the cloud. If you’re a UNIX sysadmin you already have the right stuff to succeed in this new world of utility on-demand computing, IMO.
The fact that I’ve started playing with Plan 9 and DragonFly again, is, my friends, a different post. 

Once the good weather is over I expect I’ll be back with more to say and share. There’s a sea change underway in terms of how the IT industry uses computers, and there’s a lot of interesting challenges ahead to truly enable “the cloud”. In the mean term, I am continuing to “microblog”. On Plurk, mostly. Sometimes Twitter too.  I particularly like Plurk’s ability to foster conversation with their unique time-line interface. Good job from the boy in Mississauga.
Finally, on an completely unrelated note, check out this video related to advertising and subliminal messaging. It will make you think about what you see on a daily basis and how it influences your buying decisions.
-Mark

None of us thinks Seattle is ever going to be much like Silicon Valley. We believe instead that what other cities can learn from Seattle is how to be different than the Valley, not the same.

There’s a bit of a movement afoot here in Vancouver to bring some of the excitement (and money) of the Silicon Valley startup scene to town. Glenn’s article brings some nice perspective that I think the Vancouver crew needs to keep in mind. There’s simply no place on earth like San Francisco / Silicon Valley, and keeping focussed on what makes Vancouver different is important.

So how is Vancouver different in an advantageous way? I think one thing, right now at least, is that it’s becoming increasingly difficult to bring foreign talent into the US. Glenn says this:

And this is what Michael [Arrington] loves about the Valley: that it calls out at dog-whistle frequencies to nerds across America, Russia, India and China. The single-mindedness of their migration belongs in National Geographic.

The current crackdown on H1-B Visas, and, IMO at least, America’s tactics on the global stage, are keeping some of these foreign geeks and entrepreneurs that have traditionally flocked to the Valley away. I had the privilege of going through university with a few people who are vasty smarter than me and went on to do PhD’s and do amazing research at top US schools. They’ve worked for Valley startups, and float back and forth between academia and “industry”. 3 out of the group of 6 of us have already moved back to Canada, one has moved to Dublin, and the last one is planning on moving to Montreal in the summer. And of course I worked in Seattle for 2 years, and currently work for a San Francisco startup *from Vancouver*. It’s a small sample set, yes, and a regime change in the next US election could change things entirely. But in the mean time, I think Vancouver needs to milk the fact that it’s easy to immigrate here, like Seattle there’s a variety of lifestyle advantages, yet Vancouver is still within striking distance to the Valley (there are frequent <2hr flights to SFO). Being close to center of the tech universe is absolutely critical, as Glenn points out:

The cult of the new may seem like madness but here’s the method to it: what’s often most difficult about developing a new idea is figuring out if it’s already an old idea. A business just like the one you’ve been dreaming of may already be forming within Google, or preparing to launch on its own.

When you and everyone you know spend 18 hours a day downloading, hacking, breaking, sharing, gossiping, criticizing and arguing about the Web, it’s easier to tell when an idea is truly new. And if you don’t, it’s almost impossible to catch up.

Vancouver, Seattle, and Portland all share the “quick access to the Valley” and “lifestyle” perks pluses. But only Vancouver comes with Canadian immigration policies and free healthcare.

One of Glen’s points I’m having trouble grappling with re Vancouver is the cost of life factor:

No one in the Valley can afford to grow up. Just as stressful environments delay the onset of sexual maturity in marsupials, a high cost of living – a two-bedroom house in Palo Alto typically costs more than $1.5 million — prevents people from buying homes and having children. In Silicon Valley, Seattle’s 28 year-old family man is still working his tail off for a hit.

Vancouver and Seattle both have seen dramatic increases in housing costs in the last decade. Certainly it’s nothing like the Valley or San Francisco, but it’s anything but cheap to live in the Pacific Northwest’s cities these days, and it looks like that trend isn’t going to turn around. What does that mean for startups in Vancouver? Are the demographics of who can form or be hired by a startup in Vancouver any different than the Valley? I’m not entirely sure. I think as an individual you can try a startup here with lower risk to your personal financial well-being (how anybody lives in California without healthcare is truly beyond me). Maybe that’s something. I don’t know.

The bottom line though? I can work for a startup while having a family in Vancouver. It certainly wouldn’t be impossible for me to do that in the Bay area, but it would be a lot harder. Is this a factor in how to build successful startups here? No idea, to be honest. And the truth of the matter is that I would still considering moving to the San Francisco area under the right circumstances. Perhaps that’s a hint of the ultimate solution then. Hybrid startups. Startups need to attract the best talent, from the biggest “pool” possible. The under 30s go to the Valley and sleep on the couch in crummy apartments and stay plugged into the scene. The international talent, and perhaps the talent that has or wants to start a family go to Vancouver where they can still be active within a vibrant tech community, yet can offer perks like healthcare and the outdoors to their families. Seems like a win-win to me.

I sometimes wonder what radio will become over the next, say, 20 years. I grew up with the radio on the kitchen table tuned to CBC pretty much every waking hour of the day. I was never a TV person. So it wasn’t much of a shift for me to embrace Podcasts, i.e. time-shifted radio. I don’t think “radio” will ever go away. After-all, story telling via the spoken word is what this whole human communication thing is built upon. Perhaps it won’t be called “radio” in a couple decades. But it will still be there. And that makes me happy.

From Derek, who responded to Jen, who stole it from Colleen.

Here are the ground rules:

1. After reading my answers, copy and paste the list into your comment.
2. Change my one-word responses with yours (ONE WORD, even if it kills you).
3. Submit your comment and/or post your finished list on your blog.

Your last meal: tarte
Something on your desk/work area: totoro
Your New Year’s Eve plans: family
The smallest gift you received this year: booger
The largest gift you received this year: parents
Something you wish you hadn’t eaten so much of during the holidays: pig
On your feet: toes
Your hair: fluffy
How many other countries you’ve traveled to: nfi
One country you dream of visiting: iceland
A hobby you’d like to take up/revisit this year: snowboarding
A hobby of yours that died (aww, buh-bye) this past year: freeriding
A publication you subscribe to (print): make
The most embarrassing subscription in your feed reader (if you have one): macrumors
One of your favorite stores to window shop dreamily in: soundplus
One of your favorite online stores to window shop dreamily on: ncix.com
A color you love to wear: red
Your bed pillow: kingsized
The color of your kitchen counter: greyish
What you plan to do when you get up from the computer: shit

Simple, but effective. I like it.

For Windows/Linux Mozilla Lab’s Prism is pretty much the same deal, but a little less “native feeling”.

For reasons that don’t make sense, I’m proud of the fact that these are all “straight off the camera”, i.e. no cropping or alterations of any sort. FWIW, I almost never do any editing of the photographs I take. I have nothing against post production work in the “real” or digital darkroom, btw. It’s just not my cup of tea, I guess.

I’d appreciate feedback on whether you like the VMUNIX Blues straying off the technical path from time to time. Or not.

I have to admit that I spent about 3 hours playing Peggle the other night after I ordered the Orange Box thing and Valve tossed it in for free.

I really like Ben’s reviews. His Bioshock review was awesome, even though I do think Bioshock is probably the best FPS gaming experience I’ve had since HL2 came out.

I’ve got such a massive game backlog right now I’m pretty sure I could take 3 months off and do nothing but play video games.

and my $100 Sony Ericsson W810i:

Bryan and Adam are pretty serious players. Which for a game of 8 man, two table ping pong, I think you pretty much have to be.

I can think of a few extra reasons why NetApp should port DTrace:

• It would entice me to move to GX from “traditional” ONTAP.
• ZFS is a pretty compelling WAFL competitor, but for many people it isn’t enough of a game changer on its own to warrant a move away from NetApp. ZFS + DTrace probes for NFS and iSCSI, however? Now that’s a one-two knockout punch. Add DTrace to GX and you’ve ducked that knockout.
• Linux and Windows are probably never going to have DTrace. Clustered Linux Storage On The Cheap(tm) (and, to a far lesser extent, Microsoft’s storage server) will be NetApp’s biggest mid to long term competitive pressure IMO.
• Let’s see DTrace on an EMC Symmetrix or Clarion.. exactly.

So yeah, NetApp should absolutely do DTrace. It’s a wonderful thing that Sun’s opened up and now we have FreeBSD and OS X joining Solaris with DTrace implementations. I think it would a great competitive move to put DTrace into ONTAP GX. Customers would love it.

Years in the making, the results of all this work are finally here! VMware Workstation 6 is shipping which includes VMware’s hypervisor, and Ubuntu backported the 2.6.21 VMI stuff for the 7.04 “Feisty Fawn” release! So… what’s it look like and is it any faster?!

In Workstation 6 you go to your VM Settings -> Options -> Advanced and select the “Enable VMware paravirtual kernel support” option. When the kernel boots, it spots the VMI interface:

[    0.000000] VMI: Found VMware, Inc. Hypervisor OPROM, API version 3.0, ROM version 1.0
[ 2344.765922] VMI timer cycles/sec = 1800100000 ; cycles/jiffy = 7200400 ;cycles/alarm = 18001000
[ 2345.230940] Booting paravirtualized kernel on vmi
[ 2348.826057] Time: vmi-timer clocksource has been installed.

I tested quite a bit software and development processes relevant to me to get a feel for performance. I won’t publish the full set of results because I don’t want it to be construed as a “benchmark”, but I will share one very simple test I did: I extracted and built Python.

Not bad! Basically any kind of load that generates system calls benefits the most. The python compile is mostly in user space where VMware was already fairly efficient, hence the relatively low 9% improvement. Overall, these are great results. I didn’t extensively test against the native OS since I do almost all my work within a virtual machine, but I can tell you that the paravirtualized kernel was typically between 10% and 20% slower than running on the bare hardware for the handful of simple tests I ran. Subjectively, everything felt snappier in X11/Gnome. Firefox in particular.

Kernel 2.6.22 is supposed to include Xen’s paravirt-ops interface. It’s great to see competitive cooperation in this space! I’d love to see OpenSolaris run on VMware’s hypervisor too, but in the short term at least it looks like we should get native Xen support before Solaris 11 ships. It sure would be cool if OpenSolaris could also support the paravirt-ops interface. I have no idea how technically feasible that is, however. I’m pretty amazed at just how good virtualized guest performance is getting. If I had any doubts about how prevalent hypervisors will be in the future they’ve been erased now. Exciting times in the virtualization world!

Topics include some F5 load balancer lovin’, OpenSolaris iSCSI target disapointment, with some GPL talk thrown in for good measure. It’s worth noting that I recorded this a month ago when we’d just finished running the iSCSI target through its paces and just couldn’t go forward with it as much as we really wanted to keep with our Thumper-backed ZFS/OpenSolaris architecture throughout. You can be sure, though, that we haven’t given up on zvol-based network block storage! Quite the opposite, actually, and we’ll be thrilled when it’s ready for prime time.

Oh, and coincidentally, I work for Joyent these days.

I’ve been reading more of Clayton Christensen’s “disruptive technology” material lately, and can’t help but wonder if NetApp has transitioned completely from the disruptive innovator to the sustainable innovator unable to see / unwilling to enter new markets. The new midrange FAS3000 models being a perfect example of a product line that will probably be a smashing success in existing accounts and no doubt continue to gain NetApp market share in the traditional enterprise space, but probably won’t make a single sale in the small but exploding “web 2.0″ world.

I’ve seen it first hand this week trying to secure a solid iSCSI target for a next generation web platform. We can’t buy StoreVault, VARs won’t sell it to us since it’s “for the SMED market and that’s definitely not you guys”, and we can’t buy FAS filers either since there’s not really a model or pricing structure that works for us.

Interesting, isn’t it, how these small new markets just aren’t worth going after when you’re so successful in your much larger established markets. Until, of course, they’ve grown to the point where they’re bigger than the incumbant market… I just can’t help but wonder if NetApp, perhaps along with every “network storage vendor” in the market today, are going to wake up in a couple years and wonder how they missed out on the entire “web as an application platform” revolution.

Which is why this idea Sun is tossing around is so bloody insane. The fact that executives at Sun don’t seem to realize just how risky even talking about it is worries me. The fact that they’ve actually framed the entire GPLv3 debate under the context of dual-licensing, like it’s a given? Well, that literally sent a chill down my spine. Go read the first page of responses on the forum. You can smell the fear. Fear that this isn’t a community project afterall. Fear that we’ve been betrayed. Fear that Sun actually thinks that what works for a development model like MySQL would work for OpenSolaris. Fear of what that implies for the community.

Rational or not, these types of thoughts are what emerge once licensing wars start to erupt. I’ve seen these holy wars before. Many have. They never end well.

I think what many people in the community have been getting caught up about is “Why choose GPLv3? What do we gain?”. Even as a relatively impartial outsider, I still haven’t formed a solid opinion on all the risks / rewards from that one. There’s a lot to consider, and it’s presumably why Sun opened the debate to being with. What I completely disagree with, however, is that this discussion should be taking place under the assumption that OpenSolaris could be successfully *dual* licensed. Like you could appease both license camps. What I am saying then, for the recond, is this:

Dual licensing OpenSolaris would instantly polarize and split the community, creating a world of code and community problems that could *never* be undone. I personally think it would sour the entire project, and ultimately be its undoing.

What needs to happen now? Well they need to nix the dual idea. Pronto. Then, Jonathan Schwartz himself needs to join the discussion and list the reasons why OpenSolaris would be better served by the GPL. As far as I’m concerned, even trying to explain how dual licensing might work is an excercise in futility, and the longer that idea lives the more harm will be done to the project.

Then we can move on to the more interesting discussion. Who gets to decide if OpenSolaris changes licenses?

Xserve RAID offers up to 10.5TB of high-performance storage at under $1.31 per gigabyte

The Apple Store is (as of 2PM) not listing this new config. I’m curious to see if this is simply a move to 750GB drives, or if it’s got the new FC-on-the-fan-card arrangement with all that that might imply.