NetApp autosupport rocks

It’s no secret that I love a good many things about NetApp. One of the absolute top reasons I continue to buy and recommend NetApp filers is their fantastic tech support, in particular the “AutoSupport” system built into the filer. Here’s an example.

6:27PM - I racked a full disk shelf into the cabinet of an existing filer. I hooked everything up (fibre channel connections, power), set the shelf ID (5, in this case) and made sure termination was on for the shelf. Powered the shelf up and confirmed that all status lights were green.
6:29PM - I flipped termination on the previous last shelf (ID 4) off. The fibre loops now extended up to the new shelf and the filer instantly picked up the additional 14 disks and built the redundant loop paths to them. The filer also, however, checked the status of each disk and discovered that a bunch of the disks were previously in another “aggregate” (NetApp speak for a pool of disks protected by the same RAID layout). My filer knew this was a foreign aggregate (i.e. it was created on another filer), and although it confirmed that all the disks required by the aggregate were available on the shelf, it put the aggregate into an offline state to protect what could be valuable data, and generated an error message.

Now, you might be thinking “well, that’s nice. you added the shelf live and the machine actually did some intelligent things before blindly making the disks available for use and potentially destroying a bunch of data. oh and you got a nice error message.”. And you would be correct. It is nice.

But the filer didn’t just generate an error on the console. It also sent the Systems team an email outlining the problem — “Hey! I just discovered a pile-o-disks that belong to another system! You now have a volume here that’s offline!”. It also sent an HTTP post to open a trouble ticket in our internal issue tracker. And it automatically opened a support case with NetApp detailing the error (also via a HTTPS post). Which leads me to the next event in the timeline:

6:37PM - My phone rings and NetApp informs me that they’ve received an autosupport from the filer indicating “RAID VOLUME FAILED”. I confirm that I’ve received the email as well. He says “it looks like you’ve added shelf, is that correct?”. Why yes, yes it is. He then says “Ok, just making sure. Do you know how to online a foreign aggregate, or are you planning on using those disks to be part of an existing volume?”. I confirm that I’m going to blow the disks away and buddy makes sure I know the “aggr destroy” command. Next question from NetApp guy: “From the autosupport it looks like one of the ESH modules had an old firmware. The system tried to upgrade it, but failed. I’ve opened an RMA, do you want the part delivered immediately, or can we send it over in the morning?”

Now, ignoring my failure to spot the second autosupport message indicating the shelf firmware upgrade attempt had failed (my brain must have wrote it off as being another message about the offline volume..), is that awesome or what?!? Not only did I get immediate over the phone confirmation of the problem which made me feel much more secure about issuing the commands to reclaim the disks, but it caught a problem that I hadn’t even picked up on.

This is so much smarter behaviour than my IBM FAStT or Sun / Dot Hill storage it’s not even funny. And it’s just the tip of the iceberg of why dealing with “smarter” NetApp filers is soooo much better than the braindead products of many of their competitors. Ever since I first saw autosupport in action I wondered how long it would be before the other vendors took the same approach. Here we are, a decade later, and I continue to be well served by autosupport and continue to be astonished that it’s still largely a NetApp exclusive.

Bottom line: NetApp autosupport rocks, and saves my ass yet again.

About this entry

You’re currently reading “NetApp autosupport rocks,” an entry on VMUNIX Blues

Published:: Monday, January 23rd, 2006 at 7:21 pm

Author:: mark

Category:: general, netapp, storage

mark

January 23rd, 2006
23:23 GMT

Note to the guy who emailed me anonymously saying, amongst other name calling, “netapp is for retards. a box will never be smarter than me. and making you hook the filer up to the net is inherantly [sic] insecure.”:

Either you’re a moron, or you didn’t read my post. The point is that a HUMAN looked over the error report and called me, another HUMAN, to make sure everything was a-ok. The post was titled “autosupport rocks”, not “netapp self-healing rocks”. Got it? Secondly, the filer doesn’t refuse to bloody boot if it can’t generate an autosupport call. In fact, aside from just firewalling it off (you do use network access lists, don’t you Mr. Secure?) you can actually TURN AUTOSUPPORT OFF. Imagine!

Doug Frith

January 24th, 2006
5:59 GMT

Yeah, autosupport is unique and can be implemented in a secure fashion (or turned off it that turns your crank). Being in a highly hostile ISP/telco environment for 10 years (starting with the F3XX line) I can speak from experience that it has saved my butt several times. Even basic “sent me an email when stuff happens” functionality built right in without having to dedicate a managment box (running pricey software, you know who you are) rocks.

Cheers

jbctech’s web log

January 24th, 2006
12:56 GMT

[...] The support team takes autosupports very seriously, and Mark’s article is a perfect example. I encourage you to read his article here, but basically the idea is this: Mark added a new disk shelf to an existing filer. When he added the new disks, the filer noticed that the disks were previously in another aggregate and immediately marked the aggregate as offline to preserve data. It then generated an error to inform the admin that the disks are offline. Well, it sent out a few emails, and also submitted a trouble ticket with NetApp Global Support (NGS). He received a call within EIGHT minutes from NGS and they were ready to walk him through the process to “online” his newly added disks. NGS also received information that a firmware update failed, and they were ready to ship him a new part. It’s just another example of NetApp striving to be “Fast, Simple, and Reliable.” [...]

Ben Eng

January 24th, 2006
17:35 GMT

Customers are so afraid of their software phoning home to enforce digital rights management, software license agreements, or reporting usage statistics. Of course, there are privacy concerns. auto-support seems to provide such a huge benefit that it overshadows any privacy concerns. Just noting a difference in perception.

January 26th, 2006
9:38 GMT

That’s a good point, Ben. And sysadmins are probably more conditioned than most when it comes to fearing software that “phones home”, which might explain why someone could react negatively to autosupport.

There will certainly be customers that don’t want their filers contacting NetApp, for any number of reasons. You can get the filer to email you when a problem happens but not contact NetApp HQ, of course. So “autosupport” doesn’t have to be all or nothing. Which is good. But I’m pretty sure that by default it will send autosupport messages back to NetApp tech support.

One thing to keep in mind is that these are fairly expensive storage systems with support contracts to keep them running. So the customers are paying for this service, and I can’t imagine the whole autosupport thing wouldn’t have been made crystal clear during the sales process. It’s usually front and centre in the sales pitch. It’s also very typical that first time customers have the local SE sent over to help them with the install, and that always includes going over autosupport, initiating a test autosupport and seeing what it transmits, etc. At least, that’s been my experience in both Toronto and Vancouver.

But I’m sure it’s still possible that autosupport takes someone by surprise. I honestly have no idea if the potential privacy implications are layed out in the quick start guide, or at all. When we get our next filer I’ll take a look and see what’s explained and what’s not.

rajeev

February 21st, 2006
19:15 GMT

many a time, a disk would fail on a sat evening, while the admin is busy catching up on his favorite sci-fi show (or whatever it is the admins do on Sat night). Come Mon 8 am, the admin walks into the door to find out a replacement disk sitting outside his office door. It is goodness all over.

-nuff said.

VMUNIX Blues

NetApp autosupport rocks

About this entry

6 Comments

Have your say

About

Recent Comments

Recent Posts

Categories