synchronous NFS often seems faster even than iSCSI that might take like a minute to get all your data ondisk. we use HP EVA / XP for primary storage and NFS on NetAPP for DR here, so I dont have much firsthand data on NFS performance with VMWare, but I’m still not much surprised by your results. for high IOPS I’d feel better off with a NFS-NetAPP solution over a normal iSCSI solution getting hammered, but I think the price difference proper sized filer (not clone lol) vs. iSCSI array is what makes up much of the iSCSI selling point.

in my experience the filers never seem fast for a single host but beat everything by order of magnitudes under high load, so your choice seems a very wise one.

SHEP: I’ve been gloriously VDI ignorant, and plan to stay that way. Desktops == pain. No thanks.

What I will say is that *in general* with modern day ONTAP and drives, you’re better off with the highest spindle count RAID-DP aggregates that you can get away with, not lots of little hand crafted RAID4s or whatever. YMMV of course, but I’ve definitely had better experiences loading a bunch of flexvols on a big aggregate. Drives today are just too big for RAID4, period. Don’t forget to look into FlexShare if you’re concerned about hot hosts.

> The goal is to be able to handle a
> host failure (hardware, most likely)

Indeed! It used to be that you could count on your servers lasting 5 or more years and usually only have to replace them due to performance issues, not hardware failure.

Now, you can’t count on it. In recent years, we’ve had countless computers from good vendors fail prematurely after only 2 years. I guess 2 is the new 5? And when I say “fail”, I’m not talking about a drive in a RAID array. I’m talking about motherboard and power supply failures. We’ve been bit so many times by capacitor plague!

So, I think your idea can help ease the pain of poor quality servers, inexpensively. I guess the trick is to either have two NAS devices or have one device with redundant EVERYTHING. My colleague recently saw a device where every single part (including the controller) was redundant.

On doing some research on an NFS issue I came across some other information, which I’ll include here.

One research paper:
http://www.usenix.org/events/fast04/tech/full_papers/radkov/radkov_html/head.html

gives very mixed results, pointing out that iSCSI’s main disadvantage is it is
not for sharing a filesystem among many users *and* data loss if the client
crashes before flushing caches.

“Observe that the benefits of asynchronous meta-data update in iSCSI come at the
cost of lower reliability of data and meta-data persistence than in NFS.
Due to synchronous meta-data updates in NFS, both data and meta-data updates
persist across client failure. However, in iSCSI, meta-data updates as well as
related data may be lost in case client fails prior to flushing the journal
and data blocks to the iSCSI server.”

The only time I see it performing slowly in this paper is, in a scenario
when one client is communicating to one server, the data being communicated consists
mostly of file meta-data. In that case, the difference in speed is because serious
client-side caching is done (so less network traffic). Even then the paper admits
Linux NFS isn’t the best implementation.

“In principle, we expect iSCSI and NFS to yield comparable performance for write-intensive
workloads as well. However, due to the idiosyncrasies of the Linux NFS implementation,
we find that iSCSI significantly outperforms NFS v3 for such workloads.”

“Although the NFS client can also cache meta-data, NFS clients need to perform periodic
consistency checks with the server to provide weak consistency guarantees across client
machines that share the same NFS namespace. Since the concept of sharing does not exist
in the SCSI architectural model, the iSCSI protocol also does not pay the overhead of
such a consistency protocol.”

Trying nfsvers=4 (badarg), nfsvers=3 (arg out of range - acceptable range 1-2). Something not making sense.

Basically, I find the 2GB limit annoying — but I can’t believe, considering both machines have NFSv4 support built in (as well as v3), that it would default to v2. Shouldn’t I be able to specify nfsvers= 3 or 4?

Does anyone find the 2GB filesize limit of NFS3 annoying
to use with VMware?

NFSv2 has a maximum file size of 2GB
NFSv3 allows for 64-bit file offsets which results in file sizes measured in TBs

Of course with most iSCSI products it’s pretty quick and easy to map an iSCSI target from one host to another, so this capability isn’t unique to NFS. But you do have to do a bit more “work” if you choose to use iSCSI… So the NFS advantage is simplicity more than anything. You can certainly accomplish your availability goals with iSCSI or NFS.

I’d suggest you try both and see which you prefer.

On ESX 3 I too am getting the best results by mapping virtual drives directly to iSCSI targets on the netapp filer. Beats VMDKs on NFS mounts in every test I’ve done. But that config isn’t possible in the VMware Server product which artificially restricts you from directly using iSCSI-provided LUNs for disk mapping…

Hi Mark,

I’m able to use iSCSI volumes as raw disks for my VMs with VMware Server 1.0 on Red Hat Enterprise Linux 4 without any issues.

Am I missing something in your comment?

Thanks.

Tony

I could try the VMWare tests with an Adaptec iSCSI card, but to be honest I’ll be very surprised if there’s any significant difference in performance.

Also note that I’m actually quite happy with the NFS setup at this point, BTW. It’s quick and continues to be trouble-free.

Cheers

You can initiate snapshots via rsh/ssh on the filer, but only from an “admin host”. Which implies that root on that host can do pretty much anything on the filer… In my case, I initiate the vmware suspend commands (via their vmware-perl scripts) as well as the filer snapshot commands on my existing filer admin host.

It works well, but keep in mind that I am still early into the implementation. So, YMMV.

Note that when you create VMs, you don’t have to split the vmdks into 2GB files. Whether or not it’s a good idea to split vmdks seems to be a bit of a voodoo topic, irregardless of what your backend storage is.. Personally, I’m splitting them, if for no reason than it make emergency rsync’ing a lot quicker.

I’m seriously thinking of getting a Netapp unit for VMWare (and other duties). Is it possible to instantiate a netapp snapshot from a client?

In our case, I’d like to run vmware server on redhat. For backup purposes, I’m currently doing the following:

a) send a vmware-cmd suspend to each active VM.
b) taking an LVM snapshot of my vmware directory
c) activating every VM.

I’d like to replace (b) with taking a Netapp snapshot
of an NFS mounted vmware guest os directory.

Anyone have any good experience backing up VMs that way?

Thanks,
Rob