VMware Server and NFS? Am I alone?

So the last few weeks I’ve been playing with VMWare Server. A lot. I’ve also been trying out the ESX version to see how it compares with the free offering. This is my first “serious” experience with VMware-style virtualization for server consolidation and I have to say that I’m quite impressed. Both products are solid, stable, and basically work as advertised. I’ve been running the free VMware Server (formerly GSX) product on Red Hat Enterprise 4 WS (Update 3), FWIW.

The plan, like everyone else in the universe, is to consolidate a bunch of 3 to 5 year old 1U servers that started out as application experiments and ended up becoming fairly critical parts of our shop. I’ve been testing (and will deploy) with a few 16GB quad-core X4100s. The goal is to be able to handle a host failure (hardware, most likely) and swing that host’s VMs over to another box in short order. Since most of these servers do very little IO, the natural choice is to host the VMs on one of the NetApp filers. A NAS setup, in theory, avoids having to copy VMs in the event of a host failure. I’ve also been experimenting with using the filers snapshotting abilities for nice “bare metal” recovery goodness. And this is where it gets a little interesting, because in all the literature, case studies, and chats I’ve had with current VMware customers it seems that everybody opts to use iSCSI. Not NFS.

Which, frankly, I don’t get. Cause in the testing I’ve been doing under RHEL 4 I’m getting better performance with the VMs on an NFS share than on an iSCSI LUN (off the same volume on the same filer). My theory is that with iSCSI I have to lay a journalled filesystem (ext3 in this case) across the LUN, and then of course inside the VM I’m also running ext3. I suppose I could switch to ext2 on the iSCSI LUN and/or the VM, but that just makes me nervous. Which means I’m “double journaling”. My guess is that the extra journalling is what’s making the iSCSI hosted VM run slower than the NFS hosted VM.

Stability wise, the NFS hosted VMs haven’t had a single glitch. I’ve pounded the piss out of the VMs, pulled the ethernet cable, forced reset the VM from the Server Console over and over, and outright yanked the power cord out of the host. So far everthing looks a-ok, with full filesystem checks on the guest OS showing no signs of problems.

Which begs the question, am I missing something? Why are there pages of forum posts and official docs and speaking tours highlighting iSCSI but hardly a mention of NFS-hosted VMs? Cause from where I’m standing NFS looks both simpler and faster than iSCSI for this job.

Note: I’m aware that the situation is different with ESX, where you really want to be using VMware’s special-purpose “VMFS” filesystem to hold your VMs’ virtual disks. There, iSCSI starts to make a lot more sense since you need to feed VMFS LUNs of some flavor or another. Whether or not VMFS-on-iSCSI actually makes much of a difference on ESX (vs, say, virtual disks on NFS) is something I’m going to be spending a few days testing next week. All I’ve tested so far is VMFS on local disks, and in that setup it is indeed faster than running vmdks on EXT3. Sadly, VMware’s license forbids me from actually disclosing anything even remotely resembling a “benchmark”. Sigh…

About this entry

You’re currently reading “VMware Server and NFS? Am I alone?,” an entry on VMUNIX Blues

Published:: Thursday, August 17th, 2006 at 12:38 am

Author:: mark

Category:: linux, netapp, redhat, storage, sysadmin, vmware, x4100

mark

August 29th, 2006
10:39 GMT

Hi Heath. Thanks for the explanation, but as I mentioned in my “note” the situation is different with ESX than with the free “Server” product from VMware. On ESX 3 I too am getting the best results by mapping virtual drives directly to iSCSI targets on the netapp filer. Beats VMDKs on NFS mounts in every test I’ve done. But that config isn’t possible in the VMware Server product which artificially restricts you from directly using iSCSI-provided LUNs for disk mapping… You have to use vmdk files and put them on an existing host filesystem.

I guess VMware wants to make sure people keep buying ESX, which is fine.

Note that I don’t have a performance “complaint” in either setup — quite the opposite in fact. I was simply pointing out that I haven’t seen much fanfare about hosting VMs on NFS which surprised me since in my testing it’s not only the simplest setup, but also the quickest.

Perhaps the reason for this is that most people with the cash to buy fancy NetApp filers (or an equivalent) aren’t going to blink an eye at ESX pricing, leaving the majority of VMware Server users with local disk as the only likely option. Or a Linux NFS server, which, in my experience, isn’t a a very good option…

So, while I certainly won’t disagree with you regarding NetApp’s iSCSI performance in general compared to other vendors, I don’t think that’s relevant in this scenario.

Thanks for the comment! I hope to give the EqualLogic product a try in the near future.

August 29th, 2006
10:59 GMT

Shit, I accidently deleted Heath’s comment….. Here it is:

Author: Heath M
URL: http://www.equallogic.com

I obviously work for an iscsi vendor and I can explain why your getting better results with NFS than iscsi. It’s because netapp is a great NAS device and a very poor FC or iSCSI box. All due to software RAID, WAFL on top of that and then FC or ISCSI riding on that. Our arrays consistantly beat netapp in iscsi performance and I have 30-40 customers in my area running ESX3 with our arrays and no performance complaints. Thanks for a great info site!

August 30th, 2006
11:03 GMT

Well, I’m trying to do some vmplayer testing with nfs served VMs for some experiments I’m trying to run, and all I get are vmware-vmx causing the kernel to oops.

seems I have less luck than the rest of you.

Paul

September 4th, 2006
6:05 GMT

Mark, we’re about to do EXACTLY what you’ve tried. We’ve been a Netapp NFS shop for years and years and years. We just purchased 2 copies of ESX Starter which we’ll run on 2 IBM blades using Netapp NFS for storage. I must admit your findings might change my plans slightly. I was originally thinking of running this simply as a test for lesser used hosts. The plan was to move to FC luns on two big quad, or 8, CPU IBM. While the extra CPU will be required your use of NFS might convince me to to forgo FC. Out of curiosity are you using FC or SATA drives? We have both and I would be curious as to the performance benefits of FC disk over the SATA drive, if any.

September 4th, 2006
9:25 GMT

Hi Paul, I haven’t tried with “pure” FC to the filer. Just iSCSI and NFS so far. I doubt I’ll bother to test FC since I try to avoid it wherever possible, and for our VMware deployment I/O performance isn’t really a big concern.

To be honest, it’s been my experience with FC to a filer in the past that it doesn’t really buy you much (any?) performance gain over iSCSI. FC to a NetApp filer is more about giving you flexibility if you have a setup that demands you run FC out to hosts or whatever. It’s not really about going faster.

You should give it a try yourself, Paul! I’d certainly like to know if you see any differences (performance, or otherwise). Who knows, perhaps the results will be surprising? But as I mentioned above, I wouldn’t be surprised if the FC and NFS performance weren’t substantially different.

I’m running these VMs to big aggregates of SATA disks. I did a quick test against a FC aggregate and basically I think the same generalizations you can make for FC vs. SATA hold true for VMs running on filers. I.e. FC seeks faster, SATA holds it own on sequential I/O. I’m convinced the FC vs. SATA differences have more to do with rotational speed of the platters than the protocols the drives speak… So no surprises when VMs are generating the I/O load instead of “native” hosts. Good.

I’m hoping to do some more testing soon and I’ll have a follow-up post. To date, I haven’t actually been “concerned” about the performance of the backend storage much. My testing has been more about getting a thorough understanding of how VMware Server (and ESX) behave. I’ve always found one of the best ways to deepen my understanding of any product is to push its performance limits looking for chinks in the armor. Edge case engineering, if you will. Happily, both VMware products have so far been entirely predictable on that front, which reassures me tremendously.

Jesse

September 11th, 2006
20:28 GMT

since iSCSI basically wrapps a SCSI packet in a TCP packet and sends it out, wouldn’t that mean the the overhead with ISCSI vs. NFS is significantly higher?

Not being a hard-core network guru, I could be wrong, but it’s a thought or two.

Then again, it may just be too late at night and I’ve not had nearly enough cafiene.

Jesse
http://www.sangod.com

(or maybe it’s that I have a bit of a predisposition towards FC)

roberts

September 26th, 2006
13:19 GMT

Does anyone find the 2GB filesize limit of NFS3 annoying
to use with VMware?

I’m seriously thinking of getting a Netapp unit for VMWare (and other duties). Is it possible to instantiate a netapp snapshot from a client?

In our case, I’d like to run vmware server on redhat. For backup purposes, I’m currently doing the following:

a) send a vmware-cmd suspend to each active VM.
b) taking an LVM snapshot of my vmware directory
c) activating every VM.

I’d like to replace (b) with taking a Netapp snapshot
of an NFS mounted vmware guest os directory.

Anyone have any good experience backing up VMs that way?

Thanks,
Rob

September 26th, 2006
16:11 GMT

Rob, that’s pretty much exactly what I’m doing.

You can initiate snapshots via rsh/ssh on the filer, but only from an “admin host”. Which implies that root on that host can do pretty much anything on the filer… In my case, I initiate the vmware suspend commands (via their vmware-perl scripts) as well as the filer snapshot commands on my existing filer admin host.

It works well, but keep in mind that I am still early into the implementation. So, YMMV.

Note that when you create VMs, you don’t have to split the vmdks into 2GB files. Whether or not it’s a good idea to split vmdks seems to be a bit of a voodoo topic, irregardless of what your backend storage is.. Personally, I’m splitting them, if for no reason than it make emergency rsync’ing a lot quicker.

Tony

October 3rd, 2006
10:25 GMT

Mark — Did you use a TOE? If not the scsi cmd conversion taxes your box hard. Maybe this is your bottleneck. Just a thought- I do not have any iSCSI, just FC.

Cheers

October 3rd, 2006
11:02 GMT

Hi Tony. No, I’m not using an iSCSI TOE card. In all my testing with iSCSI to date, they seem to be of dubious value. In most cases I see very little (if any) performance increase using a TOE card, and only about 10% (at most) reduction in CPU usage on the host. This is of course with recent Opteron CPUs. The situation would likely be different on slower CPUs, of course.

I could try the VMWare tests with an Adaptec iSCSI card, but to be honest I’ll be very surprised if there’s any significant difference in performance.

Also note that I’m actually quite happy with the NFS setup at this point, BTW. It’s quick and continues to be trouble-free.

October 3rd, 2006
20:51 GMT

Mark,
That is good to know; trouble-free is what I know I am striving for!
We were to go iSCSI, and then got into an EMC CX3-20, including redundant switches and hba for the same price. So the cash made that decision easy. We are only using the SAN for the virtual environment.
Do you know anyone who has setup San replication (boot luns, and vmfs luns) for redundant site (different geographic locations)? VI3 will be the chose virtualization suite. Cheers,

thewolf

October 31st, 2006
14:19 GMT

On ESX 3 I too am getting the best results by mapping virtual drives directly to iSCSI targets on the netapp filer. Beats VMDKs on NFS mounts in every test I’ve done. But that config isn’t possible in the VMware Server product which artificially restricts you from directly using iSCSI-provided LUNs for disk mapping…

Hi Mark,

I’m able to use iSCSI volumes as raw disks for my VMs with VMware Server 1.0 on Red Hat Enterprise Linux 4 without any issues.

Am I missing something in your comment?

Thanks.

Jay

January 16th, 2007
8:49 GMT

How is this going? Are you using this set up in production? I’d like to use this technique to minimize the downtime involved in moving a vm from one host to another. I’m looking at NFS because, from my understanding, it’s a no no to have 2 iSCSI initiators connecting to one target.

January 16th, 2007
10:49 GMT

Hi Jay, yes I’m using this NFS setup for production and it’s been solid so far.

Of course with most iSCSI products it’s pretty quick and easy to map an iSCSI target from one host to another, so this capability isn’t unique to NFS. But you do have to do a bit more “work” if you choose to use iSCSI… So the NFS advantage is simplicity more than anything. You can certainly accomplish your availability goals with iSCSI or NFS.

I’d suggest you try both and see which you prefer.

John Martin

August 21st, 2007
0:03 GMT

roberts 09.26.06 / 1pm

NFSv2 has a maximum file size of 2GB
NFSv3 allows for 64-bit file offsets which results in file sizes measured in TBs

August 25th, 2007
1:38 GMT

John: As you already said, there is no 2GB limit on NFSv3, so, no, I don’t find it annoying.

Peter

October 21st, 2007
21:38 GMT

Hi,
We have a SAN setup (IBM DS3400) I’m looking at a cheap NAS NFS device to use as for disaster recovery. having copies of the VMs etc on the NAS NFS device. the question I have is, wil lany NAS device do that can give NFS. the following webpage shows a device that is cheap and can give NFS connection. http://www.shopping.com/xPO-Fantom-Drives-Fantom-Drives-G-Force-1-TB-USB-MegaDisk
I was thinking about useing a device like that for backuping up to and if the SAN every fails then to run around 6 VMs … what are your thoughts .. instead of having to spend $2,500 odd on a NAS you can get something for under a $500 ..

October 21st, 2007
21:42 GMT

forgot to menation VMWARE ESX Vi 3.02 currently just waiting for 3.5

l a

November 19th, 2007
21:37 GMT

I keep hitting a 2GB limit with NFSv3 as used/provided in linux. The kernel source has options for v3 & v4 — I have both checked in servers and clients. Doesn’t make sense to me, but _my_(misconfigured by default it seems) NFS can’t access files larger than 2GB.

Trying nfsvers=4 (badarg), nfsvers=3 (arg out of range - acceptable range 1-2). Something not making sense.

Basically, I find the 2GB limit annoying — but I can’t believe, considering both machines have NFSv4 support built in (as well as v3), that it would default to v2. Shouldn’t I be able to specify nfsvers= 3 or 4?

Scott

December 13th, 2007
15:02 GMT

I wondered the same thing, and had an additional irritation here as somebody started parroting “NFS is slow” and pretty soon it was believed without confirmation. It seemed to be attaining somewhat of an urban legend status.

On doing some research on an NFS issue I came across some other information, which I’ll include here.

One research paper:
http://www.usenix.org/events/fast04/tech/full_papers/radkov/radkov_html/head.html

gives very mixed results, pointing out that iSCSI’s main disadvantage is it is
not for sharing a filesystem among many users *and* data loss if the client
crashes before flushing caches.

“Observe that the benefits of asynchronous meta-data update in iSCSI come at the
cost of lower reliability of data and meta-data persistence than in NFS.
Due to synchronous meta-data updates in NFS, both data and meta-data updates
persist across client failure. However, in iSCSI, meta-data updates as well as
related data may be lost in case client fails prior to flushing the journal
and data blocks to the iSCSI server.”

The only time I see it performing slowly in this paper is, in a scenario
when one client is communicating to one server, the data being communicated consists
mostly of file meta-data. In that case, the difference in speed is because serious
client-side caching is done (so less network traffic). Even then the paper admits
Linux NFS isn’t the best implementation.

“In principle, we expect iSCSI and NFS to yield comparable performance for write-intensive
workloads as well. However, due to the idiosyncrasies of the Linux NFS implementation,
we find that iSCSI significantly outperforms NFS v3 for such workloads.”

“Although the NFS client can also cache meta-data, NFS clients need to perform periodic
consistency checks with the server to provide weak consistency guarantees across client
machines that share the same NFS namespace. Since the concept of sharing does not exist
in the SCSI architectural model, the iSCSI protocol also does not pay the overhead of
such a consistency protocol.”

Joe Kelly

March 11th, 2008
9:45 GMT

Nice little story and an interesting idea. We’re considering using this solution here at work (and maybe at home too).

> The goal is to be able to handle a
> host failure (hardware, most likely)

Indeed! It used to be that you could count on your servers lasting 5 or more years and usually only have to replace them due to performance issues, not hardware failure.

Now, you can’t count on it. In recent years, we’ve had countless computers from good vendors fail prematurely after only 2 years. I guess 2 is the new 5? And when I say “fail”, I’m not talking about a drive in a RAID array. I’m talking about motherboard and power supply failures. We’ve been bit so many times by capacitor plague!

So, I think your idea can help ease the pain of poor quality servers, inexpensively. I guess the trick is to either have two NAS devices or have one device with redundant EVERYTHING. My colleague recently saw a device where every single part (including the controller) was redundant.

I forgot to ask, were you testing this on a gigabit or a 100 megabit LAN?

SHEP

May 4th, 2008
18:01 GMT

Mark, we are looking at running 100 vm’s (XP Pro) on a Netapp Filer and were wondering if you have tested VDI with your stuff. How many SATA drives do you have in your aggregate? We are looking to do 6 under each controller and using RAID 4. How much data do you see moving from the ESX hosts to the filer when all things are running?

May 4th, 2008
23:06 GMT

Joe: Gigabit.

SHEP: I’ve been gloriously VDI ignorant, and plan to stay that way. Desktops == pain. No thanks.

What I will say is that *in general* with modern day ONTAP and drives, you’re better off with the highest spindle count RAID-DP aggregates that you can get away with, not lots of little hand crafted RAID4s or whatever. YMMV of course, but I’ve definitely had better experiences loading a bunch of flexvols on a big aggregate. Drives today are just too big for RAID4, period. Don’t forget to look into FlexShare if you’re concerned about hot hosts.

darkfader

May 16th, 2008
8:34 GMT

Hi,

synchronous NFS often seems faster even than iSCSI that might take like a minute to get all your data ondisk. we use HP EVA / XP for primary storage and NFS on NetAPP for DR here, so I dont have much firsthand data on NFS performance with VMWare, but I’m still not much surprised by your results. for high IOPS I’d feel better off with a NFS-NetAPP solution over a normal iSCSI solution getting hammered, but I think the price difference proper sized filer (not clone lol) vs. iSCSI array is what makes up much of the iSCSI selling point.

in my experience the filers never seem fast for a single host but beat everything by order of magnitudes under high load, so your choice seems a very wise one.

Vic

May 16th, 2008
9:45 GMT

I’m using ESX3 for the very first time. I have 6 nics on a SUN blade and have decied to 3 teams of 2 NICS. I have been tolod I have to use NFS for datastore, so thats one team used for that, the second team will be for accessing VMs on ESX blade and the third team for mgmt, ie vmotion, DRS and service console. Each team will map onto its on vSwitch, one for storage, one for the vm’s and one for mgmtment Is this a vaild configuration or does the NFS and Vmotion have to be in the same Team?
All feedback appreciated.

VMware ESX на NFS - новый хит или экзотика? | about NetApp

August 31st, 2008
22:55 GMT

[...] Mark Mayo VMware Server and NFS? Am I alone? [...]

VMUNIX Blues

VMware Server and NFS? Am I alone?

About this entry

27 Comments

Have your say

About

Recently

Categories