Solaris 10 Networking Magic Revealed

While doing some serious performance analysis of Solaris 10’s (01/06) IP stack I was seeing a few cases where throughput was dropping off significantly for a short time, then picking right back up. I started digging for details on the new stack’s design, short of reading the raw code, and came across this great explanation of what’s changed in Solaris 10 compared to previous versions of Solaris.

Update: Sunay’s blog post has been turned into a feature on BigAdmin.

The fact that the only figures in the document are ASCII says everything about the target audience.

There’s some pretty interesting stuff here. The vertical perimeter / squeue concept was new to me, and there’s some important tidbits in there that I’m sure people out on the extremes of networking apps will want to pay attention to. In particular the ip_squeue_fanout global tunable and the fact that it defaults to “no fanout”. As we simultaneously move into the 10GigE and CMT era, I suspect this default may need to be revisisted. The new device driver framework, GLDv3, looks great and I’d be interested in seeing some sort of feature matrix showing off which chip / device driver support what features, like checksum offloading, fragment reassembly, etc. At the very least it’s motivated me to do some further tests comparing the Broadcom and Intel GigE cards to see if there’s any real-world performance differences for our intended uses.
Good stuff.