All proceeds from Ad Clicks goes to the author of this site.

 

Tuesday, January 09, 2007

Sun Ray, x4600, VMware ESX stories from the trenches

Two guys recently posted lots of information about how they are using Sun Rays and x4600's along with VMware ESX to power there infrastructure. to the Sun Ray Mailing List. The first is David Partington who is with the U.S. Army he has over 3000 Sun Ray's in use and reports saving upto 70% compared use normal thick clients. Lou Springer has posted several blog entries using the same technology though not as large as David's, but still an interesting read.

David's story:

We have close to 3000 Sun Ray's in production at Ft Huachuca. We have some Solaris Desktops, but the majority is Terminal Server Desktops. The problem with terminal servers is that the individual students dont have individual Windows IP's, and a few apps don't work well terminal services. To solve this issue we conducted a couple of pilots running Windows XP VM's with VMWARE ESX. The pilot was very successful and we were able to establish a baseline for scalability. What we determined was that we could run 80 users on a single 4600. Our (5)4600's have 64GB of Ram and 8 dual core CPU's. Each 4600 has 8 4GB FCAL ports attached to a 27TB file Server. For each VM session, we have allocate 2GB of RAM and 70GB of Disk. On the five 4600's we are provided 450 simultaneous XP VM's. The XP VM are served out to the Sun Ray's via the uttsc windows connector. The cost savings over a FAT client is 70%. I will post more results later.

Five Sun Fire 4600s with 8 network interfaces each = 40 -ESX VMSERVERS

Three Sun T2000 with 2 network interfaces each = 6 -Sun Ray Server 4.0

Three Sun X2200 with 2 network interfaces each = 6 -Windows 2003 Enterprise 64 bit/Terminal Servers with applications served from Softricity Soft Grid

Raritan VGA KVM over IP = 1 -Remote management for 2200 and 4600's

Raritan Serial KVM over IP = 1 -Remote management of T2000's

NetApp 3020 cluster = 4 - Filer 2 control Heads and 4 storage shells

Brocade Switch = 2 - 4GB Fiber Channel switches

Here is our volume plan for the filer.

FlexVol Name # of LUNs VM Size GB VOL Size GB LUN IDs
IGROUP
Walker1 40 25 2,500 101-140 ESX1A
Walker2 20 25 1,250 151-170 ESX1B
Walker3 45 25 2,813 201-245 ESX2A
Walker5 30 25 1,875 251-280 ESX2B
Walker6 30 25 1,875 301-330 ESX3A
Walker7 30 25 1,875 351-380 ESX3B
Walker8 30 25 1,875 401-430 ESX4A
Walker9 30 25 1,875 451-480 ESX4B
Walker10 30 25 1,875 501-531 ESX5A
Walker11 30 25 1,875 551-580 ESX5B

This allows space for each lun + 1 snapshot of each lun + 50% on each FlexVol. Snap reserve is set to 0% according to best practice.

We use one lun for each VM to allow for maximum bandwidth. This is a best practice, we did however try and put 10 VM's to a lun, and this had poor performance.

We also have one network CIFS file share as FlexVol named WHFILES with size
of 1,000GB. We use VIF single-mode with Gb interface connected to two redundant network switches on each Controller.

There is 1 port on board FC ports of Controller 1 connected to each of the 4 shelves. The on board FC ports of Controller 2 connected to the Out FC port on each of the 4 disk shelves. The other option is to go Active/Active (1 Aggregate per Controller) with each controller driving 2 shelves. Active/Active distributes the FC IO across more channels which improves performance.

For the Sun Ray Server I put all the IP's in the the host table. The way the user logs into the VM is I put an entry into the .cshrc file that equates to the user and the appriate VM. As an example if the user is using vmuser01, the users will login as that user. Vmuser01 will connect to the RDP VM running on the ESX Server that maps to vmuser01. The ESX Servers are all clustered, and as we fire up 100 VM's the load is distributed across all 5 4600's. The management of the VM's is done via the VM Console, all users are domain users and we run one real AD Server and one virtual VM AD Server.

We also run the Sun Ray Servers in Smart Card CAM Mode, if a user inserts a Smart Card I launch a Windows Terminal Server Session. I use Craig Benders terminal Server load balancing script, that is available from the Think Thin Site http://blog.sun.com/ThinkThin/entry/cam_mass_storage_workaround If you don't know who Craig is check out Blog http://blogs.sun.com/ThinGuy/category/Sun+Ray

The entire sollution sits in two racks, we will scale to 450 DTU's, and we are in the process of installing a system to support another 1200 dtu's with this configuration. The limitation of the ESX Server is not CPU, but memory. Our Future plans are to migrate to 128GB of RAM, onece the memory becomes available. If you have any questions or want more specifics let me know. We run both LAN dtu's and remote. We have 500 remote Sun Rays that are BootP relayed to the mothership (Sun ray server) from switches. We have more than 100 dtu's in a building, I put local Sun Ray servers. The windows servers are allways remote and accessed via RDP.

Take Care

David Partington
Ft Huachuca, Az


Lou Srpinger provides some more Sun Ray/x4600/VMware related blogs at:

I have some VMware stuff at
http://blog.louspringer.com/2006/10/25/esx-on-x4600/
http://blog.louspringer.com/2006/10/25/x4100-sunray-vmware-virtual-lab/
http://blog.louspringer.com/2006/10/27/the-hurdles/

10 Comments:

Anonymous other said...

David's setup is innovative and well designed. I like it. However I am under the impression it may be possible to achieve even better performances at a lower price while being more scalable.

What if processing power ("N" 4U 16-way x4600s running VMs) and storage (1 NetApp 3020 cluster) were consolidated ? For example he could just use "N*4" (to match the number of CPU cores) 2U 4-way Opteron servers with 8 local SATA disks each.

I think that such a solution would be:
- cheaper (SATA disks are roughly 50% the price of NetApp products per GB)
- faster (the VMs would run off local disks)
- more reliable (such a 2U server does not depend on the availability of a NetApp cluster to run VMs)
- more scalable (adding more VM servers can't degrade performance on a central storage cluster, since there is no cluster but only local SATA disks)

That's just my opinion. Maybe my solution doesn't meet some of his requirements :)

1:11 AM  
Anonymous Anonymous said...

I doubt it would be faster with local disks, although that would depend on how they were configured I suppose, and also what drives are being used in the netapp. 4Gb FC==400MB/sec, those internal SATA disks aren't going to BURST much over 150MB/sec unless they're WD raptors (which Sun does not sell afaik). Sustained will be lucky to push 80MB/sec for just about all modern SATA disks. Vs. a raid array full of 15k rpm 4Gb FC disks... I won't even get into the difference in IOPS.

5:04 PM  
Anonymous other said...

I think you underestimate the potential of modern NCQ SATA disks
with SATA controllers featuring a well-designed hardware interface
(AHCI, sii3124, Marvell 88SX6081 (used in the Sun x4500), etc).

Latency-wise, local SATA disks will outperform an netapp over FC.

Throughput-wise, I'll give you one example: the x4500 (not to be confused
with the x4600 used in David's setup) is a 4U server with 48 local SATA
disks. Sun has published benchmarks of the sustained read/write throughput
that can
be achieved with ZFS on top of these 48 SATA disks. They reached 2000-3000 MB/s.
Yes, Megabytes/sec. This demonstrates that the hardware
is capabable of a practical throughput 5x-7.5x higher than the theoretical
max throughput of 4Gb FC. So logically 8 SATA disks should be able to
sustain a throughput of 350-500 MB/s (with ZFS). The pb is to find a
software or hardware RAID implementation as performant as ZFS (I am sure it
does exist). Is it possible to run VMware (obviously not ESX) on top of
Solaris ? If yes, then ZFS could be used.

IOPS-wise, in my solution, each 16-way server would be replaced by four 4-way
servers. This means that the IOPS load of one 16-way server is spread on four
servers. In other words, each of my 4-way server would only need to handle
25% of the IOPS generated by a 16-way server. I lack numbers, benchmarks and
more information to say if this is feasible. But it doesn't seem totally
impossible.

6:51 PM  
Anonymous Anonymous said...

I'm also just wondering why you are using seperate KVM-over-IPboxes, all of the servers you mention, the X4600, the X2200 and the T2000 have service processors that implement KVM over IP functionality i the case of the X2200 and X4600 or in the case of the T2000 have the ability to access the serial line over the ethernet from the ALOM... that woulkd save a few thousand dollars in unneeded raritan KVM hardware ?

7:17 PM  
Anonymous Jim Klimov said...

Regarding shared storage (NetApp) vs. local disks: I believe there is a lot of rationale in favor of Storage.

The text mentioned that:
The ESX Servers are all clustered, and as we fire up 100 VM's the load is distributed across all 5 4600's

This is much easier to do when VMs are taken off the common shared storage. Local fast disks can be used for swap, rollbacks, whatever.

Besides, a specialized storage box can use dedicated and optimal XOR processors to implement RAID and logical CPU for data networking (FC, SCSI, CIFS, NFS, whatever), which may be more specialized for the task than a generic Opteron or Xeon. And dedicated to it, i.e. optimal for the expected data transfer rates/speeds.

Then again, the more disks work in parallel, the faster throughput you get. Might not be the case with multiple IO latency though (i.e. seek time for RAID5 remains close to seek times of its component disks).

While a single point of storage is also a point of failure (but it can be mirrored transparently with FC "splitters"), it is also a single point of data backup - which simplifies life (and lowers cost, if backup software products are purchased per box/CPU).

-----

Concerning the article itself: what I'd like to know is the typical workload of the boxes.

Is it a bunch of lazy Office suites which are sometimes loaded to read mail or write a report, or is it something "heavyweight"?

We do some in-house development and considered changing our C++/Java developers' workplaces from Intel desktops to SunRays.
So far the remote desktop (RDP or X11) task-sharing (Graphical IDE+compilations) has overloaded any single server we gave it.
Then again, our servers are low-end, and we're not as rich as to buy-and-test :)

So far we could only make a one-to-one mapping of developers to their old desktop boxes, which cleaned up the tables and moved the heat to server-room.

4:33 AM  
Anonymous Anonymous said...

This comment has been removed by a blog administrator.

11:18 PM  
Blogger buy said...

Cat5 extender allows you to extend using regular & inexpensive Cat5e or Cat6 UTP cable. These extenders include local & remote units. Cat5 Extender allows you to extend to long lengths over Cat5/6/7 and you no longer need to run long bulky coax cables. The available options are audio extenders, RS232 extenders,
DVI-D fiber optic cable Extender
, firewire optical extenders, USB extenders, USB KVM extender, Video Splitter extender via Cat5

and VGA extenders.

KVM extender allows you to extend PS2 keyboard, PS2 mouse and VGA video remotely up to 1000 feet away from a computer. KVM extenders are ideal for applications like public broadcasting, banks, classrooms, tradeshows, and other mission critical locations.

We also carry a wide range of DVI-D extender products like DVI-D Single link fiber optic extension cables, DVI-D repeater, DVI-D Extender Cat5, DVI-D Splitter Extender Cat5, DVI-D Single Link Fiber optic extender, DVI-D dual link fiber optic extender and DVI-D USB Fiber Optic extender.

9:14 AM  
Blogger products said...

China Wholesale has been described as the world’s factory. This phenomenom is typified by the rise ofbusiness. Incredible range of products available with China Wholesalers “Low Price and High Quality” not only reaches directly to their target clients worldwide but also ensures that wholesale from china from China means margins you cannot find elsewhere and buy products wholesaleChina Wholesale will skyroket your profits.wedding dressescheap naruto cosplayanime cosplay

9:26 PM  
Blogger products said...

Women’s nike tn Shox Rivalry est le modèle féminin le plus tendance de baskets pour le sport. tn chaussuresConcernant la semelle :Cheap Brand Jeans Shop - True Religion Jeans nike shoes & Puma Shoes Online- tn nike,Diesel Jeans le caoutchouc extérieur, l’EVA intermédiaire Levis Jeanset le textile intérieur s’associent pour attribuer à la.ed hardy shirts pretty fitCharlestoncheap columbia jackets. turned a pair of double plays to do the trick.Lacoste Polo Shirts, puma basket, Burberry Polo Shirts.wholesale Lacoste polo shirts and cheap polo shirtswith great price.Thank you so much!!cheap polo shirts men'ssweate,gillette mach3 razor bladesfor men.As for Cheap Evisu JeansCheap Armani Jeanspolo shirtsPuma shoes

9:27 PM  
Blogger theprophet said...

There was this guy who believed very much in true love and decided to take his time to

wait for his right girl to appear. He believed that there would definitely be someone

special out there for him, but none came.
Every year at Christmas, his ex-girlfriend would return from Vancouver to look him up.

He was aware that she still held some hope of re-kindling the past romance with him. He

did not wish to mislead her in any way. nike

shoxs
So he would always get one of his girl friends to pose as his steady whenever

she came back. That went on for several years and each year, the guy would get a

different girl to pose as his romantic interest. So whenever the ex-girlfriend came to

visit him, she would be led into believing that it was all over between her and the guy.

nike womens shoes The girl took all those rather

well, often trying to casually tease him about his different girlfriends, or so, as it

seemed! In fact, the girl often wept in secret whenever she saw him with another girl,

but she was too proud to admit it. Still, every Christmas, she returned, hoping to re-

kindle some form of romance. But each time, she returned to Vancouver feeling

disappointed.
Finally she decided that she could not play that game any longer. puma mens shoes Therefore, she confronted him and

professed that after all those years, he was still the only man that she had ever loved.

mens puma shoesAlthough the guy knew of her

feelings for him, he was still taken back and have never expected her to react that way.

He always thought that she would slowly forget about him over time and come to terms

that it was all over between them. nike shox

Although he was touched by her undying love for him and wanted so much to accept her

again, he remembered why he rejected her in the first place-she was not the one he

wanted. nike 360 air maxSo he hardened his heart

and turned her down cruelly. nike running

shoes
Since then, three years have passed and the girl never return anymore. They

never even wrote to each other. NIKE air shoes The

guy went on with his life..... still searching for the one but somehow deep inside him,

he missed the girl.
On the Christmas of 1995, he went to his friend's party alone. "Hey, how come all alone

this year? Where are all your girlfriends? What happened to that Vancouver babe who

joins you every Christmas?", asked one of his friend.nike air max He felt warm and comforted by his

friend's queries about her, still he just surged on.
Then, he came upon one of his many girlfriends whom he once requested to pose as his

steady. He wanted so much to ignore her ..... not that he was impolite,wholesale nike shoes but because at that moment, he

just didn't feel comfortable with those girlfriends anymore. nike shox torchIt was almost like he was being judged

by them. The girl saw him and shouted across the floor for him. Unable to avoid her, he

went up to acknowledge her.
"Hi......how are you? Enjoying the party?" the girl asked.
"Sure.....yeah!", he replied.
She was slightly tipsy..... must be from the whiskey on her hand.

1:08 AM  

Post a Comment

<< Home