Sun Ray, x4600, VMware ESX stories from the trenches
Two guys recently posted lots of information about how they are using Sun Rays and x4600's along with VMware ESX to power there infrastructure. to the Sun Ray Mailing List. The first is David Partington who is with the U.S. Army he has over 3000 Sun Ray's in use and reports saving upto 70% compared use normal thick clients. Lou Springer has posted several blog entries using the same technology though not as large as David's, but still an interesting read.
David's story:
We have close to 3000 Sun Ray's in production at Ft Huachuca. We have some Solaris Desktops, but the majority is Terminal Server Desktops. The problem with terminal servers is that the individual students dont have individual Windows IP's, and a few apps don't work well terminal services. To solve this issue we conducted a couple of pilots running Windows XP VM's with VMWARE ESX. The pilot was very successful and we were able to establish a baseline for scalability. What we determined was that we could run 80 users on a single 4600. Our (5)4600's have 64GB of Ram and 8 dual core CPU's. Each 4600 has 8 4GB FCAL ports attached to a 27TB file Server. For each VM session, we have allocate 2GB of RAM and 70GB of Disk. On the five 4600's we are provided 450 simultaneous XP VM's. The XP VM are served out to the Sun Ray's via the uttsc windows connector. The cost savings over a FAT client is 70%. I will post more results later.
Five Sun Fire 4600s with 8 network interfaces each = 40 -ESX VMSERVERS
Three Sun T2000 with 2 network interfaces each = 6 -Sun Ray Server 4.0
Three Sun X2200 with 2 network interfaces each = 6 -Windows 2003 Enterprise 64 bit/Terminal Servers with applications served from Softricity Soft Grid
Raritan VGA KVM over IP = 1 -Remote management for 2200 and 4600's
Raritan Serial KVM over IP = 1 -Remote management of T2000's
NetApp 3020 cluster = 4 - Filer 2 control Heads and 4 storage shells
Brocade Switch = 2 - 4GB Fiber Channel switches
Here is our volume plan for the filer.
FlexVol Name # of LUNs VM Size GB VOL Size GB LUN IDs
IGROUP
Walker1 40 25 2,500 101-140 ESX1A
Walker2 20 25 1,250 151-170 ESX1B
Walker3 45 25 2,813 201-245 ESX2A
Walker5 30 25 1,875 251-280 ESX2B
Walker6 30 25 1,875 301-330 ESX3A
Walker7 30 25 1,875 351-380 ESX3B
Walker8 30 25 1,875 401-430 ESX4A
Walker9 30 25 1,875 451-480 ESX4B
Walker10 30 25 1,875 501-531 ESX5A
Walker11 30 25 1,875 551-580 ESX5B
This allows space for each lun + 1 snapshot of each lun + 50% on each FlexVol. Snap reserve is set to 0% according to best practice.
We use one lun for each VM to allow for maximum bandwidth. This is a best practice, we did however try and put 10 VM's to a lun, and this had poor performance.
We also have one network CIFS file share as FlexVol named WHFILES with size
of 1,000GB. We use VIF single-mode with Gb interface connected to two redundant network switches on each Controller.
There is 1 port on board FC ports of Controller 1 connected to each of the 4 shelves. The on board FC ports of Controller 2 connected to the Out FC port on each of the 4 disk shelves. The other option is to go Active/Active (1 Aggregate per Controller) with each controller driving 2 shelves. Active/Active distributes the FC IO across more channels which improves performance.
For the Sun Ray Server I put all the IP's in the the host table. The way the user logs into the VM is I put an entry into the .cshrc file that equates to the user and the appriate VM. As an example if the user is using vmuser01, the users will login as that user. Vmuser01 will connect to the RDP VM running on the ESX Server that maps to vmuser01. The ESX Servers are all clustered, and as we fire up 100 VM's the load is distributed across all 5 4600's. The management of the VM's is done via the VM Console, all users are domain users and we run one real AD Server and one virtual VM AD Server.
We also run the Sun Ray Servers in Smart Card CAM Mode, if a user inserts a Smart Card I launch a Windows Terminal Server Session. I use Craig Benders terminal Server load balancing script, that is available from the Think Thin Site http://blog.sun.com/ThinkThin/entry/cam_mass_storage_workaround If you don't know who Craig is check out Blog http://blogs.sun.com/ThinGuy/category/Sun+Ray
The entire sollution sits in two racks, we will scale to 450 DTU's, and we are in the process of installing a system to support another 1200 dtu's with this configuration. The limitation of the ESX Server is not CPU, but memory. Our Future plans are to migrate to 128GB of RAM, onece the memory becomes available. If you have any questions or want more specifics let me know. We run both LAN dtu's and remote. We have 500 remote Sun Rays that are BootP relayed to the mothership (Sun ray server) from switches. We have more than 100 dtu's in a building, I put local Sun Ray servers. The windows servers are allways remote and accessed via RDP.
Take Care
David Partington
Ft Huachuca, Az
Lou Srpinger provides some more Sun Ray/x4600/VMware related blogs at:
I have some VMware stuff at
http://blog.louspringer.com/2006/10/25/esx-on-x4600/
http://blog.louspringer.com/2006/10/25/x4100-sunray-vmware-virtual-lab/
http://blog.louspringer.com/2006/10/27/the-hurdles/
David's story:
We have close to 3000 Sun Ray's in production at Ft Huachuca. We have some Solaris Desktops, but the majority is Terminal Server Desktops. The problem with terminal servers is that the individual students dont have individual Windows IP's, and a few apps don't work well terminal services. To solve this issue we conducted a couple of pilots running Windows XP VM's with VMWARE ESX. The pilot was very successful and we were able to establish a baseline for scalability. What we determined was that we could run 80 users on a single 4600. Our (5)4600's have 64GB of Ram and 8 dual core CPU's. Each 4600 has 8 4GB FCAL ports attached to a 27TB file Server. For each VM session, we have allocate 2GB of RAM and 70GB of Disk. On the five 4600's we are provided 450 simultaneous XP VM's. The XP VM are served out to the Sun Ray's via the uttsc windows connector. The cost savings over a FAT client is 70%. I will post more results later.
Five Sun Fire 4600s with 8 network interfaces each = 40 -ESX VMSERVERS
Three Sun T2000 with 2 network interfaces each = 6 -Sun Ray Server 4.0
Three Sun X2200 with 2 network interfaces each = 6 -Windows 2003 Enterprise 64 bit/Terminal Servers with applications served from Softricity Soft Grid
Raritan VGA KVM over IP = 1 -Remote management for 2200 and 4600's
Raritan Serial KVM over IP = 1 -Remote management of T2000's
NetApp 3020 cluster = 4 - Filer 2 control Heads and 4 storage shells
Brocade Switch = 2 - 4GB Fiber Channel switches
Here is our volume plan for the filer.
FlexVol Name # of LUNs VM Size GB VOL Size GB LUN IDs
IGROUP
Walker1 40 25 2,500 101-140 ESX1A
Walker2 20 25 1,250 151-170 ESX1B
Walker3 45 25 2,813 201-245 ESX2A
Walker5 30 25 1,875 251-280 ESX2B
Walker6 30 25 1,875 301-330 ESX3A
Walker7 30 25 1,875 351-380 ESX3B
Walker8 30 25 1,875 401-430 ESX4A
Walker9 30 25 1,875 451-480 ESX4B
Walker10 30 25 1,875 501-531 ESX5A
Walker11 30 25 1,875 551-580 ESX5B
This allows space for each lun + 1 snapshot of each lun + 50% on each FlexVol. Snap reserve is set to 0% according to best practice.
We use one lun for each VM to allow for maximum bandwidth. This is a best practice, we did however try and put 10 VM's to a lun, and this had poor performance.
We also have one network CIFS file share as FlexVol named WHFILES with size
of 1,000GB. We use VIF single-mode with Gb interface connected to two redundant network switches on each Controller.
There is 1 port on board FC ports of Controller 1 connected to each of the 4 shelves. The on board FC ports of Controller 2 connected to the Out FC port on each of the 4 disk shelves. The other option is to go Active/Active (1 Aggregate per Controller) with each controller driving 2 shelves. Active/Active distributes the FC IO across more channels which improves performance.
For the Sun Ray Server I put all the IP's in the the host table. The way the user logs into the VM is I put an entry into the .cshrc file that equates to the user and the appriate VM. As an example if the user is using vmuser01, the users will login as that user. Vmuser01 will connect to the RDP VM running on the ESX Server that maps to vmuser01. The ESX Servers are all clustered, and as we fire up 100 VM's the load is distributed across all 5 4600's. The management of the VM's is done via the VM Console, all users are domain users and we run one real AD Server and one virtual VM AD Server.
We also run the Sun Ray Servers in Smart Card CAM Mode, if a user inserts a Smart Card I launch a Windows Terminal Server Session. I use Craig Benders terminal Server load balancing script, that is available from the Think Thin Site http://blog.sun.com/ThinkThin/entry/cam_mass_storage_workaround If you don't know who Craig is check out Blog http://blogs.sun.com/ThinGuy/category/Sun+Ray
The entire sollution sits in two racks, we will scale to 450 DTU's, and we are in the process of installing a system to support another 1200 dtu's with this configuration. The limitation of the ESX Server is not CPU, but memory. Our Future plans are to migrate to 128GB of RAM, onece the memory becomes available. If you have any questions or want more specifics let me know. We run both LAN dtu's and remote. We have 500 remote Sun Rays that are BootP relayed to the mothership (Sun ray server) from switches. We have more than 100 dtu's in a building, I put local Sun Ray servers. The windows servers are allways remote and accessed via RDP.
Take Care
David Partington
Ft Huachuca, Az
Lou Srpinger provides some more Sun Ray/x4600/VMware related blogs at:
I have some VMware stuff at
http://blog.louspringer.com/2006/10/25/esx-on-x4600/
http://blog.louspringer.com/2006/10/25/x4100-sunray-vmware-virtual-lab/
http://blog.louspringer.com/2006/10/27/the-hurdles/
posted by jamesd_wi at Tuesday, January 09, 2007
10 Comments:
David's setup is innovative and well designed. I like it. However I am under the impression it may be possible to achieve even better performances at a lower price while being more scalable.
What if processing power ("N" 4U 16-way x4600s running VMs) and storage (1 NetApp 3020 cluster) were consolidated ? For example he could just use "N*4" (to match the number of CPU cores) 2U 4-way Opteron servers with 8 local SATA disks each.
I think that such a solution would be:
- cheaper (SATA disks are roughly 50% the price of NetApp products per GB)
- faster (the VMs would run off local disks)
- more reliable (such a 2U server does not depend on the availability of a NetApp cluster to run VMs)
- more scalable (adding more VM servers can't degrade performance on a central storage cluster, since there is no cluster but only local SATA disks)
That's just my opinion. Maybe my solution doesn't meet some of his requirements :)
I doubt it would be faster with local disks, although that would depend on how they were configured I suppose, and also what drives are being used in the netapp. 4Gb FC==400MB/sec, those internal SATA disks aren't going to BURST much over 150MB/sec unless they're WD raptors (which Sun does not sell afaik). Sustained will be lucky to push 80MB/sec for just about all modern SATA disks. Vs. a raid array full of 15k rpm 4Gb FC disks... I won't even get into the difference in IOPS.
I think you underestimate the potential of modern NCQ SATA disks
with SATA controllers featuring a well-designed hardware interface
(AHCI, sii3124, Marvell 88SX6081 (used in the Sun x4500), etc).
Latency-wise, local SATA disks will outperform an netapp over FC.
Throughput-wise, I'll give you one example: the x4500 (not to be confused
with the x4600 used in David's setup) is a 4U server with 48 local SATA
disks. Sun has published benchmarks of the sustained read/write throughput
that can
be achieved with ZFS on top of these 48 SATA disks. They reached 2000-3000 MB/s.
Yes, Megabytes/sec. This demonstrates that the hardware
is capabable of a practical throughput 5x-7.5x higher than the theoretical
max throughput of 4Gb FC. So logically 8 SATA disks should be able to
sustain a throughput of 350-500 MB/s (with ZFS). The pb is to find a
software or hardware RAID implementation as performant as ZFS (I am sure it
does exist). Is it possible to run VMware (obviously not ESX) on top of
Solaris ? If yes, then ZFS could be used.
IOPS-wise, in my solution, each 16-way server would be replaced by four 4-way
servers. This means that the IOPS load of one 16-way server is spread on four
servers. In other words, each of my 4-way server would only need to handle
25% of the IOPS generated by a 16-way server. I lack numbers, benchmarks and
more information to say if this is feasible. But it doesn't seem totally
impossible.
I'm also just wondering why you are using seperate KVM-over-IPboxes, all of the servers you mention, the X4600, the X2200 and the T2000 have service processors that implement KVM over IP functionality i the case of the X2200 and X4600 or in the case of the T2000 have the ability to access the serial line over the ethernet from the ALOM... that woulkd save a few thousand dollars in unneeded raritan KVM hardware ?
Regarding shared storage (NetApp) vs. local disks: I believe there is a lot of rationale in favor of Storage.
The text mentioned that:
The ESX Servers are all clustered, and as we fire up 100 VM's the load is distributed across all 5 4600's
This is much easier to do when VMs are taken off the common shared storage. Local fast disks can be used for swap, rollbacks, whatever.
Besides, a specialized storage box can use dedicated and optimal XOR processors to implement RAID and logical CPU for data networking (FC, SCSI, CIFS, NFS, whatever), which may be more specialized for the task than a generic Opteron or Xeon. And dedicated to it, i.e. optimal for the expected data transfer rates/speeds.
Then again, the more disks work in parallel, the faster throughput you get. Might not be the case with multiple IO latency though (i.e. seek time for RAID5 remains close to seek times of its component disks).
While a single point of storage is also a point of failure (but it can be mirrored transparently with FC "splitters"), it is also a single point of data backup - which simplifies life (and lowers cost, if backup software products are purchased per box/CPU).
-----
Concerning the article itself: what I'd like to know is the typical workload of the boxes.
Is it a bunch of lazy Office suites which are sometimes loaded to read mail or write a report, or is it something "heavyweight"?
We do some in-house development and considered changing our C++/Java developers' workplaces from Intel desktops to SunRays.
So far the remote desktop (RDP or X11) task-sharing (Graphical IDE+compilations) has overloaded any single server we gave it.
Then again, our servers are low-end, and we're not as rich as to buy-and-test :)
So far we could only make a one-to-one mapping of developers to their old desktop boxes, which cleaned up the tables and moved the heat to server-room.
This post has been removed by a blog administrator.
Post a Comment
<< Home