Well after working with ESXI from version 3.5 onwards in our environment and using different storage options like SCSI drives in raid 5, SAS drives in raid 6 and SATA RE drives in raid 6 and 10 we as company found ourselves in the position of needing even more server consolidation on bigger scale.
So what to do. Well we bought ourselves 2X HP DL380 G7 Server, pretty much only the RAM,CPU’s and 2X quad port nics added to existing 4 ports. Thus having 96 GB Ram 24 CPU Cores and 12 Nic ports. One HP Procurve 2910 switch to separate our iSCSI network from out LAN. 2X Next Gen Core i3 Intel cpu based datastore servers with Intel’s S1200BTS Mobo and one intel quad port nic added to give 6 nic ports plus 8Gigs of RAM and Adaptec’s 6508 Raid controller to make sure we have SAS 2 / SATA 3 support for the future.
So the next question was, what would be our best setup for the datastore iSCSI?
There where options FreeNas,Openfiler,Windows based Starwind service and, Windows itself.
Being involved with ESXi and the concept of minimizing overheads on the host OS I started looking at Openfiler and FreeNas. Downloaded both and started testing. Liked what I saw with FreeNas and stuck with it. After a few days of running on FreeNas with nic Teaming and also Round Robin enabled on ESX I still felt like I was missing some config settings some place due to low data rates while backing up my data.
Backing up: I know there are a lot of concepts around snapshots and fast backup methods, but I still believe the best the best backup is a complete offline backup while all is down. Especially in my environment where we can still shutdown servers at night and back them up.
My understanding from nic teaming was that once one team 4 1Gbps nics you should get 4 Gbps out of that team. Losing some speed due to overheads and processing packets etc. I would have been satisfied with 3 Gbps but when I did a simple Copy of a 20GB vm from one datastore to a other I got round about 25 MB/s which was far below 375 MB/s of a 3Gbps connection. So investigating why.
Not being a expert on FreeNas testing and changing settings was not a good place for me. Thus I switched to Windows 7 with Starwind’s iSCSI service for the iSCSI part. Did nic teaming with INTEL’S nic teaming feature and started testing again.
3 Raid sets in datastore server: 1 Kinston SSD 512 GB X4 drives in raid 0 on adaptec, 2 Seagate Sata 1 TB X4 Drives in Raid 0, one one adaptec and the other on the Mobo’s Intel Raid, just the check differance on performance between Mobo and Adaptec.
Kingston SSD, and it was only the V100 + editions, had a huge degraded write performance, started on +- 800 MB/s but by the end of the 20 GB file copy it ended at 420 MB/s which is still damm fast. Sata Raid sets gave 380MB/s for mobo and 400 MB/s for adaptec. Which indicates that while you get a good Server Mobo you can use the onboard Raid controller. O yes if you test file copy rates dont use Teracopy it has a limit or “bug” that it can???t copy faster than 72 MB/s or maybe it was design that way.
With testing the transfer rates done and happy with getting that speeds in writes next was the iSCSI testing on different setups.
1. 4 Nics teamed as one on the Datastore server. Hp Switch Trunked with LACP on the datastore trunks. 4 Nic setup for iSCSI on ESXi and all assigned to Software iSCSI. Target formatted as VMFS 5. Copying the same 20GB of data from a single SSD Kingston which tested good for 230MB/s read rates. Only got about 40 MB/s write rates on the all 3 different datastores.
Did some investigation and found that some people had success by changing the IOPS to 1 on a device on ESXi which is using round robin method of load balancing.
Play around with and found that if I change mine to 1000 or 10000 that is where I get most out of the write rates (85MB/s). But on my HP Switch I noticed when IOPS was set to 1 or 2 the usage per port was on about 10 ??? 15 % but all 4 at the same time, and this was giving me 40 MB/s but when on 1000 or 10000 the switch indicated that only one port was used for 20 seconds then the next one and so forth.
This was a indication to met that ESXi with the use of iSCSI still use the ports individual but when the IOPS is low it looks like all is working as one.
Further I did some test on the Windows side where I copied the some 20GB data from one datastore server to the other on the iSCSI with Nic teaming and got a write speed of 250MB/s.
Thus my conclusion. ESXi with iSCSI and Round Robin method of load balancing doesn???t work like most people think it is suppose to and I don???t know what there intentions was with it.
It is better to setup multiple iSCSI target bounded to different nics so that ESXi will still use each nic separately but you don???t share the bandwidth with other vm???s running on the same datastore.
My plan now is to have 4 targets fro 4 diffent VM???s
1 for Exchange
1 for SQL
1 for Normal file/ print servers
1 for al my Terminal servers
If anyone has any comments please don???t hesitate to give some and/or know where I went wrong in setting up something please let me know. Hope this answer some questions for someone
