Networking

A look at an iSCSI-based highly available architecture

High availability is an important consideration in any service and becomes more important all the time. Scott Lowe shared with you a look at the highly available storage infrastructure implemented at Westminster college.

High availability is an important consideration in any service and becomes more important all the time.  Scott Lowe shared with you a look at the highly available storage infrastructure implemented at Westminster college.

-------------------------------------------------------------------------------------------------------------------- 

As I tend to say a lot, I've written quite a few postings in this blog about the EMC AX4 iSCSI SAN we installed at Westminster College over the summer.  In my last posting, I shared with you a positive story regarding a minor failure of one of my AX4's storage processors that resulted in our high availability planning paying off big time.

In this posting, I'm going to share with you a snapshot of our highly available storage architecture at Westminster College and what makes it tick.

Westminster storage infrastructure 

So, what are you looking at?  It's a look at how a single server is cabled to our storage infrastructure.  At the top of the diagram sits our AX4.  We have two physical arrays; the SATA unit doesn't have any brains.  It's simply cabled to the primary unit, which houses dual controllers/storage processors, each containing dual gigabit Ethernet adapters.  Each of the physical array enclosures is connected to a separate UPS, each connected to separate electrical circuits.

As I indicated, each storage processor includes a pair of gigabit Ethernet ports used to pass iSCSI traffic between the array and any connected servers.  Each port on each controller is connected to one of two Ethernet switches in a mesh-type configuration.  In our scenario, we currently have the AX4 cabled to a pair of Dell M6220 blade-based Ethernet switches.  Each of the blades has redundant uplinks to our HP ProCurve 5412zl.  In order to keep the diagram uncluttered, this uplink scenario isn't shown in the diagram.  All iSCSI traffic is confined to its own VLAN, which does not communicate at all with the rest of the network.  At present, we do not use CHAP or any other iSCSI authentication/security methods, opting instead to simply segregate all iSCSI traffic.

Each server, including each of the blade servers connected to the storage infrastructure, has four Ethernet ports.  There are two onboard gigabit Ethernet adapters and another dual port gigabit Ethernet adapters.  For storage connections, we've used one onboard gigabit Ethernet port and one port on the expansion adapter and connected each to one of the two Ethernet switches.

The front end traffic for the server is currently using just a single gigabit Ethernet adapter, however we are in the process of converting these to bonded channels connected to different blades in our 5412zl, but in the same VLAN, of course.

We've installed EMC's PowerPath software on each server using the AX4.

The primary single point of failure in this scenario remains the individual servers.  We are working on clustered and VMware-based scenarios to correct this deficiency.

About

Since 1994, Scott Lowe has been providing technology solutions to a variety of organizations. After spending 10 years in multiple CIO roles, Scott is now an independent consultant, blogger, author, owner of The 1610 Group, and a Senior IT Executive w...

17 comments
kris
kris

Considering that you have not used CHAP, but are using a VLAN to secure the iSCSI targets, how are you preventing unauthorised access to the iSCSI targets by hosts on the iSCSI VLAN who are not supposed to have access? Are you merely excluding hosts that should not have access by not granting them VLAN access? Personally I would use CHAP and a dedicated VLAN to ensure security. In a virtualised scenario it might not matter so much as typically your VM hosts would have the access to the iSCSI network but not your VM guests which would normally only have public and private networks, but no access to the iSCSI network. Good article all up and looking foward to reading more. Regards, Kris Regards, Kris

eositis
eositis

Looks good. The problem I have run into on the design you describe has been with the power... have you reviewed the ups loading, with the assumption that one ups can fail? I have seen it happen, that when one ups fails to provide power output, all of the load dumps onto the other ups causing it to dump. Elmars

gordonmcke
gordonmcke

Have you tested failover of an entire controller? Why did you choose Powerpath over Microsoft MPIO driver?

jmacsurf
jmacsurf

Nice visual diagram Scott. I designed basically the same topology concept to our SAN expect in an HP enterprise capacity -- we use HP EVA 8000 series at our data centers and we put a 32GB EC trunk to link to the C Class blade servers via pass thru Cisco switches which carry the 20 (or so) VLANS we have running in there (mostly WMware). I would agree with the guy who mentioned teaming the server nics over a EC trunk to the core switch because as he said, fault tolerance is paramount but I think he meant configuring the LACP on the switch as opposed to the server -- Just don't overload the gig ports on the switch :) Recommend Switch-Assisted Load Balancing on the server's adapters.

jmarkovic32
jmarkovic32

I'm confused because I'm about to implement a Dell MD3000i iSCSi with a similar configuration. My Dell documentation says that I'm supposed to put my servers on the same VLAN as my SAN. I am going to have route the VLAN so that the clients can see the servers. At least that's what I think I have to do. How do the clients see the servers if the VLAN isn't routable?

me19562
me19562

Nice article. Something that you may should look in the future if you are planning to have a lot of servers using the AX4 is connect the AX4 to the HP ProCurve 5412zl instead of the Dell M6220 blade-based Ethernet switches. That will make traffic management easier. Also the HP should be more reliable and have more switching power than the Dell blades switches. On the servers you can use LACP (802.3ad) to configure all the four gigabit ports as one logical link and trunk it. That will give you load sharing and fault tolerance protection in every vlan that is trunk.

bandman
bandman

I recently picked up an AX4-5 as well, and I like hearing how other people are working with it. Mine is FC rather than iSCSI, but otherwise I'm as pleased with mine as you are with yours. I am also looking forward to hearing which cluster solution you go with. I had nothing but problems with RHCS. After getting it configured the way I wanted, it never did perform reliably, and my filesystems constantly hung and were out of sync. I was using DLM and the standard GFS2 which is packaged with CentOS5.2.

Scott Lowe
Scott Lowe

Yes - our UPS's can handle the full load if necessary. Excellent point! Scott

Scott Lowe
Scott Lowe

We have tested failover of a controller and it worked spectacularly. Hardly a blip. Honestly, we just sort of went with PowerPath because EMC recommended it. It worked, so we didn't really question it.

me19562
me19562

Actually the Switch-Assisted Load Balancing feature is basically the same as LACP. The only difference is that Switch-Assisted Load Balancing can be use in switches that don't support LACP. The Switch-Assisted Load Balancing is a feature seen in HP servers. In Scott case he is using Dell so is likely that he won't find that feature setting in the NIC software of the Dell server.

jthomas
jthomas

If he configures a 4 port Gigabit trunk on the switches and the servers, then configures the servers to support the iSCSI VLAN and the client VLAN, then he should have the best possible scenario-4 Gigs full duplex available to clients or servers. The HP switch won't even break a sweat in this scenario.

Scott Lowe
Scott Lowe

We have four NICs in each of our blade servers. Two NICs are dedicated to storage and have IP addresses from the storage VLAN which is not routable. The other NICs are for network traffic and are in the general server VLAN.

e_waters
e_waters

That's a good point. Dell switches have been known to cause issues with iSCSI.

Scott Lowe
Scott Lowe

Great reply! I appreciate the advice. Scott

darrengpettit
darrengpettit

I have set several of the AX4-5 iSCSI up on client sites. I have used the following clustering technologies at different site. MS server 2008 clustering (it actually works! )also I have used VMware HA clustering.

Scott Lowe
Scott Lowe

I've started to play with it in a lab... surprisingly easy to work with!

Editor's Picks