Data Centers

Pros and cons of Fibre Channel and Ethernet storage

Scott Reeves looks at some of the differences between Fibre Channel and Ethernet Storage. Both have their place, depending on the size of the business, and whether there is existing or new infrastructure.

A big growth area in recent times has been in the use of Storage Area Networks (SAN) and Network Attached Storage (NAS). Generally speaking, a SAN uses Fibre Channel as the communication protocol for provisioning of storage. A NAS usually uses TCP/IP as the communication protocol; storage provisioning can be done by using NFS or other file sharing applications. There are a couple of variants, namely Fibre Channel over IP (FCIP) and Fibre Channel over Ethernet (FCoE). FCIP and FCoE are not widely used at present; usually their use is confined to inter site links. This post looks at some of the differences between Fibre Channel and Ethernet. Before beginning, a definition: when Ethernet storage is mentioned, it means storage that uses Ethernet at layer 2, IP at layer 3 and TCP at layer 4.

Reliability

Fibre Channel is often compared favorably to Ethernet in terms of being a lossless protocol. Frames sent must be received in order and are not (as is sometimes the case with Ethernet) dropped by the switch. Network storage that uses TCP has a built in error-correction facility, as TCP guarantees in-order delivery of packets. What TCP cannot do is guarantee against re-transmissions when frames are dropped by switches. However, it should be borne in mind that TCP was designed at a time when networks were less robust than today. It was expected that networks would become congested and overload switches, causing packet drops. Ethernet networks these days are far more reliable.

Speed

Probably the key advantage that FC SANs once held was the speed. This may still be one of the aspects that separates FC SANs from Ethernet ones, particularly given that a Fibre Channel SAN basically is isolated from the Ethernet. However, one should also take a look into what the future holds. The respective standards bodies (www.FibreChannel.org and www.ieee802.org/3) give us an idea of where each is at in terms of line speed. For Ethernet, we can see that 10GE is here now, as is 40GE. 100GE is right behind. For Fibre Channel, the picture is a little slower. 16GFC is around, though 8GFC is more likely to be found. 32GFC is some way off; likely to be around 2014.

LUN masking

The idea of "LUN masking" exists in both Ethernet and Fibre Channel networks. However, there is an additional layer of access control in Fibre Channel at the switch level. This is known as "zoning". Zoning associates the machine addresses of a server with the storage device. It acts as an additional security mechanism. No such mechanism exists in Ethernet storage, although you could argue that a VLAN performs a similar role.

Cost

Cost considerations are another factor. A Fibre Channel SAN involves putting in Fibre Channel (or FCoE) switches, as well as storage. A NAS is probably a little bit less; even if you procure new switches, Ethernet switches are usually much cheaper than Fibre Channel switches. The other aspect of this is trained staff. Getting trained staff for Fibre Channel networks is, of course, possible, but if you have a network admin already, then they will know enough to integrate the NAS into the network and set up a VLAN. The allocation of LUNs is a relatively simple process on most Network Attached Storage devices.

Size and cost are deciding factors

Ethernet storage is a better fit for small to medium businesses, as it offers the robustness of Fibre Channel, but doesn't require extra switches and specialized cabling. The cost of deploying a NAS is therefore less. For the enterprise, the days of Ethernet storage are probably some way off; Fibre Channel is still firmly entrenched. But with increasing virtualization, Ethernet storage is likely to become more attractive, particularly for new roll outs.

The differences between Fibre Channel- and Ethernet-based storage are significant. Whilst Fibre Channel is a specific storage protocol, Ethernet is a LAN protocol. Ethernet has undergone many changes since its inception, and combined with TCP/IP can be utilized successfully for NAS. Both types can fulfill a business need, and will probably exist side by side for some time yet.

About

Scott Reeves has worked for Hewlett Packard on HP-UX servers and SANs, and has worked in similar areas in the past at IBM. Currently he works as an independent IT consultant, specializing in Wi-Fi networks and SANs.

6 comments
dennis
dennis

I have been in the industry since 1980, and run an industry analyst firm with our own, on-site test lab. Among other things, we run performance tests on storage systems that are direct attached (DAS), network attached (NAS) and storage area network attached (SAN). The interfaces we test include Ethernet, Fibre Channel, SAS, Infiniband and others. The protocols we test include file protocols such as NFS and SMB (formerly known as CIFS), and block protocols such as iSCSI, Fibre Channel, FCoE, SAS, SATA and others. First, we are not a product vendor, and have no particular bias towards iSCSI or Fibre Channel. We like and use all the interfaces and protocols, and understand that each has a place. We have tested numerous iSCSI and Fibre Channel storage systems, and measured the performance for each. It is easy for some people who write on this topic to make the mistake of not comparing apples-to-apples when they make comparisons between iSCSI and Fibre Channel. For example, many people like to make comparisons about the cost of iSCSI equipment vs. the cost of Fibre Channel equipment. However, most of these comparisons do not specify which types of equipment they are comparing. Most people in smaller shops who use iSCSI are using it with 1GbE equipment which is low-cost and very common. However, if one were to do the same comparison of iSCSI using 10GbE equipment, with Fibre Channel equipment the cost differences are relatively insignificant by comparison. We have numerous free resources on our website that help shed light on these topics. One of our more popular pages is our vendor-neutral Storage Interface Comparison page. You can find this page by entering storage interface comparison in your favorite search engine or by going to http://www.demartek.com/Demartek_Interface_Comparison.html on the Demartek website. Here we discuss transfer rates, history and roadmaps for the various interfaces, and discuss cabling and connectors for these interfaces. Several other small and large organizations find this page a useful page and also link to it. We have specific zones for iSCSI, Fibre Channel, FCoE and SSDs, where you will find free research reports on these topics. For those who want more in-depth information on iSCSI, we have our iSCSI Deployment Guide, also available for free download. No registration is required for any of these resources.

frylock
frylock

This article would have been better served if it had left out all the block (SAN) vs file (NAS) stuff and focused on the title - a comparison of network transports. Adding block vs file to the discussion just made it confusing.

TNT
TNT

You belittle the advantages of FC but it still rules the day, and is likely to continue to grow as more and more data is moved to the cloud. For example, Dell'Oro Group explained in its press release: "Brocade realized a record quarter with FC switch revenues of over $350 million. The company saw a 7% sequential increase in FC switch revenue, and continued ramping its 16 Gbps products in the quarter. Cisco had soft quarterly results, which Dell'Oro believe were due to stalled sales as customers anticipated the release of its new 16 Gbps switch platform." Speaking of Cisco, they have FINALLY decided to get in on the FC game since that is where all the action is. Welcome to the party. Gartner's latest IT Clock (http://www.gartner.com/technology/research/it-market-clock/) shows FC just reaching its Zenith in terms of life-cycle and that FC is actually more relevant in the age of virtualized data centers. Indeed, FC has represented more than 75% of SAN deployment according to Gartner. In the spirit of "full disclosure" I should state that I work at Brocade Communications, the company that built the FC switches that has set network speed records at Cal Tech and across Europe. While Ethernet might be cheaper, if you really care about your data infrastructure there's no substitute for Fibre Channel.

Marc Jellinek
Marc Jellinek

I have to second Joanne Lowery's comments. Dell put out a study in 2011 comparing iSCSI, FCOE and FC and found they were at near parity for performance when correctly configured and provisioned. http://en.community.dell.com/techcenter/storage/w/wiki/2722.aspx/ "The allocation of LUNs is a relatively simple process on most Network Attached Storage devices." Generally speaking, NAS doesn't provide the ability to provision LUNs. NAS, by definition, provides and exposes storage as if it were a file server, through file-sharing protocols (NFS, CIFS, etc). A LUN is mounted on a host as block-level storage, usually exposed as a virtual hard drive on a single target system (can be multiple if clustering is used). That capability is not found in NFS or CIFS, that requires iSCSI or FC; ie: a Storage Area Network (SAN). The quality of information in this article is not what it could be

Joanne Lowery
Joanne Lowery

Interesting how you disparage iSCSI and relegate it off into the NAS nethers. iSCSI works very well as a lower costs means of "SAN" (not NAS) storage using TCP/IP. iSCSI doesn't have the huge investment costs of a fibre fabric, and can be integrated into an existing structured cable environment. It may not have the 4Gb/s transfer rate of fibre, but with dual 1GB controllers and dedicated switches the cost / performance levels are very good. IOPS achieved would allow a small business to run multiple VM servers while using cheaper storage. iSCSI SAN is not NAS, the transfer protocols are totally different even though both can use TCP/IP as network transports.

Joanne Lowery
Joanne Lowery

I suspect the author has confused iSCSI with NAS, when iSCSI is a true SAN implementation using ethernet and not fibre. His complaints about loosing data through switches is confusing. To believe that iSCSI has lower data integrity VS fibre because of switch issues is stunning news to me when I have clients reliant on this technology.

Editor's Picks