Discussion on:

26
Comments

Join the conversation!

Follow via:
RSS
Email Alert
?AoE is cheap, but the more supported iSCSI isn?t that much more expensive when you consider the outstanding resources behind it.?
Let?s see, a TOE based iSCSI card is about $500 and that?s per node you want to support. A good gigabit card is about $50. What happens when you go to 10 gbit Ethernet? A 10 gbps card costs $800 while there is no such thing as a TOE based 10 gbps yet. When there is such a thing, the costs are scary.
?AoE isn?t "enterprise-ready" yet.?
Says who? Fiber Channel proponents use the iSCSI isn?t ready argument.
?AoE is not as scalable as iSCSI or Fibre Channel when you consider location?i.e., with Fibre Channel and iSCSI, you can scale your storage throughout the organization. This is primarily due to the inability to route AoE traffic.?
First of all, very few people need to route storage and routing is considered a security liability. Second, since when is regular Fiber Channel routable? If you have to have routing, you can always resort to iSCSI though very few people use it. Long distance replication works very well at the application layer anyways. Why bother with routability when it?s never used 99% of the time? If you need it over multiple buildings, you can always span VLANs.
?ATA disks are not as reliable as their SCSI counterparts. However, many iSCSI implementations use ATA disks, too.?
That?s a bit of a contradiction since you admit iSCSI uses ATA as well. The fact of the matter is, ATA and iSCSI can use flash, RAM, PATA, SATA, SAS, SCSI, or anything else you can think of as a storage medium. Since ATAoE is simpler, one can argue that it?s less likely to break and therefore more reliable.
0 Votes
+ -
Contributr
All true.
Scott Lowe 18th Aug 2006
You make good points. However:

* "Let?s see, a TOE based iSCSI card is about $500 and that?s per node you want to support. A good gigabit card is about $50. What happens when you go to 10 gbit Ethernet? A 10 gbps card costs $800 while there is no such thing as a TOE based 10 gbps yet. When there is such a thing, the costs are scary."

Many don't use ToE and are using just Gigabit NICs for their iSCSI implementations. Bear in mind that AoE will use this same NIC and possibly that $800 10Gb NIC as well. So, from this persepctive, I don't necessarily see the cost benefit of AoE.

"Says who? Fiber Channel proponents use the iSCSI isn?t ready argument."

True, but there is much broader support for iSCSI than AoE at this point. That said, AoE support does seem to be growing. Personally, I think the FC are wrong when they say that iSCSI isn't ready yet.

"First of all, very few people need to route storage and routing is considered a security liability. Second, since when is regular Fiber Channel routable? If you have to have routing, you can always resort to iSCSI though very few people use it. Long distance replication works very well at the application layer anyways. Why bother with routability when it?s never used 99% of the time? If you need it over multiple buildings, you can always span VLANs."

You make a very good point here that I had not considered. And, you're right about routing. My iSCSI implementation is on a completely separate network with no connection to the outside world, and it's going to stay that way. However, with iSCSI, FC (FCIP/iFCP), there is obviously some need for storage routing for some organizations. In this regard, AoE would not fit the bill for these companies, but this would come out in the technology research portion of the project.

"That?s a bit of a contradiction since you admit iSCSI uses ATA as well."

It was intended to be a contradiction happy What I intended to say is: "ATA disks aren't as reliable, but heck, iSCSI is using them, so maybe this isn't a big deal."

Personally, after doing a lot of research on AoE, I think it'll do well, at least on the low end for now. When iSCSI first came on the scene, there was a lot of grumbling that it wasn't a "real" storage technology and ran at "only" 1Gbps. AoE will likely go through a similar process and will hopefully grow as more vendors add support for the technology.

Thanks, George, for the comments! A friendly debate is always fun!

Scott
"Many don't use ToE and are using just Gigabit NICs for their iSCSI implementations. Bear in mind that AoE will use this same NIC and possibly that $800 10Gb NIC as well. So, from this persepctive, I don't necessarily see the cost benefit of AoE."

There is a HUGE difference. If you don't use a TOE, you will crank the CPU with all that TCP overhead in iSCSI. AoE overhead is extremely low, we're talking in the range of 10 times less overhead. You can try to do 10 gbps Ethernet with AoE with no TOE, forget about that with iSCSI.
0 Votes
+ -
Contributr
I'm running an iSCSI SAN with nothing but Gb NICs in my servers. We've been in this configuration for well over a year. One of the services we run is Exchange, with two thousand mailboxes on a single server, all stored on the iSCSI SAN. This server, on average, runs way less than 10% utilization. We also have our primary student database running from the SAN - also, very low utilization.

A friend of mine works for an architectural firm with huge files stored on the iSCSI SAN and is also using just Gb NICs - very satisfactorily.
Exchange servers don't push a whole lot of traffic. Most of the time it's under 100 mbps if you actually measure it with PERFMON. Couple that with some very fast CPUs and the performance is very good. But that doesn't change the fact that iSCSI has many times more overhead than AoE. You can say that the overhead of iSCSI is acceptable on a really fast server, but I prefer putting server CPU cycles to other use.

If routing of storage is not needed the vast majority of time, why bother with the security issues and the overhead?
0 Votes
+ -
I disagree
pgm554 Updated - 18th Aug 2006
I asked the Novell Brainshare cluster engineer about TCP offload cards and iSCSI. at this weeks Linux World in SF.


He referred to a test bed article from SAN Week or some other publication that did a benchmark using TOE cards and just a regular stack configuration on a modern file server with a good CPU (Xeon ,Opteron) and there was little, if any, difference in CPU utilization.

It did make some difference in older boxes (say 3 to 5 years old or more).

TOE seems to be more hype and marketing than substance.
0 Votes
+ -
But that doesn't change the fact that iSCSI has many times more overhead than AoE. You can say that the overhead of iSCSI is acceptable on a really fast server, but I prefer putting server CPU cycles to other use.

If routing of storage is not needed the vast majority of time, why bother with the security issues and the overhead?
0 Votes
+ -
I work at a small company called Small Tree Communications. We are a driver engineering company and we write network drivers for Mac space. We offer networking solutions that you cannot buy anywhere for Macs such as our new 10Gb Intel Cards.

The reason I mention is that we have our own iSCSI Initiator and an AoE Initiator for Mac.

To address 10Gb and AoE specifically, Small Tree's 10Gb cards are not TCP offload engines.

Intel has actually hid away from those because of the tremendous costs (and resultant pricing) that goes along with it. These days with faster memory and cpus, it doesn't really pay off either.

The Intel cards we use have a set of features called I/O AT that provides a number of efficiencies to improve performance including receive side scaling. We have the ability to pull data from the card using multiple CPUS.

I thought this would be more relevant when I explained that we've passed data at near line rate speeds on 10GbE to an AoE device for Mac.

Granted, I'm not familiar to much with the Windows and Linux AoE options. It might be a different story there.
0 Votes
+ -
pricewatch.com
jwd81 30th Sep 2010
where are you buying toe cards, from CDNOW?
Check out pricewatch.. $160-180..
0 Votes
+ -
Thanks for the information.
stress junkie Updated - 18th Aug 2006
This article is the first that I've heard of ATA over Ethernet. It's good to know. My business is based exclusively on very small business clients. I absolutely need to know the new and affordable architectures available and their pros and cons.

Now, how does all of this compare to a normal file server running a normal operating system over a normal TCP/IP network?

As far as ATA disks are concerned, when you compare the overall price of disks, and the price/byte, ATA disks are so much more affordable and high performance that it seems to me that a good RAID 5 box with SATA II disks is highly preferable to SCSI disks. If you look at disks from on line retailers you still see prices such as Seagate Cheetah U320 SCSI 10,000 RPM 300 GB disk at $700 (tigerdirect.com) vs. Seagate Baracuda SATA II 7,200 RPM 320 GB disk at $110 (tigerdirect.com). You can purchase six SATA II disks for the price of one SCSI but the specs aren't all that different. SCSI U320 vs SATA 300, 10,000 RPM vs. 7,200 RPM. This is just an example where I tried to find similar specs between the SCSI and SATA II hard disks to compare the prices as fairly as is possible.

Okay. So. It seems to me that the SATA II wins if you use a RAID 5 disk configuration. Any insights about my price/performance comparison?

The only thing that I have rarely, if ever, seen is true hot swap capability with SATA. I don't recall if I saw Adaptek RAID cards with true hot swap or not. I haven't looked at RAID hardware cards for about a year so my memory about the exact feature sets is failing me right now. Since most hard disks, SCSI or otherwise, are mounted inside a chassis that you have to open to access the disks, it seems that hot swapability isn't usually available in SCSI machines anyway. You have to power off the machine before you remove the covers to access the disks. The only true hot swapable disk arrays that I've seen were Storage Works arrays from Digital Equipment Corp/Compaq/HP/whatever.

Everyone, please let me know what you think. Thanks in advance. happy
"Now, how does all of this compare to a normal file server running a normal operating system over a normal TCP/IP network?"

First of all, AoE doesn't use TCP/IP and it's nonroutable. The beauty of that is that it's more secure and MUCH faster because of far less overhead.

Second, AoE is only for block level storage when you have to mount a native drive for your particular server application. You don't need to use it for File Serving applications.

Believe it or not, RAID using software today is actually better than a hardware RAID controller.
0 Votes
+ -
What??????
pgm554 Updated - 18th Aug 2006
And this is coming from somebody that espouses
TOE?

Software RAID SUCKS when compared to REAL RAID controller.

Lose a disk or two and see just how much a REAL HW RAID controller matters.

And by real.I mean something with a co processor (like an Intel i960) on board.

You must be thinking some cheapo Silcon Image or the lower level Promise/Adaptec cards.

Anything under list of $300 these days, is just a chip that does software (SW XOR) RAID,and that, is not true HW RAID.

Software RAID better than true HW RAID,you have got to be kidding!
0 Votes
+ -
Have you done the measurements?
georgeou Updated - 19th Aug 2006
First of all, I didn't say I recommend TOE. I said I recommend the use of AoE which doesn't require TOE or CPU overhead.

Second, I would have never said software RAID was better a few years ago, I swore by hardware RAID. But as I really started measuring and testing the performance of storage, I realized that hardware RAID controllers in the $300 range had write speeds in the 20 megabyte/sec range when a single drive is between 40 and 80 megabyte/second range. I've seen recent measurements of a $600 RAID controller that had write speeds in the 50 megabyte/second range. While that's certainly better, it still leaves much to be desired.

Now I was shocked by something recent I saw. I saw a hardware storage appliance using AoE without a hardware RAID controller crank out 90 megabyte/second write speeds using a single Pentium 4 3 GHz processor. Sure the CPU was cranked up very high, but what else would you have that CPU do in a storage appliance? I use to work for a company that produced high-end storage devices and I had long conversations with the engineers that worked on the storage products. He also told me the same thing that it?s cheaper to just use the CPU for software RAID.

Now as for this business of ?true? RAID, are you somehow suggesting that software RAID is somehow ?fake?? What you really mean is dedicated processor RAID or general purpose processor RAID. The only difference is cost and something like a $400 dual-core XEON 5100 core 2 based CPU has massive general purpose performance with massive economy of scale. That second core will sit there doing nothing most of the time anyways and making it boost RAID 5 performance well above any $600 RAID controller is a very nice side effect of dual-core processors.
0 Votes
+ -
Can you boot from a software RAID? From what I remember you can't. This would just add a little complexity and a little additional cost if you wanted the extra performance that you say software RAID provides.

Extra cost...I wouldn't want the OS on a critical server running with out RAID so I would run the OS off a hardware RAID controller with a software RAID running off a different controller for performance reasons.

Bill
0 Votes
+ -
Non RAID boot partition
stress junkie Updated - 21st Aug 2006
I haven't figured out how to do this in Windows but in other OSes you can have a small boot partition with the kernel and the RAID software on it. That allows you to have most of your system software on a software RAID partion. The thing that I hate about this solution is that you have to give up one of your precious 4 primary partitions to a tiny boot partition. That's becoming less of an issue as disks get so big that you almost have to use an extended partition with logical "drives" in order to keep your blocks per cluster small. All the same I hate making a primary partition tiny, but that's how you make a system disk with software RAID.
0 Votes
+ -
Good to know
wdewey@... 21st Aug 2006
Will the system be able to either fully boot or rebuild a failed drive on an initial load (assuming it didn't loose the drive with the boot parition)? I could see loosing a whole array because of a failed drive cause the software to crash on load. If it could rebuild the drive on load then a backup drive with the proper partition scheme and a backup of the initial load software should get you up and running even from losing the boot drive.

Bill
0 Votes
+ -
Yes
stress junkie Updated - 21st Aug 2006
OK. So. If you have RAID 1 (mirroring) then you also mirror the boot partition. You can do this by hand. You also have to mirror the boot sector of the boot disk.

If you have RAID5 (redundant striping) then you copy the boot partition and the boot sector of the boot disk to the other members of the RAID5 array.

This is all pretty easy in non-Windows operating systems. I haven't tried it with Windows systems but it could work if you use a non-Windows boot loader such as grub and you keep your NTLDR, and other Windows boot files on these boot partitions. The more I think of it the more it seems that it should be pretty easy to do with Windows using the grub boot loader.

On the other hand Windows may already have this capability with its own boot loader. I'm not sure but it seems that it should. happy
0 Votes
+ -
I'm just a power user'ish, reared on Macs and
things generally plug compatible and
supplying global enetrpise network
infrastructure supporting continental scale
server centre backup.

When I had to use PCs and my first hard drive
started to fail I naturally thought I'd replace it
just like I do for a Mac SCSI or a customers
network attached storeage. You can't.

Why? Because manufacturers in the PC world
interpret the ATA/IDE interface differently so
plug compatibility is by manufacturer and
model generation. If you want to be very afraid
read it up.

Better stick to iSCSI which is a real standard
and no doubt masks the ATA variations (a
hard disk symphony??). If there is a niche its
probably manufacturer dependent so you
would potentially have to change the interface
for each architecture.

I'd much rather have a universal interface
which means I can back up to anything
anywhere on TCP/IP via ethernet on down. For
local back up what's wrong with USB 2 and an
external HD anyway, fast and universal - I
expect a repeater will take it beyond 10 metres
also?

Am I wrong about ATA variations in the latest
market?

For the above reasons I don't see the need for
yet another access connection for storage
backup.

Brian Catt 01932 772731
... I thought that the hardware compatibility issues between Macintoshes and Intel machines was based on the little endian/big endian differences between their CPUs. That changed the layout of the lines in the buses and card adapters. I'd be interested to find out if the new Macintoshes based on Intel chips have more hardware component compatibility with PCs than the Macintoshes based on Motorola and Power PC CPUs did.
0 Votes
+ -
I wasn't thinking about interoperability or
compatibility issues betwen Macs and PCs,.
Just the SCSI interface they used to use was a
standard, as is ethernet and TCP/IP. The ATA
standard is not, its a typical PC quasi
standard per manufacturer because of the
hobbyist box of bits nearly working together/
great job creation scheme for techies/ PC
"architecture" - OK it makes them cheap. This
is not a Mac or PC supremacy thing, but I bet
Apple under Jobs have got the Intel integration
so tightly locked down it doesn't any support
sites offering drivers for every imaginable
device you can attach, bcause they make a
whole comuter and are draconian about
developer interfaces. Can you imagine a Mac
only ever user's response to a registry or
device manger - "why would I need that"
springs to mind? So for all the above reasons
I don't think PC ATA is a good idea adding any
value, too many versions. Perhaps make PCs
standardise on the Mac version of ATA on IDE,
if they are actually using some version of that
. But I digress.......AoE still pointless I think.
0 Votes
+ -
AoE is a direct Fiber Channel replacement when you need block level storage. AoE is not routable, which is very good for security and routing of block level storage across multiple sites is rarely needed. You can use iSCSI for that rare exception.

AoE uses far less CPU resources than iSCSI and doesn't even require a TCP offloading engine to perform well.
0 Votes
+ -
So all AT over Ethernet is, is a peer to peer type of storage method(NetBIOS vs. something like IPX or IP)and if that is anything like NetBIOS peer to peer (which it sounds like),I would be highly suspicious of its ability to scale significantly.

Doesn't sound like anything that even remotely belongs in a network that can scale.
0 Votes
+ -
Fiber channel does not route. AoE does not route. Fiber channel costs 10 times more than AoE. It's as simple as that.
If you have that problem and have to have what
will probaly be a spinal tap solution specific to
the ATA variation you use to take local mass
data dumps (off running servers most likely?)
- great. But sounds like the niche of a niche.

How many people really need such a point
solution for a bit more speed? Can't you just
take a back up using regular RAID device off
line from the cluster in these 24x7 hot backup
environments (the only one I can see that is
really important enough). Use the
virtualisation manager to move the hot data
elswhere then back up the shadow and just
wait a bit longer?

Not enough ATA on ethernet apps to fill a
football stadium I bet.

I doubt there are any real security issues in
terms of transferring data over what would be
a Gigabit ethernet WLAN or WAN, local or
national in scale (as in Nortel's NAS clusters).

I think if you are serious about servers and
their backup you probably go buy a real and
existing solution from EMC, IBM or Hitachi, not
patch one together with PC after market bits.

Let the boxes do it.

Once upon a time there was a debate about
faster raster image processors for printing far
more CPU efficient algorithms than Adobe.
Adobe won by marketing and
standardisation/commoditisation as CPU
power increased. Ditto with Microsoft
overweight code. In networking we talked
about Net BUI on serial WANs (SDH) to speed
up global coms, but its Frame or IP that gets
used (Frame is much more efficient than IP
BTW and has class of service). Increasingly
stuff is on IP which is inherently wasteful
(actually much IP goes over Frame
infrastructure still but bandwidth and its cost
have diverged at light speed).
In all cases horse power is used rather than
sophistication to crush the problem with
generic solutions, in the great American way.

So no, i don't think there is a commercial
reason for AoE at all on further reflection. If it
was my product group to manage I'd can it in
favour of improving broader mainstream
solutions.
Fiber channel does not route.
AoE does not route.
Fiber channel costs 10 times more than AoE.
AoE follows KISS engineering principles and the official standard is about 10 times shorter than iSCSI.
AoE is just good engineering.

It's as simple as that. AoE will have a slow adoption curve like iSCSI because the IT world typically likes to follow and not lead. A lot of people in IT don't want to do something unless everyone else is doing it. But Linux storage appliances are jumping on AoE big time and the Linux kernel even has AoE target and initiator built in. Microsoft on the other hand has closed their eyes on AoE for now until Linux gives then a big shock when they start eating up storage appliance market share.
0 Votes
+ -
Frame Relay
wdewey@... 21st Aug 2006
A friend of mine works for an ISP and the only thing they do Frame Relay is local access at T-1 speeds. For everything else they use ATM or SONET. Why? Because Frame Relay has more overhead which makes it more expensive to run at higher speeds and ATM is cheaper (also has built in QoS). SONET was the only gigabit or higher protocol for a long time. Now, in many places this is being replaced by IP networks because it runs at gigabit speeds and is cheaper. Why run a T3 link over fiber when you can get a gig service for hardware that is 1/3 the price? You can buy a 10 Gig card for $800 or a SONET card at ?????

Same situation here. Why by iSCSI or fiberchannel if AoE is all you need.

Bill
Keyboard Shortcuts:
Prev
Next
Toggle
Join the conversation
Formatting +
BB Codes - Note: HTML is not supported in forums
  • [b] Bold [/b]
  • [i] Italic [/i]
  • [u] Underline [/u]
  • [s] Strikethrough [/s]
  • [q] "Quote" [/q]
  • [ol][*] 1. Ordered List [/ol]
  • [ul][*] · Unordered List [/ul]
  • [pre] Preformat [/pre]
  • [quote] "Blockquote" [/quote]

Join the TechRepublic Community and join the conversation! Signing-up is free and quick, Do it now, we want to hear your opinion.