Hi everyone, first post here, and I am a bit overwhelmed when it comes to finding the ideal backup approach for our business. It's a long post, but I will try to make it as concise as possible, so please bear with me.

This is our current backup setup:

-Backup environment:
5 custom built backup Servers running Windows 2008 or 2008 R2
Backup Software: Symantec Backup Exec 2012
Backup Hardware: 4-bay Promise SATA caddy
Servers that need to be backed up: include Linux (CentOS), Windows 2000, 2003, and 2008. we also have some servers running VMWare ESXi but they're currently being backed up using the GhettoVCB script

-Current backup strategy:
We have about 8 TB of data spread across the 5 backup servers. The backup servers backup other servers over the network using Gigabit network. We used hard drives for backing up, and we do a full backup over the weekend and incremental over the weekday. Every Friday, we will swap out the disks for new ones, and the old ones get shipped to off-site storage.

-Why we're looking for a new approach:
The hardware is old, and running Backup Exec 2010 and Windows 2008 or 2008 R2 make the servers quite slow, so we're due for a hardware refresh
Also, data size we needed to backup has grown from around 2-3 TB 3 years ago to around 8-10 TB every week, so we want to keep an open eye for more scalable solution because as it stands the data may increase even more
Backups have not been very consistent, and we're not sure if it's due to hardware or backup exec or both.
Every Friday swapping out the hard drive can be a real pain and annoyance, so we're opting for a trayless solution

-What we're looking for:
We're open to changing from Backup Exec 2010. We tried to upgrade Backup Exec 2012, but it was so horrible that we promptly uninstalled it and reverted back to Backup Exec 2010, but the replacement needs to be known to be reliable though
We want the server hardware to be reliable as well, we get all our servers from Dell, and instead of getting custom-built server, we're thinking using Dell servers will be more reliable.
But since Dell servers don't have enough free drive bays, using internal drive caddies with 2 or more bays on the servers seem out of the questions.

-Options we're looking at:
I am now testing this setup: using a Dell PowerEdge T110 II server, which has a built-in eSATA port, but it does not seem hot-swappable, I've tested it with http://ca.startech.com/HDD/Docking/4-Bay-eSATA-USB-3-to-SATA-Hard-Drive-Docking-Station-for-25-35-HDD~SATDOCK4U3E and http://istarusa.com/raidage/products.php?series=Custom%20Storage%20Tower&sub=eSATA-Port%20Multiplier&model=DAGE540DE-PM . Insertion and removal of hard drives don't get recognized until reboot.
I've also tried using these two eSATA cards: http://ca.startech.com/Cards-Adapters/HDD-Controllers/SATA-Cards/2-Port-SATA-6-Gbps-PCI-Express-eSATA-Controller-Card~PEXESAT32 and http://ca.startech.com/Cards-Adapters/HDD-Controllers/SATA-Cards/2-Port-PCI-Express-eSATA-Controller-Card~PEXESATA2 , they allow hot-swapping, but they cause the backup to go very very very slow (shows on backup exec as 4-5 MB per minute to around 70 MB per minute, versus 800MB to over 2000 MB per minute for the old backup servers). Some users suggested that Silicon Image chipset don't work well on Dell servers.
I then proceeded to try Rosewill RC-225, it's hot swappable, goes very fast, but would always give me error about SCSI controller error.
Another possibility: Custom build a backup server, but will definitely need suggestions on reliable components. This way we might be able to use the eSATA controller cards without compatibility issues. We will also be open to use another internal SATA multi-bay caddy

-Mandatory requirements for our backup:
-Cloud storage and anything over the internet storage are not feasible due to sheer amount of data and security concerns
-Relatively reliable and graceful, we're not looking to get a convoluted setup, but a setup robust enough that it may fail from time to time, but it won't continue to fail over time
-Easy hard drive replacements and off-site storage for old backup is required.

-Questions we have:
-Is a true server, for example, using Xeon processor and/or ECC RAM needed for a reliable backup server setup? Or an i3/i5 be ok?
-Any way to make the built-in eSATA port on the PowerEdge T110 II hot-swappable?
-Which eSATA controller cards are reliable? Or more importantly, would you consider eSATA to be reliable?
-Is there any fundamental flaws to our backup strategy and what we're trying? Or anything we overlooked?
-How do you guys build a reliable backup environment with minimal hassle?
-What kind of setup would you recommend instead?

Thanks for taking the time to read this, and I'm open to suggestions. Thanks!