Storage

Building a robust test-lab at home with virtualization

Blogger Brad Bird is building a home lab for testing, development, and training -- it's power extended with virtualization. Do you have a test lab? How would you set it up?

Virtualization is not a new topic but how many of us use it at home? I hang around with a bunch of geeks, so all of my colleagues have been virtualizing at home since it has been possible.

Early last year, I spent a couple of thousand dollars and set out to build a lab I could use for testing/development/training. Basically, the lab is for anything and everything that I cannot do on a company's production servers, including developing presentation scenarios that companies like so that they will hire me to reproduce them on their production servers!

I thought I would share what set up I am using, in case you're thinking about this too. Or, if you have one of your own, I would appreciate your thoughts on the home lab.

So this is what I am working with:

  • 1 router/4-port gigabit switch (working as gateway/firewall)
  • 1 16-port gigabit switch
  • 1 4GB AMD 64-bit (dual core) small form factor PC with roles:
    • Server 2008 Domain Controller
    • Virtual Machine Manager 2008 R2
    • SQL Server 2008 x64 physical host
    • Virtual Server 2005 R2 SP1 host
  • 1 16GB Intel 64-bit (quad core) small form factor PC as a Hyper-V host with VMs:
    • Server 2008 R2 Domain Controller
    • System Center DPM 2007 Server
    • MOSS 2007 Server
    • System center Operations Manager 2007 R2 Server
  • 1 4GB Dell Latitude D820 Core Duo laptop
  • 1 4GB Dell latitude D830 Core2 Duo laptop (just found out this goes to 8GB, sweet!)

Plans:

  • The 16GB system is doing fine since I have been conservative when allocating resources to my VMs; I can maybe get an SCCM 2007 R2 VM on there as well to showcase operating system deployment, patch distribution, and application virtualization.
  • I went to the D830 possibly controlled from an external eSATA drive as a DPM 2010 server with some mass storage to showcase the product and to back up my entire environment.
  • Mass storage is needed. I was going to purchase either a Drobo, or maybe a Dell MD3000i, Thecus, or the like (chime in here with experiences). I don't have to purchase iSCSI mass storage since I can use Starwind to create iSCSI targets to connect my systems to.
  • The 16-port switch was purchased to retire an older 10/100 8-port switch because some network cards in my systems and in my OfficeJet printer won't communicate below 1000MB and my router only has 4 ports. I may need another switch to handle iSCSI dedicated traffic.
  • I intend to leave my lab accessible all the time so I am going to purchase a Fortigate or comparable device to handle port filtering and offset processor load from my router/gateway.
  • I may need another system to install VMWare ESX (VSphere) on since I am showing customers how to manage both MS and VMWare virtual environments.

So far, $4,000 has been spent on my lab over the last 30 months. More is needed but as you can see, because of virtualization, these dollars can go pretty far.

Virtualization is great but it does not eliminate all costs, as you can see.

IT pros, have a look at what I am doing and if I can help anyone with more details, I am happy to do so. I am also looking for experiences or feedback to help improve my lab.

About

Brad Bird is a lead technical consultant and MCT certified trainer based in Ottawa, ON. He works with large organizations, helping them architect, implement, configure, and customize System Center technologies, integrating them into their business pr...

48 comments
ZorianRotenberg
ZorianRotenberg

Brad, I just wanted to say thank you for using StarWind and for mentioning us. Additionally, if the readers are not familiar, you can set up a true active-active High Availability storage cluster with StarWind (either for VMware or Hyper-V) and have automatic failover on the storage side in case of failure. Certainly this type of setup is used in production by companies and not for a home iSCSI storage. Thank you again, Zorian Rotenberg CEO, StarWind Software www.starwindsoftware.com

techrepublic
techrepublic

I have a low-dollar home lab, but it's quite large, comprising (at this writing) 21 computers, including 3 notebooks (one a MAC). My cost for hardware: ZERO. It's made up of computers and parts left to me by clients whose computers I've repaired, upgraded, or replaced. Of course, that means it's not state-of-the-art or leading edge technology (and no 64-bit), but out of that conglomeration I've been able to put together 3 Windows servers (2003), and could run Server 2008 if the need arose. The machines range all the way from a 100 MHz 486 (suitable for an older Novell server) to a 2 GHz AMD Athlon. Not all the computers are actually plugged in and bootable at any given momenet, for a variety of reasons, ranging from space and power requirements to software. Ah, software - that has the potential to be a real big expense. The Technet subscription seems to be a good deal. What I have done, however, is taken advantage of my school's participation in the MSDNAA (MSDN Academic Alliance) program. I can legally download (or burn to CD/DVD at school) a variety of software ranging from MS-DOS 6.22 all the way up to Server 2008, Visual Studio 10, and a lot more, as well as receive valid product keys. Here's the money part, this is what cost me, but it has been well worth it: I also have a full-size equipment rack with Cisco routers and switches, a PIX firewall, and electrical outlet boxes I built myself for power distribution. I think I've spent about $2000 on all that. It includes 2620 series routers and 2950 series switches, and a Cisco 7000 router. That really came in handy when I was studying for my CCNA under an instructor who doesn't believe in hands-on training (go figure) and switched to nights, leaving me in the lurch. This lab has been a resource of inestimable value, enabling me to experiment and learn in an actual hands-on environment. I thoroughly recommend setting up a home lab to anyone who is (or aspires to be) in a position of installing, configuring, and/or maintaining systems in an environment where it wouldn't be possible to experiment.

acanario
acanario

I recently purchased a laptop to serve as a laptop for my house since i cannot work with the equipments at work. Laptop specs: HP dv6 p- core 2 duo 2.36 GHz processor. 500 GB HDD, 1.0 GB video card. 8.0 GB of physical memories. Running windows 7 X64. VMWare 7.0: virtual machine 2 windows 2k8 DC 2 windows 2K8 application 1 of this servers is a WDS server as well as a DHCP 1 of the DC is running DNS services 2 Ms windows 7 X86 client. 1 Ms windows xp x68 client 2 Ms Windows 2k3 print/file server. i can have all of them running at the sametime and my system still have some resource to run the host O/S. did not to spend 4000 USD.

mike
mike

I have 2 AMD 2+ Dual Core with 8G ram, promise technologies TX4 SATA controllers(vsphere compatible) running vsphere 4. I use Openfiler for a NAS with 1TB disks. I purchased 2 cisco routers and a gb switch on ebay. All cost less than $1500 May not be extremely fast but works great for testing and proof of concept.

cwhitmore
cwhitmore

I noticed you mentioned Starwind as the iSCSI SAN. If you plan to showcase DR I would suggest looking at Datacore's SanMelody. It has all of the DR options (sync and async mirroring), as well as thin provisioning.

paulenet
paulenet

Brad, I think you have a pretty nice setup, and that the only real key in my opinion is that your environment allows you to combine the most capability, flexibility, and ease of use for your R&D needs that you can afford. Having said that, one tip I would suggest is that if you plan on running SCVMM anywhere, allocate a seperate, physical box for it with a minimum of 4 to 8 gigs RAM, and run SCVMM from there, to connect to and manage your Hyper-V host. The purpose is to free up as many resources on the host as possible, to simply concentrate on running VM's. On your Win2K8 host, I would only run Hyper-V, and keep the rest of the bare metal install clean. Here is my setup: Cisco Gig-E firewall/router with wireless N, and VPN (2) Intel SC5400 5U (pedestal) chassis, running 3GHZ dual-quad core processors, 32 Gigs RAM, running Win2K8 w/ Hyper-V (on bare metal). Both are running Intel RS2BL040 RAID controllers in SAS mode, with one 2TB RAID-0 array, and one 5.5TB RAID-6 array. I run a seperate Intel 3Ghz quad-core server with 8 Gigs of RAM for my own dev machine, and for running SCVMM to connect to the Hyper-V hosts. On each host, I run all my VM's on the RAID-0 array, for optimal performance, and save data, install bits, etc., on the RAID-6 array. Each of the VM's are backed up nightly onto the RAID-6 array, so if there were a failure on the RAID-0 array, I simply power up the VM's from the RAID-6 location to mitigate server downtime, while building out a clean RAID-0 array again, then moving the VM's back to the RAID-0. BTW: A while back, I experienced a drive failure on my RAID-0 array, but because I also had the RAID-6 array with enough available space, the RAID-0 array was actually rebuilt without losing any data, and even without any of the VM's going down! (I would shutter at using RAID-0 under any other scenario.) The two hosts allow me to also build out load-balanced VM's when and where necessary, or just to do performance testing and comparisons between single VM's and load balanced VM's. This setup allows me the most scalability and flexibility, while maintaining a small physical footprint on office shelving instead of taking over a room with server racks and cables running everywhere (like I used to do).

sfarhanj
sfarhanj

nice environment set, but isn't the amount is too much ???

moabrunner
moabrunner

Why go with Windows Server 2008?? I would have and do go with CentoOS 5.4 using Xen. You can setup multiple virtual sessions of Windows Server.

eternal_life
eternal_life

I have and use with great success another buildance, based on disconnect from all net.connections inc. all telephones set in off mode. It works at the best for me, as it impossible to share if for online work I can not view the buildance.

Jimbo Jones
Jimbo Jones

Check out a free and very powerful router emulator called Dynamips and its graphical front-end Dynagen(www.dynagen.org). It is not a simulator but a true emulator. By creating a suitable control file, and assuming you have legal copies of any required IOS images, you can boot multiple routers in a Windows machine or VM, connect them together virtually, and even attach their virtual interfaces to physical nics for a zero-cost router. Not as fast as the real thing but they can do BGP, EIGRP, etc. Comes with a pix emulator too. Great for CCIE labs.

ilya.shick
ilya.shick

in case of ESX 3.5/4.0 just check BIOS/INTEL to see V option

TechRepub
TechRepub

couple of things, Dell T105's are always on sale on tech bargin rss feeds for around $230. Currently using 2 at home for my VMware ESXi hosts. With these boxes you can use the existing SATA systems instead of getting PERC cards, another good system to use are used Optiplex 740's Switch must be vlan capable IMO Openfiler for the iSCSI or currently at my house is an Infrant/Netgear NAS Not knowing your house, I would run some extra power circuits to your server room, and look at the AC requirements you are going to build up. Don't forget to add UPS to this kind of setup due to the investment costs. My current setup for home is as follows Cisco ASA5505, $515 Home and Dev network connected 2 16 port switches, 1 for each network, $450 1 T105 Servers Upped the RAM to 8 GB, added a 2nd nic 1x160 GB(original), 1x1 TB hard drives in each, $600 each with upgrades 1 Dell T110, Quad Core, 8 GB RAM, 2 nics, 1X320 GB HD, 2x1 TB HD's. Windows Server 2008, VMWare Workstation, I will create ESXi VM's (ESXi Guest, inside workstation) on this box for testing and then transfer them to other hosts as needed Netgear NAS with 3 TB, used for home, with a firewall rule to the test network, $800 when bought in 2007 I utilize my work MSDN account for the OS to test on. I can also run the critical work VM's on the T105 if needed, slow, but workable if the 3 other machines in order of backup machines are otherwise utilized. HTH

Jeff7181
Jeff7181

Throw some Linux clients and servers in there. Add a WiFi access point. Get a VPN going.

bdskp
bdskp

the 4,000.00 spent on your lab include software purchases (window server 2008, SQL 2008, SCOM, etc)? I assume not but thought I'd ask. That's part of the roadblock to building my own testing lab (software). So..much..money..

Walthy
Walthy

I see Starwind has some free products and 30-day trial downloads. Would it be possible to get a deal on something like the TechNetPlus from Microsoft for prototype/testing environments. 30-days is not long enough to really test a fully integrated working environment. How about full version software in a prototyping environment like we are talking about here. I would love to be able to convert an older server into an iSCSI SAN and to test out environments like you have with your software. If I don't know about it and how to use it, it is very hard to recommend something to our bosses or customers.

brad
brad

Nice setup! I had a boneyard of well stocked parts but needed to do away with much of it since the wife factor took over... ;) Now I have some parts but am finding the need to stay as close to bleeding edge as I can afford. Incidentally, I did all of my MCSE certifications on lab machines built from older equipment.

Walthy
Walthy

This is a great reason for continuing education. Take a class at your local college/community college and take advantage of academic discounts and other programs. For years I was always in school. Our local school used the motto "lifelong learner," and I think that really applies to computer geeks. I'm semi-retired now (working enough to keep health insurance premiums paid, up 17% this year alone) but am trying to keep up with everything. Am even looking to get some certifications. Didn't used to need them so much, but do now. In retirement I plan to do a lot of non-profit work, but it still costs real money to keep up. I also use older generation equipment in my lab, but still new enough to support virtualization and x64. Buying enough memory is expensive though, need to stay with 2GB modules. May even get around to trying linux.

brad
brad

Hi acanario, good call on the laptop. The reason I split the systems up was to separate roles and load. Also, Laptop runs with 1 SATA drive on 1 spindle unless you use external drives or swap out DVDROM if you can... Disk will undoubtedly be a bottleneck for you.

bulk
bulk

Take a look at Windows Storage Server 2008 from your Technet downloads. Normally an OEM only product, this is available as 32 or 64 bit and has ISCSI target included, together with single-instance store. I have this running on a cheap Asus M3N WS AMD mobo with onboard video and two onboard gig NICs, together with 2 x 2TB SATA disks. Found an ATX min-rackmount case and quiet fans. Works really nice. Richard

brad
brad

Hi Mike, I would be interested to know the model of routers and GB switch. Why 2 routers btw?

brad
brad

I apreciate hearing about this cwhitmore, I will check it out ;). Thanks!

brad
brad

Hi paulenet, I am running SCVMM on my 4GB system which is a physical DC and SQL 2008 server as well. My choice was because I cannot put any more RAM in the system and it does not support H/W virtualization or a better CPU that would... ;). Since I am stuck with that system unless I wanted to overhaul and change motherboard, CPU, RAM I think I have maximized it. Time will tell if I loaded that system with too many roles... I am monitoring it with SCOM 2007 R2. I also just found out that I can't even gt another hard disk in the case... :(. See below: http://owsug.ca/blogs/brad/archive/2010/03/07/Latest-tinkering-and-Virtual-Lab-craziness-_1320_-Make-it-work-on-a-budget_2620_.aspx

brad
brad

Hi sfarhanj, the amount was to build a secondary system. The cost was mostly for a small form factor case, a small form factor board that could accomodate a good amount of affordable RAM and a powerful CPU. I bought a QPack2 case with an ASUS Intel board, a Quad core CPU, and 16GB of DDR3 800MHz RAM. I also bought 2TB of local storage for that cost. This was at the beginning of 2009. My lab has served me well since then. The choice of equipment is mainly because I needed something that I could use in my home lab and travel to a customer site for demo with if required. As an independent consultant, these are the strange requirements that I am faced with :).

brad
brad

Hey Shane, I can see that you're partial to CentOS. Most of my customers use Server 2008. As a lightweight solution, I could have used Hyper-V server which compares to the VMWare microkernel. Also, as a virtual machine MVP I must be aware of the Microsoft offerrings (it's a Microsoft award). Server 2008 is an additional OS with host OS to consider as additional overhead that is true. In the server OS evolution, Server 2008 R2 is without a doubt the most solid and secure flavour yet.

david.hunt
david.hunt

I have to agree. I would never consider Windows as a virtualisation host (except I do use VMware Workstation on WinXP-Pro on my laptop). Too much maintenance that requires reboots and down-time for all Guests. Linux also uses less resources and runs faster on given hardware.

paulenet
paulenet

...is because that is what most corporate environments either are using, or will be using in the very near future. Also, since most environments will run Windows VM's, it makes more sense to run Win2K8 and Hyper-V on bare metal to support such VM's. True, there are environments that run Xen and other esoteric hypervisors, but most people want to set up environments at home so they can develop or maintain skillsets that are going to be in the highest demand. What dictates the highest demand is what is used the most in the corporate world, so it is no surprise why people want to build out their environment with Windows Server 2008 and the rest of the Microsoft product stack. Also, VMWare is heavily used in the corporate world, and has been ahead of Microsoft on this front. I have had extensive experience with it, and believe that VM ESX is a great tool, but after comparing it to Hyper-V and Win2K8, my money and my R&D is primarily betting on Microsoft for the future, and I believe that most companies will follow suit, and therefore maintain a higher demand for IT professionals that have experience with technologies such as Win2K8, Hyper-V, System Center, SCVMM, etc.

DNSB
DNSB

The commonest one I've see is comfort with Microsoft's offerings due to familiarity with them in the work environment. They may be using VMWare products in a work environment but want to learn more about alternative virtualization products. You never know when an extra bit of knowledge will come in handy. My current test setup is Linux/Xen based. I've played with/used linux since I first booted it from floppy so comfort level wasn't an issue. I've also used VMWare and Microsoft virtualization products in that test setup and can say I am qualified to be dangerous in all three environments.

itstechnical
itstechnical

I truly appreciate the 'home virtualization' piece as well as the intelligent comments posted by other working pros. It seems too often a good article is written only to be dumbed-down by off-the-wall comments wondering why some Microsoft product was used instead of Unix/Linux/Apple (fill in your preference). I am looking to do this very thing in my lab and now I can really save some time & $$$. Thx!

brad
brad

Wow, am I glad I read this... Thanks Jimbo!

david.hunt
david.hunt

Some good tips about the Dell systems on which you run ESXi. I've played with OpenFiler and it looks good, but don't yet have the need for a separate filesystem.

brad
brad

Thanks HTH, when I said lab on the cheap... this is my home office :). The power stuff is iffy but I am not loading anything right now since I am using PC class hardware. If I start buying servers, I suspect I may have my marriage to work on in addition to my home lab :). Actually if my requirements get any tougher, I am looking into sharing a rack with some other ITPROs here in town in a hosted model. THEN I would get a couple of servers and go to town. I'll look into the storage components that you suggested, btw, thanks!

brad
brad

Thanks Jeff181, that is in the plans. I have SCOM 2007 R2 installed and I intend to do some customer facing demos of Linux and Unix monitoring.

bulk
bulk

is your friend! It's aimed exactly at people like me (and possibly you) who are trying to learn at home on limited budgets. Here in Switzerland it's the equivalent of about $500 pa and has copies is just about every item of MS software you might need available for download and legal use within your lab environment. I can't understand how any MS tech could NOT have a subscrition! Richard

moabrunner
moabrunner

Because I have the same issue you do $$$ I have been using Centos 5.4 linux and Xen. You can put a server session and do what you need with it and if you screw it up, just reload the session.

brad
brad

Actually, the cost is purely hardware as I have a TechNet subscription and I can download any software that I need and am licensed to use just not for production use :).

Scott Jones
Scott Jones

Get a Technet subscription for under $300, for all of the MS software.

brad
brad

Hi Richard, yeah storage server has been on my radar for a while now. I may get into Unified communications server and if I do, that product will be a must.

paulenet
paulenet

...you run out of physical space to add one more hard drive. ;-) It definitely sounds like you are maxing your value out for the hardware infrastructure you currently have, so I would say that is pretty darn good. I think the SCVMM on your 4Gig system with DC role, and SQL 2008 will probably work out just fine. When you add the largest DIMMs your board will handle (in the context of building a host, say with 4Gig DIMMs), it generally costs more per Gig than say, a typical 4 Gig system that has (2) 2Gig DIMMs, of course. When building out a host with Hyper-V, then adding SCVMM, as you know, there are some pretty beefy prerequisites to get all the SCVMM features (SQL Server, IIS, etc.). When I initially built out my two Intel host servers, I noticed quite a bit of memory usage with SCVMM added, as apposed as just having the hosts with DC and Hyper-V roles. With SCVMM on them, RAM usage went up about between 2Gig to 3Gigs, which I was not exactly thrilled about, even though my hosts at the time had 16 Gigs RAM each. Since then, I have filled them both out to 32 Gigs RAM each, but I now keep SCVMM running on my main dev machine, as I would rather run the risk of dogging it a bit, rather than clobbering my host, if that makes sense. In any case, it sounds like you are really milking out your equipment as much as possible. The longer you can hold out before having to at least switch out the motherboard, CPU, RAM, etc., the better savings it will be in the long run, at least until you have determined that you need heavier metal and are ready to bite a bigger bullet. Until then, squeeze all the VM's out that you can with what you have. Just be sure you have enough fans to keep things cool with all those hard drives in confined spaces. ;-)

brad
brad

Hey Shane, neat suggestion. I've been meaning to get into Xen. As a virtual machine MVP I need to know as much as possible about all facets of virtualization. I used to use Citrix when working at Ottawa U. Ping me at brad@bradbird.ca. Tell me about your setup or post here.

david.hunt
david.hunt

Although you can now get ESXi cost free, ESX only runs on qualified hardware, so for a home lab situation, it will be the hardware that is the major cost, neglecting power costs. Although people often only consider hardware / software costs at home, let me tell you that if you run systems at home on a 24/7 basis, as I do, it is very easy for that to be a significant percentage of your power bill. It would be worth doing the sums before you pay more in power than the cost of the equipment. It was the power cost aspect that drove me to virtualise at home. Running one system 24/7 is much greener (and more cost effective) than running two or three.

david.hunt
david.hunt

I too have been using CentOS x86_64, currently at 4.7, with VMware Server and only 4GB of memory with an Intel dual core CPU. I can expand to 8GB if needed on the current motherboard. {The system is five years old and going strong. All I've had to do is replace a couple of failed disks, which isn't too stressfull, since I'm running RAID-1.} As I'm mostly doing Linux work, it doesn't slurp as much memory as Windows. That said, it is rare to see a vendor of Windows applications who actually knows the real resource requirements of their application. They mostly just say 4GB memory... I've even had one suggest 8GB with the 32bit version of Windows !! One pleasant surprise was a vendor who supplied a set of benchmark results, from which I could derive a good estimate of production resource requirements. Obviously, you can do with less resources in a test lab scenario, as long as you aren't doing stress testing and have only a couple of concurrent users. It works really well for integration work. I've found this setup works really well for up to five concurrent VMs as well as the base system being my home Samba Domain Controller. Over that and things really start to grind.

Walthy
Walthy

Technet Plus subscription for less than $300 until 3/31/2010. Regularly $349. It is worth every penny. I recently learned about it and am setting up a prototype network in my home office so I can work on clients systems to test out new ideas like virtualization and to see what works and what doesn't. All of the MS Dynamics, Servers, and Applications are available for testing. It's a great deal. It is an annual subscription rate but it is great to be able to try out upgrades and new configurations without putting out any additional money up front. This program is more for network admins than developers as it doesn't include programming tools.

brad
brad

The TechNet subscription is worth its' weight in gold.

brad
brad

Thanks paulenet, the memory specifics are exactly the kind of feedback I like to see.

brad
brad

Hey David, thanks for the heads up. I am fortunate to live in Quebec, Canada which is a huge global Hydro electricity supplier. My electricity bill never exceeds $80CAD/month. Also, since my lab is on the cheap (using PC class equipment) I am doing fairly well since I do run these systems 24/7. Your coments are still very relevant and if my needs do increase within the next year or two, I am considering partnering with a couple of other ITPros in Ottawa and splitting the cost to have a rack with several servers hosted in a datacenter. Then I suppose I'll have to start buying actual servers :(.

brad
brad

awesome setup. Great use of virtualization!

Editor's Picks