In this series I describe my attempt to learn VMware ESXi by building a small, low-power virtualization box running FreeNAS and MythTV. I have pretty much no previous knowledge of virtualization and ESXi, so I’ve started basically from scratch by googling and reading stuff around the web. This means that I learn as I go and I fully expect to make some mistakes along the way.
I’ve been running a MythTV DVR (digital video recorder) for 6+ years. It’s a small, silent box in the corner of my living room that records TV programs, plays back videos and acts as a music server. It has gone through several hardware changes and currently contains, among other things, an ITX form factor motherboard, a low-power Sandy Bridge i3 CPU and a quad DVB-T2 PCIe tuner.
A year or two ago I built myself a small NAS server based on FreeNAS. I tried to keep it as simple as possible since I didn’t need anything too special: I just wanted a centralized place to hold some documents, videos and music, and some storage space for backups. So unlike the eight-disk monster builds with SAS controllers, ECC memory and whatnot, which you can read about around the web, I bought a second-hand motherboard with integrated AMD E350 CPU, 4 GB of cheap RAM, a single Western Digital Red 3TB drive and a chinese PicoPSU knockoff (meaning I run the computer using a laptop power brick). Worked like a charm.
Now, both of these systems work fine. Great even. So I didn’t have any immediate need to change anything.
I had become interested in virtualization some time ago and started wondering if I should try out some stuff just to learn. Sure, you can download and install VirtualBox on just about any computer, which I had tried a while back, but it didn’t seem interesting and exciting enough. I wanted to learn ESXi, and what better way than to build a dedicated virtualization whitebox which would incorporate the functionalities of the MythTV PVR / HTPC and my NAS.
So I decided the requirements for the build:
- It should support running a MythTV backend with DVB recording capabilities
- It should support running FreeNAS with ZFS, and have good performance (meaning an 80+ MB/s transfer rate)
- It should have some spare capacity for playing around with other virtualized computers
- It should be quiet and power-efficient, and for an extra challenge, be small in size so I can hide it in the corner of my home office (so no huge full tower case, thank you)
- It shouldn’t cost a fortune
Selecting the hardware
I knew from my research that VT-d was the important keyword when selecting the hardware for the build. It stands for Intel Virtualization Technology for Directed I/O and it’s Intel’s implementation of IOMMU. It allows the virtualization host to assign certain hardware resources directly to a VM (a virtualized machine). In ESXi, the feature is called PCI passthrough – you can essentially give a VM direct access to PCI devices, providing that the hardware supports VT-d.
Why is this important? For one, the MythTV VM needs access to the physical DVB tuner hardware to be able to record TV broadcasts. Also, giving FreeNAS direct, exclusive access to a SATA controller should yield the same sort of performance and reliability as when building a standalone, non-virtualized NAS box.
Both the CPU and the motherboard has to support VT-d. Finding a VT-d compatible CPU is easy; most i7 and i5 CPU’s support it. Finding a reasonably priced consumer motherboard that supports VT-d is a bit harder. Most motherboards support VT-x, meaning basic virtualization technology, but not VT-d. Even if the motherboard chipset supports VT-d, it is not enough since it has to be supported by the motherboard UEFI. Most motherboards do not have VT-d UEFI support; some support it in one UEFI version, only to have it removed in a later update. No idea why.
After some digging I found out that I’d have the best chance with ASRock’s motherboards. I finally found the H97M Anniversary, which had everything I needed: VT-d support (I checked the UEFI section of the manual), a Realtek RTL8111GR NIC which can be made to work with ESXi, 3 PCIe slots, mATX form factor and a very reasonable price (around 70 euros).
I also found a good deal on a second-hand i7-4785T CPU, which has 4 cores, 8 threads and a very low 35W TDP, which translates to low power usage and silent operation. Along with the CPU came a Cooler Master Hyper EVO 212 cooler which, due the huge size of its heatsink, should be able to cool the low-power CPU with a very low fan RPM – or perhaps even without a fan at all (might be worth trying out at some point).
For RAM I bought 16 GB of DDR3, which I thought should be enough for a start: I wanted to give FreeNAS 8 GB, the MythTV backend 2-4 GB and the rest would be free for playing around with VMs. I still had room for two more sticks of RAM in case I needed more.
I decided to use my PicoBox Z3-ATX-200 (chinese PicoPSU knockoff) for power supply (PSU). It has many advantages over traditional PSUs: silent operation, high efficiency, takes up very little space etc. Now I know some of you gasp at the thought of running a virtualization box with an i7 four-core CPU on a chinese PicoPSU that cost around thirty bucks. I realize I’m taking somewhat of a gamble here, but the PSU worked without a hitch in the FreeNAS box and I’m running it with a high-quality laptop power brick. The combination is rated for 120W which is much less than what I anticipated the entire box would use.
For host storage I decided to use a combination of a USB stick for the ESXi installation and an old SSD I had lying around for logs and VM datastore. My reasoning behind installing on a USB stick was that I could buy two sticks and take regular backups (clones) of the one currently in use. If something would happen and the primary stick would fail, I could just pop in the backup stick.
I know some people have built all-in-one ESXi & FreeNAS setups where the ESXi host is booted from a USB stick, the motherboard SATA controller is passed through to a FreeNAS VM from where storage is shared using iSCSI or NFS over a virtual storage network back to the host. My first thought is that it sounds risky – if the FreeNAS VM is down for any reason, ESXi loses access to its datastores, which I assume means other VMs go down as well. I didn’t like the idea of being that dependent of the FreeNAS VM, so I decided the safest bet would be to have a separate datastore disk for ESXi. This meant that ESXi would use the onboard SATA controller and I had to get a separate PCIe controller for the FreeNAS VM.
Finding a suitable SATA controller was a bit tricky since it has to be supported by ESXi. Natively ESXi supports fancy LSI controllers that cost three-figure amounts, but no cheap consumer-grade alternatives. However, I found out that by using some ESXi installation modifications, courtesy of the ESXi homelab community, support for various consumer SATA chipsets could be added. So I decided to try my luck and order a no-name PCIe SATA controller based on the ASM1061 chipset from eBay for a very reasonable 10 euros. As for disks, I wanted to use my old WD Red 3 TB disk, but this time ZFS mirrored, so I went out and bought another one.
As for the MythTV VM, I decided to use my existing TBS 6285 quad tuner DVB-T2 PCIe card from my old MythTV build. The VM would primarily use the NAS for storage but run on a virtual disk big enough to have space for a few days worth of recordings, in case there are problems with the NAS.
Here’s a summary of the hardware I ended up using:
- Motherboard: ASRock H97M Anniversary
- CPU: Intel i7-4785T
- Cooler: Cooler Master Hyper 212 EVO
- RAM: 2 x 8GB Kingston HyperX Fury Black DDR3-1600 CL10
- PSU: Z3-ATX-200 wide-input with 120W laptop power brick
- USB (ESXi boot disk): Sandisk Ultra Fit USB 3.0 16 GB
- SSD (ESXi datastore): Samsung 830 256 GB
- PCIe SATA controller (FreeNAS VM): no-name ASM1061 with two SATA ports
- HDDs (FreeNAS VM): 2 x Western Digital Red 3 TB
- PCIe DVB tuner (MythTV VM): TBS 6285 quad DVB-T2 tuner
I’ll be putting the hardware together in the next part of the series so stay tuned.