Fedora 13, Virtualization, and Local DNS

My New Setup – Part 1

I’ve had a blog for quite some time, and realized that I haven’t been posting as often as I could. Which is why I figured – why not post my progress on my current geeky endeavours?

Recently I have been having a lot of fun and putting a lot of man hours into setting up a new private testing environment on a cheap Compaq tower running a 3.2 GHZ AMD X2 processor. For a while, I have been using a lesser of the known Linux distributions called Scientific Linux. I chose this distro back then for two reasons. First because I am most comfortable with RHEL based distros. Second because it was in between the stability of Cent OS and staying more up to date with the likes of Fedora. I have run Cent OS before, and I was unhappy with how long major updates made it into their repositories. Not a major factor for some people, and actually a good thing for most, but in my case it was not an ideal testing environment. Back then I didn’t think Fedora was a stable enough platform, but things have changed, and now I am all for Fedora 13.

Fedora 13

First thing was to run the downloaded Fedora 13 Live disk to just check out the desktop and see if would like it. Not a big surprise there, I did very much. It was clean and polished, and came with many nice features for a ‘noob’ as well as a seasoned systems administrator. So on my first try I wiped my system saying goodbye to Scientific, and installed directly from the Live disk. This was a bad idea. After the install I realized I completely missed the functionality of the Anaconda installer from all RHEL based distros. I didn’t get to pre-configure anything. I started uninstalling much of the unnecessary bloat, before I eventually stopped wasting time and went in another direction altogether. I found that on the fedoraproject.org downloads section they push mainly desktop versions or spins as they call them. The Live disk is the most prominent of the available downloads. I tried as much as I could to find another installable disk version on their site, but I had to give up and look elsewhere. It wasn’t too hard, there are hundreds of mirrors all hosting different flavors of Fedora, and the Fedora project is quite large. It wasn’t long before I found a plain ol’ vanilla Fedora 13 install DVD ISO image ready for my download click.

So now with the right version of the install disk, I tried again, and Anaconda was there as assumed. I was able to configure the repositories I wanted, and configure the package groups before actually installing. It was a lot easier to clean up the system after this installation. There are a few things I immediately uninstalled that came by default in Fedora 13. The most notable of these being the NetworkManager. This is a network management application for the purpose of ease-of-use on the desktop. Since this machine was going to be on a static IP for a testing environment, I didn’t need yet another layer of network configuration above what is already provided by the network service itself. With NetworkManager uninstalled, I had to turn on the network service in run levels 3 and 5. There are of course a few other bloated pieces I will not mention because it is all a matter of opinion and personal taste anyway.

With the desktop slimmed down to something I was happy with, it was time to start installing the software I actually needed. The first thing to do was to install and configure a VNC server (the package named vncserver), and configure my user account for an automatic session on boot. There are plenty of examples of how to do this on the web, but the process has become even more simple in Fedora 13. In this distro, after you run the command vncserver as a specific user, the ~/.vnc/xstartup file is already pre-configured for a regular GNOME desktop session. I guess people got the hint that there is no reason for VNC without a desktop, isn’t that what SSH is for ;)? The VNC setup is so I can eventually unplug the monitor from this machine and use it for another workstation, but have this machine ready to manage through screen sharing.

Logical Volumes, Virtualization, and SELinux

The next thing on the list was to hook up another hard drive and prepare it for a new logical volume group to keep things separate from the main system’s. That’s when I noticed the disk utility provided by Fedora did not allow for the creation of volume groups, only logical volumes on pre-existing volume groups. So I had to install lvm-manager to do this in a decent GUI, and I created a volume group called ‘vg_servers’ on my hard drive that I formatted with the disk utility application.

With the volume group set up I was finally ready to install Qemu KVM, and Virt-Manager for Virtualization. At first I thought it started getting too easy from here. I defined my ‘vg_servers’ volume group as a storage pool in virt-manager changing the ‘onboot’ option to favor this pool instead of the default. Then I created a new 8 GB logical volume to use for storage for a new 256 MB virtual machine. I checked the default virtual network that qemu-kvm sets up automatically and pinged the virtual gateway to make sure it was ok. Then on the new VM, I installed a minimal version of Fedora from the same install ISO I used before. This time with nothing but the kernel and base packages. Everything was going swimmingly until I started to get SELinux errors on the host machine upon trying to configure the files on the newly created VM. I found out that this is because the physical disk the volume group of ‘vg_servers’ was using, and the volume the VM was using for storage all had the wrong SELinux context. I researched this a while until I found that I needed to add the context of virt_image_t to the physical disk mounted (/dev/sdb[0-9]+), as well as to the volumes mounted within the group (/dev/dm-[0-9]+), and the actual logical volumes themselves /dev/vg_servers/node-[0-9]+). I did this using regular expression matching with the SELinux Manager included in the desktop, pretty painless, and viola! No more SELinux errors upon writing to the VM. I also did an entire SELinux file system relabel just to safe, and because I am anal.

Virtual Networking

After all of this I could configure the VM to my liking, and after which, I tried some virtual networking. First I made sure I could ping the VM from the host, this was good. Then I tried to ping the host from the VM this was good. Then I tried to ping an external domain from the host, this was good. Then I tried to ping an external domain from the VM, no good! I was scratching my head for a while until I realized I hadn’t bridged the real ‘eth0’ interface on the host. So I created the file /etc/systemconfig/network-scripts/ifcfg-br0 and added all the necessary settings for the bridge. This is what it looks like:

DEVICE=br0
TYPE=Bridge
BOOTPROTO=none
DNS1=127.0.0.1
GATEWAY=(Local IPv4 router address)
IPADDR=(Local IPv4 address)
NETMASK=255.255.255.0
ONBOOT=yes
IPV6INIT=no

I turned off IPv6 because my router doesn’t support it anyway. This is also something to note. Fedora 13 comes IPv6 ready for all possible implementations. You will also realize why I am using loopback for DNS1 later in this post. Then I edited my /etc/systemconfig/network-scripts/ifcfg-eth0 file to use this bridge. This is what that file looks like:

DEVICE=eth0
HWADDR=(MAC address)
ONBOOT=yes
TYPE=Ethernet
IPV6INIT=no
USERCTL=no
PREFIX=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
NAME="System eth0"
UUID=(UUID output from uuidgen)
BRIDGE=br0

After creating the bridge, I had to reboot the system for it to initiate correctly. Once up and running again, I started to ping everything again, and it all seemed to worked fine so far (I hadn’t yet used loopback for DNS1 for this testing). I could hit the VM from the host, as well as external domains, and I could still hit the host from the VM as well external domains.

DNS and Port Forwarding

With networking running smoothly, the next thing on my list was installing and configuring BIND. What a headache is all I have to say. I don’t think this post is long enough to include all the different problems I was having with BIND, but I will try to summarize a few.

First of all, iptables needs port 53 open. Once I modified iptables I noticed the virtual networking was broken again. I restarted the system and I found that there was now duplicate entries for port 53 in iptables. That’s when I realized something with qemu-kvm and/or virt-manager was modifying iptables on boot. It was automatically opening the ports of 53 and 67. I still haven’t found out where or why it does this, but I just had to be careful from then on before modifying iptables.

The second thing to do was to define a zone for all of my static IPs on the local network of xx.xx.15.0, along with a reverse lookup zone. Then I defined another zone for all of my IPs on the virtual network of xx.xx.122.0, also with a reverse lookup zone. They are the zones of ‘lan’ and ‘vlan’ respectively. I then pinged all of the newly configured domains, and received all the expected responses. Not too shabby 😉

With BIND up and running, it was now a decision of where and how I wanted to eventually host my HTTP testing environment. I decided that the host machine will act as a simple router and firewall only forwarding local requests to a shared IP on the virtual network. I set up this forwarding only for port 80, and again with iptables. The destination of the forward was then, to my only configured VM. Afterward I tested on a completely different work station and set the primary DNS to my Fedora host machine’s IP on the local network. From that workstation I traced the route on port 80 from the domain I that I had just configured in BIND, which also pointed to the Fedora host machine, everything was as expected.

Cloning Virtual Machines

Now it was time to try to stretch this virtualization setup to its so-called limit. I cloned my configured VM to something I called ‘node2’ to see if there were any problems, and there was. Although I named the VM ‘node2’ the hostname after login was still the same as the original VM, and the network interfaces were screwed as well.

What happens is that virt-manager automatically adds a new network interface with a new MAC upon the time of the cloning. When doing this, it also modifies the file /etc/udev/rules.d/70-persistent-net.rules in the clone to add the new interface under the name of ‘eth1’. It uses ‘eth1’ because ‘eth0’ already exists from the orginal machine being cloned. The problem is that this process does not simply create a new interface, it replaces the old from the original VM. It also doesn’t even create the /etc/systemconfig/network-scripts/ifcfg-eth1 file for the new configuration. Upon starting up this new clone, the only interface active is the loopback. I could not start ‘eth1’ on the new machine because it didn’t exist. This all seemed kind of dirty, so I decided to clean things up.

I went with the route of completely deleting the new rule added in udev for ‘eth1’, but changing the MAC address in that same file for ‘eth0’ to the address that ‘eth1’ was defined for. Then I also changed the MAC address in the ifcfg-eth0 network script to same, as well as the IP address. Then I changed the hostname of the VM in /etc/systemconfig/network to match the new VM’s name. I double checked the network run levels, rebooted, and there was the right hostname along with the ‘eth0’ interface running with the correct IP and MAC address. I started the other virtual machine and with no interference. This is a minor set back in the cloning procedure that I was hoping for. Never the less, with this procedure in place, I quickly made another clone of the same machine in under 5 minutes. Again, not too shabby…

With a small cluster of VMs and port forwarding from the host to the virtual network in place, I was ready to start with the web server configuration. This is where I am currently at right now. I have been researching web servers to use as an alternative to the majors (Apache and IIS), but I believe that is for another post 😉

Stay tuned for more geekishness.