Diskless Frontend

From MythTV Official Wiki
Revision as of 19:32, 26 March 2007 by Wesmoc (talk | contribs) (Pulled the selinux argument.)

Jump to: navigation, search

This page describes the various methods for creating diskless frontends on a variety of Linux distributions.

Introduction

Understanding the architecture

A diskless boot system is comprised of a couple different component that enables the entire boot process to work.

  • First the machine requires a boot mechanism. This can be PXE, which is a feature of the network card to boot entirely from the network. This could also be a boot floppy, CD, USB key, or even a small hard drive used only to kickstart the boot process.
  • If the machine is booting using PXE, the PXE boot code acquires a DHCP/BOOTP address from a DHCP server that contains information on where to find the bootloader. This is similar to a Grub or LILO boot screen normally found on a Linux system. This PXE bootloader defines where the init filesystem (Initrd) and the kernel are located and can be downloaded using TFTP
  • If the machine is booting using a local media method, the bootloader on that media directs the boot environment to the init filesystem and the kernel.
  • Once the init filesystem and kernel have been downloaded and loaded onto a Ramdisk, the remainder of the filesystem is mounted via NFS from the server.
Important.png
WARNING: If you are thinking about diskless and have an Nvidia card, there is a known bug installing their driver on such a system (see Installation fails on NFS). This should only apply to building the driver on that system - installing from an RPM should be fine.

Creating a test system

I found that the VMware Player can provide an excellent system to test PXE booting and the network boot in general. The VMware Player provides a virtual PC environment that also emulates a PXE network adapter. By using this environment, you can quickly test changes to the DHCP, TFTP, and the kernel itself. Once you can get this virtual machine to boot, it is a simple matter to get a physical machine working.

Use this PXE.VMX file as a starting point for a machine you can test with

config.version = "8"
virtualHW.version = "3"
ide0:0.present = "FALSE"
ide0:0.filename = "WindowsXPPro.vmdk"
memsize = "132"
MemAllowAutoScaleDown = "FALSE"
ide1:0.present = "TRUE"
ide1:0.fileName = "auto detect"
ide1:0.deviceType = "cdrom-raw"
ide1:0.autodetect = "TRUE"
floppy0.present = "FALSE"
ethernet0.present = "TRUE"
usb.present = "TRUE"
sound.present = "FALSE"
sound.virtualDev = "es1371"
displayName = "PXE"
guestOS = "linux"
nvram = "PXE.nvram"
MemTrimRate = "-1"
ide0:0.redo = ""
ethernet0.addressType = "generated"
uuid.location = "56 4d aa 67 e4 36 4a 07-33 f5 00 d9 37 a3 6c b2"
uuid.bios = "56 4d aa 67 e4 36 4a 07-33 f5 00 d9 37 a3 6c b2"
ethernet0.generatedAddress = "00:0c:29:a3:6c:b2"
ethernet0.generatedAddressOffset = "0"
tools.syncTime = "TRUE"
ide1:0.startConnected = "FALSE"
uuid.action = "create"
checkpoint.vmState = ""

Configuring the DHCP server

In order for any network boot to function, a DHCP server must be running on the network and support the custom configuration options needed for the network boot. While this could be on any operating system, these instructions refer to the Linux DHCPD implementation.

PXE boot

To support PXE booting, the following lines must be added to your /etc/dhcpd.conf file and the dhcpd service restarted

allow booting;
allow bootp;
class "pxeclients" {
   match if substring(option vendor-class-identifier, 0, 9) = "PXEClient";
   next-server 192.168.1.100;
   filename "linux-install/pxelinux.0";
}

The next-server directive indicates the TFTP server that the system will contact. The filename is the init file that will be loaded to provide the bootloader. This example is specific to the Redhat and Fedora Core implementation and may vary depending on your distribution.

//Alternative dhcp.conf, assume your PXE root is /mnt/pxe, and you assign your box a static IP of 192.168.1.60:

#option domain-name "domain";
default-lease-time 600;
max-lease-time 7200;
ddns-update-style ad-hoc;
subnet 192.168.1.0 netmask 255.255.255.0 {
   range 192.168.1.60 192.168.1.60;
   }
  option option-150 code 150 = text ; 
   host mythtoo { 
       hardware ethernet 00:14:2A:B3:E4:1C; 
       option option-150 "/mnt/pxe/boot/grub.lst"; 
       option routers 192.168.1.1;
       next-server 192.168.1.90;
       filename "/mnt/pxe/boot/pxegrub"; 
       filename "/mnt/pxe/pxelinux.0";
   }
Important.png
Note: This configuration gives the option to use both pxegrub AND pxelinux, whichever one your box picks up first. I show this option to give a wider group of people the ability to boot, assuming they have the appropriate files.

TFTP server

The TFTP server for your distribution is required for PXE booting and needs to be installed.

Gentoo specific note: If you blindly emerge "tftp" you will emerge hpa-tftpd. I found that I needed to edit the /etc/conf.d/in.tftpd file and add the -l (listen) option to get the tftpd server to start working.

DHCP server

In order to mount the NFS root partition, the DHCP server will also need to provide a root path and IP address to the booted system

option root-path "/diskless/i386/RH9/";

This option directs the DHCP client to the root filesystem. This example is specific to the Redhat and Fedora Core implementation. Please refer to your distribution documentation for details.

group {
       host vmware {
               hardware ethernet 00:0c:29:a3:6c:b2;
               fixed-address 192.168.1.250;
       }
}

These options provide a fixed IP to the diskless client based on the mac address. Note that this example definition matches the VMware player defined mac address used during testing.

Building the kernel

A diskless kernel requies some options that normal kernels may not include

CONFIG_NFS_FS=y
CONFIG_ROOT_NFS=m (some instructions say this needs to be in the kernel but I had problems with that and didn't need it ultimately)
CONFIG_IP_PNP=y
CONFIG_IP_PNP_RARP=y
CONFIG_IP_PNP_BOOTP=y
CONFIG_IP_PNP_DHCP=y

Make sure to also include your Ethernet driver in the kernel and not as a module. For the VMware Player test system, use the AMD PCNET32 driver.

Distribution specific instructions

Mac OS X

Please refer to the Diskless Mac-Mini Howto for complete details on this environment

Redhat and Fedora Core

Redhat and the subsequent Fedora Core releases provide a fairly straightfoward method of creating a diskless boot environment that can support both PXE and Netboot or Etherboot environments.

These releases provide a redhat|system-config-netboot script which does most of the heavy lifting. It creates a PXE boot image as well as kernel and a snapshot folder for each netboot system. The snapshot folder stores all the host specific files such as /dev, some parts of /etc and /var. The main root filesystem is mounted read only to provide sharing between multiple systems. It uses a pivot-root function to overlay the host specific files and folders on top of the read only portions of the filesystem.

TFTP server notes

The TFTP service is part of the xinetd installation. Enable it by editing your /etc/xinetd.d/tftp file

service tftp
{
        disable = no
        socket_type             = dgram
        protocol                = udp
        wait                    = yes
        user                    = root
        server                  = /usr/sbin/in.tftpd
        server_args             = -s /tftpboot -v
        per_source              = 11
        cps                     = 100 2
        flags                   = IPv4
}

Set the disable option to no. I also added the -v option so that it provides additional verbose logging to /var/log/messages which can help with troubleshooting problems.

Creating the root filesystem

Creating the root filesystem is basically a matter of copying a working installation that you want to make your client boot to. This is mostly likely a hard disk installation on the hardware you will be booting from.

Start by making the directories

mkdir -p /diskless/i386/RH9/root/
mkdir /diskless/i386/RH9/snapshot/

Next, copy the entire filesystem (exclude /proc/ and /sys/ directories) into the new location

rsync -a -e ssh --exclude='/proc/*' --exclude='/sys/*' installed-system.example.com:/ /diskless/i386/RH9/root/

Exporting the filesystems

Using NFS, export the root and snapshot filesystems so that they can be accessed by the diskless system

/diskless/i386/RH9/root/     *(ro,sync,no_root_squash)
/diskless/i386/RH9/snapshot/ *(rw,sync,no_root_squash)

You can replace the * with a hostname, an IP address or range, or domain name (*.myhomedomain.org) to limit access to the host(s) specified if desired.

Creating the boot images

The system-config-netboot package provides the scripts necessary to create a diskless configuration. It requires the installation of the busybox and busybox-anaconda packages. My target and source system was Redhat 9. The redhat-config-netboot script provided with Redhat Enterprise Linux is massively broken. The system-config-netboot script provided with Fedora Core 4 had some additional python dependencies that I didn't care to work out so I chose to use the Fedora Core 3 script which worked fine.

http://download.fedora.redhat.com/pub/fedora/linux/core/updates/3/i386/system-config-netboot-0.1.30-1_FC3.noarch.rpm

Neither the suggested 0.1.30-1_FC3 nor the current 0.1.40-1.FC5 versions of system-config-netboot worked for me on FC5. However, I found a new version 0.1.41-1.FC6 that worked flawlessly.

http://download.fedora.redhat.com/pub/fedora/linux/core/development/source/SRPMS/system-config-netboot-0.1.41-1.FC6.src.rpm

  • If you already have a boot structure in place, be aware that this script will blindly overwrite your previous pxelinux.cfg/default file. Take a backup of the entire directory BEFORE starting the script.
  • Running this script will first ask you to create a diskless configuration from an installed kernel. If you built a kernel with the options outlined earlier and installed it on the server then choose that kernel. I found it is easiest to use the same kernel for both the server and the diskless client whenever possible. It makes buildling kernel modules for the diskless kernel alot easier because the root filesystem on the diskless machine is read-only and it can make some installers throw fits, like nvidida.
  • The linux-install folder under your /tftpboot directory will be populated with the kernel specific files including the initrd and the kernel itself. It will also create the snapshot folders.
  • Next, click New on the next window to appear to create a new host identifier which populates the snapshot folder with information specific to this diskless host that will be booting from the root partition created earlier.

However system-config-netboot-0.1.41-1.FC6 does not work flawlessly on an FC5 or FC6 system. To make it work, you need two changes.

1) FC5 & FC6: Add "ramdisk_blocksize=1024" to the append line of the boot stanza below.

2) FC6 only: Include /sbin/mount.nfs in the initrd.img after it is generated. This can be done as follows (as root).

cd to the boot directory. On my system this is /tftpboot/linux-install/mythfe.fc6

#> zcat initrd.img >| /tmp/initrd
#> mount -o loop /tmp/initrd  /mnt
#> cd /mnt
#> cp /sbin/mount.nfs sbin/
#> cd -
#> umount /mnt
#> gzip /tmp/initrd 
#> mv /tmp/initrd.gz initrd.img

3) FC5 (with newer kernels) & FC6: Several people have found that the snapshot directory is not being mounted rw using the newer kernels. The following is a fix involving modifying the disklessrc file which is run after the kernel boots. It references the same initrd.img file above

# mkdir -p /tmp/initrd-mount
# zcat initrd.img >| /tmp/initrd
# mount -o loop /tmp/initrd  /tmp/initrd-mount

Then you must edit /tmp/initrd-mount/disklessrc and make the following change (adding the line in bold).

...
echo Mounting Snapshot directories
mount -t nfs $NFS_IP:${NFS_DIR}/snapshot /.snapshot -o rw,nolock &&
mount -o remount,rw /.snapshot &&
{
...

Save and exit the file, then...

# umount /tmp/initrd-mount
# gzip /tmp/initrd
# mv /tmp/initrd.gz /tmp/initrd.img

And finally copy /tmp/initrd.img back over the original.

Configuring the PXE bootloader

If you are using PXE, edit your /tftpboot/linux-install/pxelinux.cfg/default file to include a new option

label FC5
    kernel FC5/vmlinuz
    append initrd=FC5/initrd.img init=disklessrc root=/dev/ram0 ramdisk_size=60000
NFSROOT=192.168.1.100:/diskless/i386/FC5 SNAPSHOT=unichrome ip=dhcp
  • label is the text you will type in when the PXE bootloader prompt appears
  • kernel specifies the directory relative to the /tftpboot/linux-install folder where the kernel lies

append specifies a series of options to pass to the kernel

  • initrd specifies the directory relative to the /tftpboot/linux-install folder where the init filesystem is
  • init is the script that is run when the filesystem boots. This disklessrc file is the script that mounts all the filesystems needed for the diskless client
  • root is the device where the init filesystem lives. This is a ramdisk.
  • ramdisk_size is needed to specify the size of your init filesystem. Mine was over 6MB so I had to provide this line to increase the size of the ramdisk to hold it
  • NFSROOT tells the kernel where to mount the root filesystem from. It will append /root to the end.
  • SNAPSHOT tells the disklessrc file which snapshot folder to mount
  • ip tells the kernel how to get it's IP address for netbooting


Alternate Configuration

The following alternate configuration is also useful when the root-path option is present in DHCP configuration, and thus not required in the append section.

label linux
    kernel bzImage-2.6.19.diskless
    append root=/dev/nfs ip=dhcp
  • bzImage-2.6.19.diskless is the kernel image to load which must be present in /tftboot

Important.png Note: In this scenario no initial ramdisk (initrd) is required but the kernel must have the following options built in (not as modules):

  • The driver for the booting NIC
  • CONFIG_IP_PNP
  • CONFIG_IP_PNP_DHCP
  • CONFIG_NFS_FS
  • CONFIG_NFS_V3
  • CONFIG_NFS_V3_ACL
  • CONFIG_ROOT_NFS

And the root-path DHCP option specified as something like

option root-path "192.168.0.1:/netboot/frontend";
  • 192.168.0.1 is the IP address of your NFS server.
  • /netboot/frontend is the export that will become root on the remote server.

Configuring a netboot system

If you are using a CD, USB key, or other media configure your bootloader appropriately. Copy the initrd.img and vmlinuz files from the /tftpboot/linux-install/RH9 folder to your installation media. In my work I happened to use a small local hard disk to jumpstart the process and had my /etc/grub.conf file configured as

title Netboot
        root (hd0,0)
        kernel /vmlinuz-netboot ro root=/dev/ram init=disklessrc ramdisk_size=60000  
NFSROOT=192.168.1.100:/usr/local/media/diskless/i386/RH9 SNAPSHOT=unichrome ip=dhcp
        initrd (hd0,0)/initrd-netboot

The (hd0,0) in the initrd line was very important because once the kernel started, it didn't understand where to find the initrd file unless the specific devices was provided. This is different than the PXE method which knows where both reside.

Gentoo Specific with configuration file examples

DHCP Server

The default /etc/dhcp/dhcpd.conf from emerge dchp proved to be more data than necessary for the simple needs of DHCP I need for *only* the PXE machines on my network (I have a DHCP server on my router that does the DHCP for the entire network, I don't need a second one ;) ). So, I junked the original:

mv /etc/dhcp/dhcpd.conf /etc/dhcp/dhcpd.conf.orig

And created this new one, I'll try to break down the sections below the code box:

#option domain-name "domain";
default-lease-time 600;
max-lease-time 7200;
ddns-update-style ad-hoc;
subnet 192.168.1.0 netmask 255.255.255.0 {
   range 192.168.1.60 192.168.1.60;
   }
  option option-150 code 150 = text ; 
   host mythtoo { 
       hardware ethernet 00:14:2A:B3:E4:1C; 
       option option-150 "/mnt/pxe/boot/grub.lst"; 
       option routers 192.168.1.1;
       next-server 192.168.1.90;
       filename "/mnt/pxe/boot/pxegrub"; 
       filename "/mnt/pxe/pxelinux.0";
    }

This example assumes you do not need the DHCP functionality for your network from this DHCP server, it's only DHCP'ing for your PXE boxes, hence the comment on the domain-name option. The subnet and netmask will be particular to your network, this one assumes you have a simple 192.168.1.1 based network (192.168.1.0/24). I gave a range of 1 IP rather than specifying fixed-address in the 'host mythtoo {' section; why? Because I'm eccentric, or maybe it was an issue with obtaining a valid /etc/resolv.conf ;) The other options should be pretty self-explanatory: hardware ethernet is the MAC address of the client (the diskless machine) option option-150 was my life saving option that i uncovered somewhere online, it was necessary for pxegrub to boot. option routers was where I specified what routers were on my network, after I did this I was able to access the internet from the PXE box, otherwise I was restricted to the internal network (which, for a mythbox might be great). next-server is the IP of the server where your TFTP server lives. This happens to be the same server I have everything on, but it seems necessary to spell it out anyway. filename This is where you specify your bootloader, pxelinux.0 for PXELinux obviously, and pxegrub for Grub :D Notice the full path, not a chroot'd path.

TFTP Server

Default emerge of "tftp" pulls in HPA-TFTPD, and is called by in.tftpd. All that was not exactly easy to find if you don't read your emerge output. From there, the default /etc/conf.d/in.tftpd doesn't work, I had to modify it to look like:

# /etc/init.d/in.tftpd
# Path to server files from
INTFTPD_PATH="/mnt/pxe"
# For more options, see tftpd(8)
INTFTPD_OPTS="-s -l ${INTFTPD_PATH}"
Important.png
Note: The addition of the "-l" option, this enables tftpd to listen (why doesn't it listen by default?). To actually get my PXE client to boot, I had to /etc/init.d/in.tftpd stop which is very odd, but works for me, YMMV.

PXE Bootloading Options

I had a great time figuring out how the PXELinux boot loader works, but once I did, it was solid. By default it will search a chroot'd / directory (the / of your PXE client as specified in your TFTPD file). So, you will be creating a configuration file. You have a slew of options: You can use the MAC address of the Client, in the form of: 00-01-a1-c1-d1-f1-x1 (note the lowercase) Or you can use the Hex version of your IP: C0A8013C (PXELinux comes with an application to convert an IP to Hex: gethostip) Or any octet short of that (so C0A8013, COA801.. and so on) OR you can just use the word default. I chose all 3 ;) So you create a directory called pxelinux.cfg in your / directory, and inside it you place your configuration files as described above (MAC address, so on), which should contain something that resembles this:

DEFAULT /boot/bzImage-2.6.16 ip=dhcp root=/dev/nfs nfsroot=192.168.1.90:/mnt/pxe

Should be relatively self-explanatory. /boot/bzImage-2.6.16 is the kernel image in the /mnt/pxe/boot directory. nfsroot is the exported directory from my NFS server (my NFS server's IP is 192.168.1.90)

Just about done.

Final points

In your /etc/conf.d/net file, the following entry keeps the settings already obtained and starts eth0, thus enabling other network services :

config_eth0=( "noop" "dhcp" )


I also had to emerge nfs-utils to get it to stop complaining about netmount not working. It may take 3-4 tries of booting to get your RC's straightened out to let you through to a login.

Conclusion

Once the system is booted, there may be various tweaks and settings the need to be modified. Working with a diskless system can be tricky but since you have access to the root filesystem on the server and can use chroot /diskless/i386/RH9 to change into the environment it can lighten the load.