Hi all

I played around a little bit with Ubuntu and PXE to learn how diskless linux clients are working. Furthermore I had the idea of having a BOINC cluster with diskless crunching nodes.

I know that Dotsch is already providing a distribution based on Ubuntu. My primary goal was knowledge, the secondary goal was a working solution. You don't have to read my tutorial, nor do you have to use it. I know some things are ugly and you would do them different, especially if you are a linux system developer or a administrator. If you are not interested in my tutorial, but in diskless crunching nodes, please take a look at http://www.dotsch.de/boinc/

---

Introduction

Because I wanted to test everything before buying the real hardware, I decided to simulate everything with VMware Workstation 7.0.0. I took Ubuntu 9.10 Desktop in 32Bit, because my 64Bit Dell Precision 670 Workstation doesn't allow me to activate the VT flag for my Xeon CPU's. Therefore all my virtual machines are doomed to run in 32Bit mode. If you managed to setup everything with 64Bit, please let me know. Even if you find a BIOS hack to enable VT on Dell Precision 670, please let me know. :-)

The first idea was to have a single read-only system for every diskless client, except for /home. This worked very well, until i tried to execute BOINC. Unfortunately BOINC reported all the time that a BOINC instance was already running. Because the filesystem where BOINC was stored (/home) was read- and writable, but the error continued, I decided then to have my diskless linux system read- and writable. I still don't understand where BOINC needs write access to, except for the BOINC (and working) directory , where it was installed.


Step 0

The (virtual) master node uses 1024MB RAM, a 20GB harddisk and two ethernet interfaces. The first interface (eth0) is in bridged mode, the second interface (eth1) is bound to VMnet9, which is not related to bridged, NAT or host-only. Every virtual machine with a ethernet interface on VMnet9 is therefore in a (virtual) private LAN with no direct access to any other LAN or WAN except through the master node. All clients node have 1024MB RAM, no harddisk and one ethernet interface (eth0) bound to VMnet9.


Step 1 - Ubuntu Installation

Install Ubuntu as usual on the master node, but don't run any updates on the system. This is because the clients are using the same kernel as the master node and updating the clients system caused me some problems. Therefore wait with updating the master node until everything is working.

Because the diskless system uses just the main branch of Ubuntu, we want to use the same on the master node. Open a console and type in the following:
Now delete (or comment out) everything in the file. Just the following two lines should be in there:
Now do the following back in the console:


Step 2 - Network configuration

You need to configure both ethernet interfaces. You can do it with the file /etc/network/interfaces or with the Network Manager. To have things simple I did it with the Network Manager.

eth0: DHCP (or if you wish you can use here a static IP)

eth1: 192.168.2.1 (Address)
255.255.255.0 (Netmask)
192.168.2.1 (Gateway)
192.168.2.1 (DNS)

Note: If the network you connect eth0 to already uses the 192.168.2.0/24 network, you have to use a different one for eth1. All the instructions here assume that the network on eth1 is the above one. If this is not the case for you, every later instruction has to be corrected for your case.


Step 3 - DHCP basic configuration

Now we have to tell DHCP server which interface it must use. Back in the console type in the following:
Change the following line:
to this one:


Step 4 - DHCP detailed configuration

Again within the console:
Delete (or comment out) everything in the file and add the following:
Note: For each new diskless client you have to add a new host entry with different IP address, hostname and MAC address. You have to replace 00:00:00:00:00:00 with the IP address of your diskless client.

Now you can restart the DHCP server with typing in the following command in the console:
Note: Everytime you change the config file, you have to restart the DHCP server.


Step 5 - NFS configuration

Now we make our NFS filesystem up and running. In the console type in:
To grant clients access to the folder, we have add the location to the config file:
Add the following line:
After saving the config file, execute the following within the console:


Step 6 - TFTP configuration

Within the console type in the following:
Change the following line:
to this one:
We have to comment out an entry in inetd.conf. Type in
Change the following line:
to this one:
Back in the console execute the following:
Now we can restart the TFTP server:
Note: It is possible that the (old) TFTP config within /etc/inetd.conf is still active. If you experience any problems here. Just restart the master node.

Step 7 - Kernel configuration

Type in the console:
Write down the following content in the file:
We use the same kernel as on the master node. In the console do the following:
The difference is the initrd.img.
Change the lines:
to:
Back in the console do:
Now do the opposite of the above:
Change the lines:
back to:
finally we do:


Step 8 - Client system installation

Execute the following within the console and wait:


Step 9 - Client system configuration

Add the following to the file:
We must configure the ethernet interfaces of the clients:
Add the following to the file:
To avoid an apparmor error on boot up, we have to do the following:
Add the following to the file:
Now we set the file executable:
To allow the DHCP assigned hostname to be set, we have to empty the following file.
Note: If there is an empty line in the file, the hostname is empty. You have to completly remove everything within the file, even spaces and empty lines.


Step 10 - Gateway installation

To allow the clients internet access, we set up the master node as gateway:
Replace the following line:
with this one:
Then do:
Our master node acts now as router, but we want NAT too.
To have the NAT restored at reboot, we have to do:
Add the following to the file:
Now we set the file executable:


Step 11 - User Management

As we don't have a user management like LDAP, we must make sure that the user and group id's of the diskless client are the same as on the master node. First of all, we add a boinc user on the master node, which is called boinc-client.
Then we have to find out the user and group id's of this user.
This returned me the following:
Now we know the id's we must use for the clients. Now we can change to the diskless system with:
Then we add a group and a user corresponding to the master node setup:
The master node and all clients are using the same user and group id's for the boinc user.

We leave our diskless system with:
As you may noticed. The boinc user on all clients has its home at /diskless/nfsroot/home, in case you are on the master node, and on /home in case you are on the client nodes. But this is the same location differently mounted respectively exported by NFS. The boinc user on the master node has its home in /home, but not in /diskless/nfsroot/home. But we want a system wide home storage for the boinc user, therefore we must linking the home to a different location.
Note: Every normal user you are adding to the master node, will have it's home on /home and will not be seen by the diskless clients. You may say that you can specify the home by adding --home to adduser. This is correct. However the startup script for the master node would be different then. You can write a script which will work, but the script must then findout itself, where the home of the boinc user is. My script assumes the home in /home.


Step 12 - BOINC installation

Every node must have its own BOINC installation. If different nodes are using the same (copied) installation, then different nodes would have the same computer id, which confuses the BOINC server somehow. At least that was what I discovered. Therefore every node must have it's custom installation. But we want to do this a little bit automated.

First of all, we download the desired BOINC version:
Then we rename it:
Note: Another thing you can do is to have multiple boinc installers within /home/boinc-client. You can then add a symlink to the desired version and name it boinc-installer.sh. This is maybe a better way to have an overview of the version. It may be necessary to change my startup script then, because it sets execution rights for boinc-installer.sh. To be fine you can change the rights right after downloading BOINC.

We must add a startup script to all nodes. We do it first for all client nodes, and copy it then for the master node:
Change the following line:
To this ones:
As you see, the if part starts an already (configured) installation of BOINC. The else part just installs a clean BOINC to /home/boinc-client/<node>/BOINC. I tried to automate the attach to a new project too. However it failed. Therefore this script is just useful to start already configured installations and to do clean installations on new nodes. If you want to participate into multiple BOINC projects it would be even better to do it manually. Once a node is running you can attach new projects trough the BOINC manager easily.

Now we have to copy the file to the master node as well:


Step 13 - SSH configuration

To manage the diskless clients, we added a SSH Server during the debootstrap process. Now we have to configure our client nodes to allow remote access (without) password.

We generate a SSH keypair. Don't use a passphrase. You can use one, but then you will be asked all the time unless you are using a key management tool on your master node.
We copy the public key to the client system.
You should be able to login passwordless to all client machines from the master node.

If you want to login passwordless from every node to every node, you have to copy id_rsa and id_rsa.pub to the client system too:
Note: This setup allows you to connect passwordless from the master node to any client node, and to connect passwordless from any client node to any other client node. It does not allow a passwordless connection from any client node to the master node. If you want to have that as well, you have to add the authorized_keys file on the master node too.


Step 14 - BOINC configuration

As we now have SSH access to any node, we can start with the (initial) configuration of the BOINC installations. Start every diskless client you want to configure. For every client you started, a new folder with the hostname of the diskless client will be generated within /diskless/nfsroot/home/boinc-client/ respectively /home/boinc-client/. If you see the BOINC folder in there, you can start with configuring the corresponding node. Login to it via SSH:
This will attach the node to a certain project, downloads the corresponding files, and will start to download work units. After the download of the work units is finished, you can press CTRL-C to quit BOINC. To manage this BOINC installation a password must be set.
The node is now ready for crunching. Just restart it and the BOINC crunching will start automatically.
Note: Repeat this step for every node. If you don't to it, it will idle. The same procedure can be done for the master node, if you wish to do so.


Step 15 - BOINC management

If you installed BOINC on the master node, you can run start the BOINC manager as follows:
The nohup command allows you to close the console terminal. Otherwise the BOINC manager would get killed if you close the console terminal.


Step 15 - Improvements

If we are managing a node, we always are using their IP addresses to connect to them. This is true for SSH, but also for the BOINC manager. As we don't have a DNS server, we can workaround this by using /etc/hosts.
Add a line with IP and hostname for every node you want to manage with its hostname.

You can do the very same for the clients.
Do the same as above, but add an additional line:


Thanks

The following websites helped me.

• https://help.ubuntu.com/community/DisklessUbuntuHowto
  
• http://web.mit.edu/rhel-doc/4/RH-DOCS/rhel-sg-de-4/s1-firewall-ipt-fwd.html
  
• http://www.esl.eu/de/forum/1206/8583/442025/
  
• http://ubuntuforums.org/showthread.php?t=380101
  
• http://www.cyberciti.biz/tips/find-out-if-fil...ditional-expressions.html