[Testbed-admins] Emulab Rebuild Problems

[Barry Trent <btrent@triticom.com>]

I'm working on re-building our small emulab with new hardware here at 
Architecture Technology in Minnesota and I'm having some trouble.

The first big problem I think I have solved: We are using Cisco 2948 
switches for the experimental network. I figured the type field in the 
node_types and nodes tables should be 'cisco2948'. Wrong. The 2948 is 
actually in the 4000 class of Catalyst switches! The difference appears 
to be that it uses "community string indexing" for some of its MIBs. 
(Described here: 
ftp://ftp-sj.cisco.com/pub/mibs/supportlists/wsc4000/wsc4000-communityIndexing.html). 


So -- note for posterity: For the Cisco 2948 switch, set its type value 
to 'cisco4000' in the type fields of the node_types and nodes tables.

Now the new problem I'm up against:

We PXE boot our testbed machines and they load the freebsd.newnode and 
appear in the "New Testbed Nodes" page of the web interface. We "Search 
switch ports for selected nodes". Our 5 experimental interfaces are 
properly discovered but the control interface isn't. I figure this 
shouldn't be a big problem(?) -- we enter this manually.

When we try to actually "Create" the node the operation appears to succeed:

-----
/usr/testbed/www
pc20 succesfully added!
Re-generating dhcpd.conf
Restarting dhcpd: /usr/local/bin/sudo -S /usr/local/etc/rc.d/2.dhcpd.sh stop
Restarting dhcpd: /usr/local/bin/sudo -S /usr/local/etc/rc.d/2.dhcpd.sh 
start
 dhcpd wrapperSetting up nameserver
Running exports_setup
Rebooting nodes...
Rebooting 192.168.10.109


Finished - when you are satisifed that the nodes are working
correctly, use nfree on boss to free them from the emulab-ops/hwdown
experiment.
-----

BUT, the node never reboots. It just sits there at the login prompt of 
the freebsd.newnode boot image. It never "reports in" to the hwdown 
experiment (although we can see that there is now a machine in the 
hwdown experiment, the idle time just keeps going up). No entry for the 
machine actually gets placed into the re-generated dhcpd.conf file 
either -- I presume that one should. If we reboot manually we go right 
back to the newnode image.

An ideas/suggestions on how to troubleshoot this problem?