[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Testbed-admins] Failure in delay node bootup
- To: Adit Ranadive <adit262@cc.gatech.edu>
- From: "Jason Shupe" <jshupe@ISI.EDU>
- Subject: Re: [Testbed-admins] Failure in delay node bootup
- Date: Fri, 29 May 2009 22:08:52 -0700
On our installation, we run either FBSD410-STD or FBSD62-STD on our
delay nodes, depending on the vintage of the hardware.
On Fri, May 29, 2009 at 10:23:02PM -0400, Adit Ranadive wrote:
> Hello,
>
> Im trying to set a latency on a link in the experiment. According to the
> software it does this by inserting
> a delay node.
> Unfortunately, the delay node does not seem to boot up. There is a failure
> in trying to boot the OS on the
> delay node.
>
> I have 2 questions:
> 1) What should be the value of the delay_osid field for the node type?
> I have set it to FC6-STD. Its not mentioned in the wiki installdocs.
>
> 2) I have commented out the port control code in the snmp part since our
> switches dont
> support the STACK-MIB. Would this have any effect on the delay node
> creation and inserting
> artificial latency into the links?
>
> Below is the output of the experiment log.
>
> Thanks,
> Adit
>
> ------------------
>
> There were 1 failed nodes.
>
> 1/10 pc2800's with a system osid of "FC6-STD" failed to boot: tbdelay0(pc1)
>
>
> --------- /usr/testbed/expwork/XenHack/LargeTest/swapexp.x46zYo --------
> Running 'tbswap in XenHack LargeTest'
> Beginning swap-in for XenHack/LargeTest (15). 05/29/2009 19:48:43
> TIMESTAMP: 19:48:43:952831 tbswap in started
> Checking with Admission Control ...
> Mapping to physical reality ...
> TIMESTAMP: 19:48:43:982856 assign_wrapper started
> assign_wrapper improved started
> TIMESTAMP: 19:48:44:526252 assign_wrapper started
> TIMESTAMP: 19:48:44:533137 TOP started
> Resetting DB before updating.
> opened topfile
> Minimum nodes = 10
> Maximum nodes = 10
> TIMESTAMP: 19:48:44:593332 TOP finished
> TIMESTAMP: 19:48:44:594842 assign_loop started
> Assign Run 1
> TIMESTAMP: 19:48:44:597668 ptopgen started
> ptopargs -p XenHack -e LargeTest
> TIMESTAMP: 19:48:45:585442 ptopgen finished
> TIMESTAMP: 19:48:45:590656 assign started
> assign -P XenHack-LargeTest-34323.ptop XenHack-LargeTest-34323.top
> BEST SCORE: 5.22 in 17000 iters and 0.603738 seconds
> TIMESTAMP: 19:48:47:625145 assign finished
> TIMESTAMP: 19:48:47:627527 reserving started
> TIMESTAMP: 19:48:48:685713 reserving finished
> Successfully reserved all physical nodes we needed.
> TIMESTAMP: 19:48:48:698062 assign_loop finished
> TIMESTAMP: 19:48:48:700652 LoadPhysResources started
> TIMESTAMP: 19:48:48:730596 LoadPhysResources finished
> TIMESTAMP: 19:48:48:748585 interpreting started
> TIMESTAMP: 19:48:48:754669 interpreting finished
> TIMESTAMP: 19:48:48:756651 uploading started
> TIMESTAMP: 19:48:58:39247 uploading finished
> TIMESTAMP: 19:48:58:41257 assign_wrapper finished
> TIMESTAMP: 19:48:58:462900 assign_wrapper finished
> Mapped to physical reality!
> Fetching tarballs and RPMs (if any) ...
> TIMESTAMP: 19:48:58:476288 tarfiles_setup started
> TIMESTAMP: 19:48:59:613907 tarfiles_setup finished
> Setting up mountpoints.
> TIMESTAMP: 19:48:59:620485 mountpoints started
> TIMESTAMP: 19:49:04:446116 mountpoints finished
> TIMESTAMP: 19:49:04:448911 named started
> Setting up named maps.
> TIMESTAMP: 19:49:05:421144 named finished
> TIMESTAMP: 19:49:05:427916 gentopofile started
> Generating ltmap (again) ...
> TIMESTAMP: 19:49:06:468740 gentopofile finished
> Resetting OS and rebooting.
> TIMESTAMP: 19:49:06:473804 launching os_setup
> Setting up VLANs.
> TIMESTAMP: 19:49:06:488015 snmpit started
> TIMESTAMP: 19:49:07:176843 os_setup started
> TIMESTAMP: 19:49:07:221645 rebooting/reloading nodes started
> Creating VLAN 15 as VLAN #72 on c4506-netlab ... Succeeded.
> Applying VLAN changes on c4506-netlab ... Succeeded
> Creating VLAN 15 as VLAN #72 on c4006-netlab ... Succeeded.
> Applying VLAN changes on c4006-netlab ...reboot (pc1): Attempting to reboot ...
> reboot (pc11): Attempting to reboot ...
> reboot (pc12): Attempting to reboot ...
> reboot (pc14): Attempting to reboot ...
> reboot (pc16): Attempting to reboot ...
> reboot (pc18): Attempting to reboot ...
> reboot (pc19): Attempting to reboot ...
> reboot (pc20): Attempting to reboot ...
> reboot (pc5): Attempting to reboot ...
> reboot (pc7): Attempting to reboot ...
> reboot (pc1): Successful!
> reboot (pc11): Successful!
> reboot (pc12): Successful!
> reboot (pc14): Successful!
> reboot (pc16): Successful!
> reboot (pc18): Successful!
> reboot (pc19): Successful!
> reboot (pc20): Successful!
> reboot (pc5): Successful!
> reboot (pc7): Successful!
> reboot: Done. There were 0 failures.
> reboot (pc14): child returned 0 status.
> reboot (pc5): child returned 0 status.
> reboot (pc19): child returned 0 status.
> reboot (pc16): child returned 0 status.
> reboot (pc1): child returned 0 status.
> reboot (pc20): child returned 0 status.
> reboot (pc12): child returned 0 status.
> reboot (pc7): child returned 0 status.
> reboot (pc11): child returned 0 status.
> reboot (pc18): child returned 0 status.
> TIMESTAMP: 19:49:09:263007 rebooting/reloading finished
> Waiting for local testbed nodes to finish rebooting ...
> TIMESTAMP: 19:49:09:266581 Local node waiting started
> . Succeeded
> Creating VLAN 16 as VLAN #73 on c4506-netlab ... Succeeded.
> Applying VLAN changes on c4506-netlab ... Succeeded
> TIMESTAMP: 19:49:17:120682 snmpit finished
> Setting up email lists.
> TIMESTAMP: 19:49:17:126207 genelists started
> TIMESTAMP: 19:49:19:553106 genelists finished
> Clearing port counters.
> TIMESTAMP: 19:49:19:560929 portstats started
> ep: pc16:1;pc5:1;pc1:1;pc20:1;pc11:1;pc19:1;pc12:1;pc14:1;pc7:1;pc1:3;pc18:3
> TIMESTAMP: 19:49:20:819758 portstats finished
> Still waiting for pc14 - it's been 1 minute(s).
> pc14 is alive and well
> pc5 is alive and well
> pc19 is alive and well
> pc16 is alive and well
> Still waiting for pc1 - it's been 1 minute(s).
> Still waiting for pc1 - it's been 2 minute(s).
> Still waiting for pc1 - it's been 3 minute(s).
> Still waiting for pc1 - it's been 4 minute(s).
> Still waiting for pc1 - it's been 5 minute(s).
> Still waiting for pc1 - it's been 6 minute(s).
> Still waiting for pc1 - it's been 7 minute(s).
> Still waiting for pc1 - it's been 8 minute(s).
> *** Giving up on pc1 - it's been 8 minute(s).
> *** os_setup: Rebooting pc1 and waiting again ...
> reboot (pc1): Attempting to reboot ...
> *** node_reboot-reboot_node: pc1 appears dead; will power cycle.
> pc1 now rebooting
> reboot: Done. There were 0 failures.
> pc20 is alive and well
> pc12 is alive and well
> pc7 is alive and well
> pc18 is alive and well
> pc11 is alive and well
> Still waiting for pc1 - it's been 1 minute(s).
> Still waiting for pc1 - it's been 2 minute(s).
> Still waiting for pc1 - it's been 3 minute(s).
> Still waiting for pc1 - it's been 4 minute(s).
> Still waiting for pc1 - it's been 5 minute(s).
> Still waiting for pc1 - it's been 6 minute(s).
> Still waiting for pc1 - it's been 7 minute(s).
> Still waiting for pc1 - it's been 8 minute(s).
> *** Giving up on pc1 - it's been 8 minute(s).
> *** WARNING: os_setup:
> *** pc1 may be down. This has been reported to testbed-ops.
> TIMESTAMP: 20:05:28:70663 Local node waiting finished
> OS Setup Done.
> *** ERROR: os_setup:
> *** There were 1 failed nodes.
> ***
> *** 1/10 pc2800's with a system osid of "FC6-STD" failed to boot: tbdelay0(pc1)
> TIMESTAMP: 20:05:28:82448 os_setup finished
> *** ERROR: tbswap: Failed to reset OS and reboot nodes.
> Cleaning up after errors; will try again.
> Stopping the event system
> TIMESTAMP: 20:05:30:556814 snmpit started
> Removing VLANs.
> Removing VLAN #72 on c4506-netlab ... Succeeded.
> Applying VLAN changes on c4506-netlab ... Succeeded
> Removing VLAN #73 on c4506-netlab ... Succeeded.
> Applying VLAN changes on c4506-netlab ... Succeeded
> Removing VLAN #72 on c4006-netlab ... Succeeded.
> Applying VLAN changes on c4006-netlab .... Succeeded
> TIMESTAMP: 20:05:49:503613 snmpit finished
> Freeing failed nodes.
> TIMESTAMP: 20:05:49:510515 nfree started
> Moving [Node: pc1] to [Experiment: emulab-ops/hwdown]
> TIMESTAMP: 20:05:51:246287 nfree finished
> Resetting named maps.
> TIMESTAMP: 20:05:51:255901 named started
> TIMESTAMP: 20:05:52:246354 named finished
> Resetting email lists.
> TIMESTAMP: 20:05:52:251809 genelists started
> TIMESTAMP: 20:05:53:182196 genelists finished
> Trying again...
> Mapping to physical reality ...
> TIMESTAMP: 20:05:53:196686 assign_wrapper started
> assign_wrapper improved started
> TIMESTAMP: 20:05:53:739012 assign_wrapper started
> TIMESTAMP: 20:05:53:745435 TOP started
> Reserved pnodes = 9
> Resetting DB before updating.
> opened topfile
> Minimum nodes = 10
> Maximum nodes = 10
> TIMESTAMP: 20:05:53:896226 TOP finished
> TIMESTAMP: 20:05:53:897601 assign_loop started
> Assign Run 1
> TIMESTAMP: 20:05:53:900261 ptopgen started
> ptopargs -p XenHack -e LargeTest
> TIMESTAMP: 20:05:54:892052 ptopgen finished
> TIMESTAMP: 20:05:54:895724 assign started
> assign -P XenHack-LargeTest-34705.ptop XenHack-LargeTest-34705.top
> BEST SCORE: 3.92 in 17000 iters and 1.75685 seconds
> TIMESTAMP: 20:05:58:919921 assign finished
> TIMESTAMP: 20:05:58:922135 Moving Old Reserved nodes to emulab-ops/oldreserved and back started
> [Node: pc14] already reserved in holding reservation.
> [Node: pc5] already reserved in holding reservation.
> [Node: pc19] already reserved in holding reservation.
> [Node: pc16] already reserved in holding reservation.
> [Node: pc20] already reserved in holding reservation.
> [Node: pc12] already reserved in holding reservation.
> [Node: pc7] already reserved in holding reservation.
> [Node: pc18] already reserved in holding reservation.
> [Node: pc11] already reserved in holding reservation.
> TIMESTAMP: 20:06:00:747435 Moving Old Reserved nodes to emulab-ops/oldreserved and back finished
> TIMESTAMP: 20:06:00:750892 reserving started
> TIMESTAMP: 20:06:01:629320 reserving finished
> Successfully reserved all physical nodes we needed.
> TIMESTAMP: 20:06:01:639431 assign_loop finished
> TIMESTAMP: 20:06:01:644875 LoadPhysResources started
> TIMESTAMP: 20:06:01:668451 LoadPhysResources finished
> TIMESTAMP: 20:06:01:691428 interpreting started
> TIMESTAMP: 20:06:01:699186 interpreting finished
> TIMESTAMP: 20:06:01:701142 uploading started
> TIMESTAMP: 20:06:10:928878 uploading finished
> TIMESTAMP: 20:06:10:931595 assign_wrapper finished
> TIMESTAMP: 20:06:11:352810 assign_wrapper finished
> Mapped to physical reality!
> Fetching tarballs and RPMs (if any) ...
> TIMESTAMP: 20:06:11:360773 tarfiles_setup started
> TIMESTAMP: 20:06:12:394989 tarfiles_setup finished
> Setting up mountpoints.
> TIMESTAMP: 20:06:12:406339 mountpoints started
> TIMESTAMP: 20:06:17:237299 mountpoints finished
> TIMESTAMP: 20:06:17:240119 named started
> Setting up named maps.
> TIMESTAMP: 20:06:18:215230 named finished
> Marking nodes for reboot.
> TIMESTAMP: 20:06:18:229522 gentopofile started
> Generating ltmap (again) ...
> TIMESTAMP: 20:06:19:283013 gentopofile finished
> Resetting OS and rebooting.
> TIMESTAMP: 20:06:19:291059 launching os_setup
> Setting up VLANs.
> TIMESTAMP: 20:06:19:321911 snmpit started
> TIMESTAMP: 20:06:20:1195 os_setup started
> TIMESTAMP: 20:06:20:43837 rebooting/reloading nodes started
> Creating VLAN 19 as VLAN #72 on c4506-netlab ... Succeeded.
> Applying VLAN changes on c4506-netlab ... Succeeded
> Creating VLAN 19 as VLAN #72 on c4006-netlab ... Succeeded.
> Applying VLAN changes on c4006-netlab ...reboot (pc11): Attempting to reboot ...
> reboot (pc12): Attempting to reboot ...
> reboot (pc14): Attempting to reboot ...
> reboot (pc16): Attempting to reboot ...
> reboot (pc17): Attempting to reboot ...
> reboot (pc18): Attempting to reboot ...
> reboot (pc19): Attempting to reboot ...
> reboot (pc20): Attempting to reboot ...
> reboot (pc5): Attempting to reboot ...
> reboot (pc7): Attempting to reboot ...
> Succeeded
> Creating VLAN 18 as VLAN #73 on c4506-netlab ... Succeeded.
> Applying VLAN changes on c4506-netlab ... Succeeded
> TIMESTAMP: 20:06:29:936388 snmpit finished
> Setting up email lists.
> TIMESTAMP: 20:06:29:944681 genelists started
> TIMESTAMP: 20:06:30:893555 genelists finished
> Clearing port counters.
> TIMESTAMP: 20:06:30:899003 portstats started
> ep: pc17:1;pc18:1;pc5:0;pc7:0;pc16:3;pc19:3;pc20:3;pc12:3;pc14:3;pc11:3;pc17:3
> TIMESTAMP: 20:06:32:392983 portstats finished
> reboot (pc11): Successful!
> reboot (pc12): Successful!
> reboot (pc14): Successful!
> reboot (pc16): Successful!
> reboot (pc17): Successful!
> reboot (pc18): Successful!
> reboot (pc19): Successful!
> reboot (pc20): Successful!
> reboot (pc5): Successful!
> reboot (pc7): Successful!
> reboot: Done. There were 0 failures.
> reboot (pc14): child returned 0 status.
> reboot (pc5): child returned 0 status.
> reboot (pc19): child returned 0 status.
> reboot (pc16): child returned 0 status.
> reboot (pc17): child returned 0 status.
> reboot (pc20): child returned 0 status.
> reboot (pc12): child returned 0 status.
> reboot (pc7): child returned 0 status.
> reboot (pc11): child returned 0 status.
> reboot (pc18): child returned 0 status.
> TIMESTAMP: 20:06:39:924405 rebooting/reloading finished
> Waiting for local testbed nodes to finish rebooting ...
> TIMESTAMP: 20:06:39:931535 Local node waiting started
> Still waiting for pc14 - it's been 1 minute(s).
> Still waiting for pc14 - it's been 2 minute(s).
> pc14 is alive and well
> pc5 is alive and well
> pc19 is alive and well
> pc16 is alive and well
> Still waiting for pc17 - it's been 2 minute(s).
> Still waiting for pc17 - it's been 3 minute(s).
> Still waiting for pc17 - it's been 4 minute(s).
> Still waiting for pc17 - it's been 5 minute(s).
> Still waiting for pc17 - it's been 6 minute(s).
> Still waiting for pc17 - it's been 7 minute(s).
> Still waiting for pc17 - it's been 8 minute(s).
> *** Giving up on pc17 - it's been 8 minute(s).
> *** os_setup: Rebooting pc17 and waiting again ...
> reboot (pc17): Attempting to reboot ...
> *** node_reboot-reboot_node: pc17 appears dead; will power cycle.
> pc17 now rebooting
> reboot: Done. There were 0 failures.
> pc20 is alive and well
> pc12 is alive and well
> pc7 is alive and well
> pc18 is alive and well
> pc11 is alive and well
> Still waiting for pc17 - it's been 1 minute(s).
> Still waiting for pc17 - it's been 2 minute(s).
> Still waiting for pc17 - it's been 3 minute(s).
> Still waiting for pc17 - it's been 4 minute(s).
> Still waiting for pc17 - it's been 5 minute(s).
> Still waiting for pc17 - it's been 6 minute(s).
> Still waiting for pc17 - it's been 7 minute(s).
> Still waiting for pc17 - it's been 8 minute(s).
> *** Giving up on pc17 - it's been 8 minute(s).
> *** WARNING: os_setup:
> *** pc17 may be down. This has been reported to testbed-ops.
> TIMESTAMP: 20:22:58:26970 Local node waiting finished
> OS Setup Done.
> *** ERROR: os_setup:
> *** There were 1 failed nodes.
> ***
> *** 1/10 pc2800's with a system osid of "FC6-STD" failed to boot: tbdelay0(pc17)
> TIMESTAMP: 20:22:58:38624 os_setup finished
> *** ERROR: tbswap: Failed to reset OS and reboot nodes.
> Cleaning up after errors; will try again.
> Stopping the event system
> TIMESTAMP: 20:23:00:507184 snmpit started
> Removing VLANs.
> Removing VLAN #72 on c4506-netlab ... Succeeded.
> Applying VLAN changes on c4506-netlab ... Succeeded
> Removing VLAN #73 on c4506-netlab ... Succeeded.
> Applying VLAN changes on c4506-netlab ... Succeeded
> Removing VLAN #72 on c4006-netlab ... Succeeded.
> Applying VLAN changes on c4006-netlab ... Succeeded
> TIMESTAMP: 20:23:13:342456 snmpit finished
> Freeing failed nodes.
> TIMESTAMP: 20:23:13:353466 nfree started
> Moving [Node: pc17] to [Experiment: emulab-ops/hwdown]
> TIMESTAMP: 20:23:15:111176 nfree finished
> Resetting named maps.
> TIMESTAMP: 20:23:15:120689 named started
> TIMESTAMP: 20:23:16:112461 named finished
> Resetting email lists.
> TIMESTAMP: 20:23:16:117908 genelists started
> TIMESTAMP: 20:23:17:154099 genelists finished
> Trying again...
> Mapping to physical reality ...
> TIMESTAMP: 20:23:17:168464 assign_wrapper started
> assign_wrapper improved started
> TIMESTAMP: 20:23:17:722132 assign_wrapper started
> TIMESTAMP: 20:23:17:728781 TOP started
> Reserved pnodes = 9
> Resetting DB before updating.
> opened topfile
> Minimum nodes = 10
> Maximum nodes = 10
> TIMESTAMP: 20:23:17:878296 TOP finished
> TIMESTAMP: 20:23:17:880401 assign_loop started
> Assign Run 1
> TIMESTAMP: 20:23:17:883767 ptopgen started
> ptopargs -p XenHack -e LargeTest
> TIMESTAMP: 20:23:18:878090 ptopgen finished
> TIMESTAMP: 20:23:18:881813 assign started
> assign -P XenHack-LargeTest-35232.ptop XenHack-LargeTest-35232.top
> BEST SCORE: 3.92 in 17000 iters and 1.76006 seconds
> TIMESTAMP: 20:23:22:905615 assign finished
> TIMESTAMP: 20:23:22:907938 Moving Old Reserved nodes to emulab-ops/oldreserved and back started
> [Node: pc14] already reserved in holding reservation.
> [Node: pc5] already reserved in holding reservation.
> [Node: pc19] already reserved in holding reservation.
> [Node: pc16] already reserved in holding reservation.
> [Node: pc20] already reserved in holding reservation.
> [Node: pc12] already reserved in holding reservation.
> [Node: pc7] already reserved in holding reservation.
> [Node: pc18] already reserved in holding reservation.
> [Node: pc11] already reserved in holding reservation.
> TIMESTAMP: 20:23:24:721234 Moving Old Reserved nodes to emulab-ops/oldreserved and back finished
> TIMESTAMP: 20:23:24:726087 reserving started
> TIMESTAMP: 20:23:25:614820 reserving finished
> Successfully reserved all physical nodes we needed.
> TIMESTAMP: 20:23:25:621611 assign_loop finished
> TIMESTAMP: 20:23:25:624423 LoadPhysResources started
> TIMESTAMP: 20:23:25:642900 LoadPhysResources finished
> TIMESTAMP: 20:23:25:662589 interpreting started
> TIMESTAMP: 20:23:25:669765 interpreting finished
> TIMESTAMP: 20:23:25:671814 uploading started
> TIMESTAMP: 20:23:34:850240 uploading finished
> TIMESTAMP: 20:23:34:852882 assign_wrapper finished
> TIMESTAMP: 20:23:35:273310 assign_wrapper finished
> Mapped to physical reality!
> Fetching tarballs and RPMs (if any) ...
> TIMESTAMP: 20:23:35:280862 tarfiles_setup started
> TIMESTAMP: 20:23:36:312095 tarfiles_setup finished
> Setting up mountpoints.
> TIMESTAMP: 20:23:36:319109 mountpoints started
> TIMESTAMP: 20:23:41:129350 mountpoints finished
> TIMESTAMP: 20:23:41:132118 named started
> Setting up named maps.
> TIMESTAMP: 20:23:42:106374 named finished
> Marking nodes for reboot.
> TIMESTAMP: 20:23:42:120533 gentopofile started
> Generating ltmap (again) ...
> TIMESTAMP: 20:23:43:245637 gentopofile finished
> Resetting OS and rebooting.
> TIMESTAMP: 20:23:43:253754 launching os_setup
> Setting up VLANs.
> TIMESTAMP: 20:23:43:282931 snmpit started
> TIMESTAMP: 20:23:43:949340 os_setup started
> TIMESTAMP: 20:23:43:993256 rebooting/reloading nodes started
> Creating VLAN 22 as VLAN #72 on c4506-netlab ... Succeeded.
> Applying VLAN changes on c4506-netlab ... Succeeded
> Creating VLAN 22 as VLAN #72 on c4006-netlab ... Succeeded.
> Applying VLAN changes on c4006-netlab ...reboot (pc11): Attempting to reboot ...
> reboot (pc12): Attempting to reboot ...
> reboot (pc14): Attempting to reboot ...
> reboot (pc16): Attempting to reboot ...
> reboot (pc18): Attempting to reboot ...
> reboot (pc19): Attempting to reboot ...
> reboot (pc20): Attempting to reboot ...
> reboot (pc5): Attempting to reboot ...
> reboot (pc6): Attempting to reboot ...
> reboot (pc7): Attempting to reboot ...
> . Succeeded
> Creating VLAN 21 as VLAN #73 on c4506-netlab ... Succeeded.
> Applying VLAN changes on c4506-netlab ... Succeeded
> TIMESTAMP: 20:23:57:692697 snmpit finished
> Setting up email lists.
> TIMESTAMP: 20:23:57:700871 genelists started
> TIMESTAMP: 20:23:58:649645 genelists finished
> Clearing port counters.
> TIMESTAMP: 20:23:58:655092 portstats started
> ep: pc6:1;pc18:1;pc5:0;pc7:0;pc16:3;pc19:3;pc20:3;pc12:3;pc14:3;pc11:3;pc6:0
> TIMESTAMP: 20:24:00:343293 portstats finished
> reboot (pc11): Successful!
> reboot (pc12): Successful!
> reboot (pc14): Successful!
> reboot (pc16): Successful!
> reboot (pc18): Successful!
> reboot (pc19): Successful!
> reboot (pc20): Successful!
> reboot (pc5): Successful!
> reboot (pc6): Successful!
> reboot (pc7): Successful!
> reboot: Done. There were 0 failures.
> reboot (pc14): child returned 0 status.
> reboot (pc5): child returned 0 status.
> reboot (pc19): child returned 0 status.
> reboot (pc16): child returned 0 status.
> reboot (pc6): child returned 0 status.
> reboot (pc20): child returned 0 status.
> reboot (pc12): child returned 0 status.
> reboot (pc7): child returned 0 status.
> reboot (pc11): child returned 0 status.
> reboot (pc18): child returned 0 status.
> TIMESTAMP: 20:24:03:971058 rebooting/reloading finished
> Waiting for local testbed nodes to finish rebooting ...
> TIMESTAMP: 20:24:03:978306 Local node waiting started
> Still waiting for pc14 - it's been 1 minute(s).
> Still waiting for pc14 - it's been 2 minute(s).
> pc14 is alive and well
> pc5 is alive and well
> pc19 is alive and well
> Still waiting for pc6 - it's been 2 minute(s).
> Still waiting for pc6 - it's been 3 minute(s).
> Still waiting for pc6 - it's been 4 minute(s).
> Still waiting for pc6 - it's been 5 minute(s).
> Still waiting for pc6 - it's been 6 minute(s).
> Still waiting for pc6 - it's been 7 minute(s).
> Still waiting for pc6 - it's been 8 minute(s).
> *** Giving up on pc6 - it's been 8 minute(s).
> *** os_setup: Rebooting pc6 and waiting again ...
> reboot (pc6): Attempting to reboot ...
> *** node_reboot-reboot_node: pc6 appears dead; will power cycle.
> pc6 now rebooting
> reboot: Done. There were 0 failures.
> pc16 is alive and well
> pc20 is alive and well
> pc12 is alive and well
> pc7 is alive and well
> pc18 is alive and well
> pc11 is alive and well
> Still waiting for pc6 - it's been 1 minute(s).
> Still waiting for pc6 - it's been 2 minute(s).
> Still waiting for pc6 - it's been 3 minute(s).
> Still waiting for pc6 - it's been 4 minute(s).
> Still waiting for pc6 - it's been 5 minute(s).
> Still waiting for pc6 - it's been 6 minute(s).
> Still waiting for pc6 - it's been 7 minute(s).
> Still waiting for pc6 - it's been 8 minute(s).
> *** Giving up on pc6 - it's been 8 minute(s).
> *** WARNING: os_setup:
> *** pc6 may be down. This has been reported to testbed-ops.
> TIMESTAMP: 20:40:21:788743 Local node waiting finished
> OS Setup Done.
> *** ERROR: os_setup:
> *** There were 1 failed nodes.
> ***
> *** 1/10 pc2800's with a system osid of "FC6-STD" failed to boot: tbdelay0(pc6)
> TIMESTAMP: 20:40:21:800884 os_setup finished
> *** ERROR: tbswap: Failed to reset OS and reboot nodes.
> Cleaning up after errors.
> Stopping the event system
> TIMESTAMP: 20:40:24:303659 snmpit started
> Removing VLANs.
> Removing VLAN #72 on c4506-netlab ... Succeeded.
> Applying VLAN changes on c4506-netlab ... Succeeded
> Removing VLAN #73 on c4506-netlab ... Succeeded.
> Applying VLAN changes on c4506-netlab ... Succeeded
> Removing VLAN #72 on c4006-netlab ... Succeeded.
> Applying VLAN changes on c4006-netlab .... Succeeded
> TIMESTAMP: 20:40:39:819179 snmpit finished
> Tearing down virtual nodes.
> TIMESTAMP: 20:40:39:827359 vnode_setup -k started
> vnode_setup running at parallelization: 10 wait_time: 120
> Vnode teardown finished.
> TIMESTAMP: 20:40:41:739472 vnode_setup finished
> Freeing nodes.
> TIMESTAMP: 20:40:41:744844 nfree started
> Releasing all nodes from experiment [Experiment: XenHack/LargeTest].
> Moving [Node: pc18] to [Experiment: emulab-ops/reloadpending]
> Moving [Node: pc12] to [Experiment: emulab-ops/reloadpending]
> Moving [Node: pc11] to [Experiment: emulab-ops/reloadpending]
> Moving [Node: pc16] to [Experiment: emulab-ops/reloadpending]
> Moving [Node: pc20] to [Experiment: emulab-ops/reloadpending]
> Moving [Node: pc14] to [Experiment: emulab-ops/reloadpending]
> Moving [Node: pc19] to [Experiment: emulab-ops/reloadpending]
> Moving [Node: pc7] to [Experiment: emulab-ops/reloadpending]
> Moving [Node: pc5] to [Experiment: emulab-ops/reloadpending]
> Moving [Node: pc6] to [Experiment: emulab-ops/hwdown]
> TIMESTAMP: 20:40:43:632295 nfree finished
> Resetting named maps.
> TIMESTAMP: 20:40:43:638965 named started
> TIMESTAMP: 20:40:44:621253 named finished
> Resetting email lists.
> TIMESTAMP: 20:40:44:629371 genelists started
> TIMESTAMP: 20:40:45:577918 genelists finished
> Resetting DB.
> Failingly finished swap-in for XenHack/LargeTest. 20:40:45:660855
> TIMESTAMP: 20:40:45:662472 tbswap in finished (failed)
> *** ERROR: swapexp: tbswap in failed!
> Cleaning up and exiting with status 1 ...
> **** Experimental information, please ignore ****
> Session ID = 11640
> Likely Cause of the Problem:
> There were 1 failed nodes.
>
> 1/10 pc2800's with a system osid of "FC6-STD" failed to boot: tbdelay0(pc1)
> Cause: unknown
> Confidence: 0.7
> Script: os_setup
> **** End experimental information ****
> _______________________________________________
> Testbed-admins mailing list
> Testbed-admins@flux.utah.edu
> http://www.flux.utah.edu/mailman/listinfo/testbed-admins