|
Hello, Im trying to set a latency on a link in the experiment. According to the software it does this by inserting a delay node. Unfortunately, the delay node does not seem to boot up. There is a failure in trying to boot the OS on the delay node. I have 2 questions: 1) What should be the value of the delay_osid field for the node type? I have set it to FC6-STD. Its not mentioned in the wiki installdocs. 2) I have commented out the port control code in the snmp part since our switches dont support the STACK-MIB. Would this have any effect on the delay node creation and inserting artificial latency into the links? Below is the output of the experiment log. Thanks, Adit ------------------ There were 1 failed nodes.
1/10 pc2800's with a system osid of "FC6-STD" failed to boot: tbdelay0(pc1)
--------- /usr/testbed/expwork/XenHack/LargeTest/swapexp.x46zYo --------
Running 'tbswap in XenHack LargeTest'
Beginning swap-in for XenHack/LargeTest (15). 05/29/2009 19:48:43
TIMESTAMP: 19:48:43:952831 tbswap in started
Checking with Admission Control ...
Mapping to physical reality ...
TIMESTAMP: 19:48:43:982856 assign_wrapper started
assign_wrapper improved started
TIMESTAMP: 19:48:44:526252 assign_wrapper started
TIMESTAMP: 19:48:44:533137 TOP started
Resetting DB before updating.
opened topfile
Minimum nodes = 10
Maximum nodes = 10
TIMESTAMP: 19:48:44:593332 TOP finished
TIMESTAMP: 19:48:44:594842 assign_loop started
Assign Run 1
TIMESTAMP: 19:48:44:597668 ptopgen started
ptopargs -p XenHack -e LargeTest
TIMESTAMP: 19:48:45:585442 ptopgen finished
TIMESTAMP: 19:48:45:590656 assign started
assign -P XenHack-LargeTest-34323.ptop XenHack-LargeTest-34323.top
BEST SCORE: 5.22 in 17000 iters and 0.603738 seconds
TIMESTAMP: 19:48:47:625145 assign finished
TIMESTAMP: 19:48:47:627527 reserving started
TIMESTAMP: 19:48:48:685713 reserving finished
Successfully reserved all physical nodes we needed.
TIMESTAMP: 19:48:48:698062 assign_loop finished
TIMESTAMP: 19:48:48:700652 LoadPhysResources started
TIMESTAMP: 19:48:48:730596 LoadPhysResources finished
TIMESTAMP: 19:48:48:748585 interpreting started
TIMESTAMP: 19:48:48:754669 interpreting finished
TIMESTAMP: 19:48:48:756651 uploading started
TIMESTAMP: 19:48:58:39247 uploading finished
TIMESTAMP: 19:48:58:41257 assign_wrapper finished
TIMESTAMP: 19:48:58:462900 assign_wrapper finished
Mapped to physical reality!
Fetching tarballs and RPMs (if any) ...
TIMESTAMP: 19:48:58:476288 tarfiles_setup started
TIMESTAMP: 19:48:59:613907 tarfiles_setup finished
Setting up mountpoints.
TIMESTAMP: 19:48:59:620485 mountpoints started
TIMESTAMP: 19:49:04:446116 mountpoints finished
TIMESTAMP: 19:49:04:448911 named started
Setting up named maps.
TIMESTAMP: 19:49:05:421144 named finished
TIMESTAMP: 19:49:05:427916 gentopofile started
Generating ltmap (again) ...
TIMESTAMP: 19:49:06:468740 gentopofile finished
Resetting OS and rebooting.
TIMESTAMP: 19:49:06:473804 launching os_setup
Setting up VLANs.
TIMESTAMP: 19:49:06:488015 snmpit started
TIMESTAMP: 19:49:07:176843 os_setup started
TIMESTAMP: 19:49:07:221645 rebooting/reloading nodes started
Creating VLAN 15 as VLAN #72 on c4506-netlab ... Succeeded.
Applying VLAN changes on c4506-netlab ... Succeeded
Creating VLAN 15 as VLAN #72 on c4006-netlab ... Succeeded.
Applying VLAN changes on c4006-netlab ...reboot (pc1): Attempting to reboot ...
reboot (pc11): Attempting to reboot ...
reboot (pc12): Attempting to reboot ...
reboot (pc14): Attempting to reboot ...
reboot (pc16): Attempting to reboot ...
reboot (pc18): Attempting to reboot ...
reboot (pc19): Attempting to reboot ...
reboot (pc20): Attempting to reboot ...
reboot (pc5): Attempting to reboot ...
reboot (pc7): Attempting to reboot ...
reboot (pc1): Successful!
reboot (pc11): Successful!
reboot (pc12): Successful!
reboot (pc14): Successful!
reboot (pc16): Successful!
reboot (pc18): Successful!
reboot (pc19): Successful!
reboot (pc20): Successful!
reboot (pc5): Successful!
reboot (pc7): Successful!
reboot: Done. There were 0 failures.
reboot (pc14): child returned 0 status.
reboot (pc5): child returned 0 status.
reboot (pc19): child returned 0 status.
reboot (pc16): child returned 0 status.
reboot (pc1): child returned 0 status.
reboot (pc20): child returned 0 status.
reboot (pc12): child returned 0 status.
reboot (pc7): child returned 0 status.
reboot (pc11): child returned 0 status.
reboot (pc18): child returned 0 status.
TIMESTAMP: 19:49:09:263007 rebooting/reloading finished
Waiting for local testbed nodes to finish rebooting ...
TIMESTAMP: 19:49:09:266581 Local node waiting started
. Succeeded
Creating VLAN 16 as VLAN #73 on c4506-netlab ... Succeeded.
Applying VLAN changes on c4506-netlab ... Succeeded
TIMESTAMP: 19:49:17:120682 snmpit finished
Setting up email lists.
TIMESTAMP: 19:49:17:126207 genelists started
TIMESTAMP: 19:49:19:553106 genelists finished
Clearing port counters.
TIMESTAMP: 19:49:19:560929 portstats started
ep: pc16:1;pc5:1;pc1:1;pc20:1;pc11:1;pc19:1;pc12:1;pc14:1;pc7:1;pc1:3;pc18:3
TIMESTAMP: 19:49:20:819758 portstats finished
Still waiting for pc14 - it's been 1 minute(s).
pc14 is alive and well
pc5 is alive and well
pc19 is alive and well
pc16 is alive and well
Still waiting for pc1 - it's been 1 minute(s).
Still waiting for pc1 - it's been 2 minute(s).
Still waiting for pc1 - it's been 3 minute(s).
Still waiting for pc1 - it's been 4 minute(s).
Still waiting for pc1 - it's been 5 minute(s).
Still waiting for pc1 - it's been 6 minute(s).
Still waiting for pc1 - it's been 7 minute(s).
Still waiting for pc1 - it's been 8 minute(s).
*** Giving up on pc1 - it's been 8 minute(s).
*** os_setup: Rebooting pc1 and waiting again ...
reboot (pc1): Attempting to reboot ...
*** node_reboot-reboot_node: pc1 appears dead; will power cycle.
pc1 now rebooting
reboot: Done. There were 0 failures.
pc20 is alive and well
pc12 is alive and well
pc7 is alive and well
pc18 is alive and well
pc11 is alive and well
Still waiting for pc1 - it's been 1 minute(s).
Still waiting for pc1 - it's been 2 minute(s).
Still waiting for pc1 - it's been 3 minute(s).
Still waiting for pc1 - it's been 4 minute(s).
Still waiting for pc1 - it's been 5 minute(s).
Still waiting for pc1 - it's been 6 minute(s).
Still waiting for pc1 - it's been 7 minute(s).
Still waiting for pc1 - it's been 8 minute(s).
*** Giving up on pc1 - it's been 8 minute(s).
*** WARNING: os_setup:
*** pc1 may be down. This has been reported to testbed-ops.
TIMESTAMP: 20:05:28:70663 Local node waiting finished
OS Setup Done.
*** ERROR: os_setup:
*** There were 1 failed nodes.
***
*** 1/10 pc2800's with a system osid of "FC6-STD" failed to boot: tbdelay0(pc1)
TIMESTAMP: 20:05:28:82448 os_setup finished
*** ERROR: tbswap: Failed to reset OS and reboot nodes.
Cleaning up after errors; will try again.
Stopping the event system
TIMESTAMP: 20:05:30:556814 snmpit started
Removing VLANs.
Removing VLAN #72 on c4506-netlab ... Succeeded.
Applying VLAN changes on c4506-netlab ... Succeeded
Removing VLAN #73 on c4506-netlab ... Succeeded.
Applying VLAN changes on c4506-netlab ... Succeeded
Removing VLAN #72 on c4006-netlab ... Succeeded.
Applying VLAN changes on c4006-netlab .... Succeeded
TIMESTAMP: 20:05:49:503613 snmpit finished
Freeing failed nodes.
TIMESTAMP: 20:05:49:510515 nfree started
Moving [Node: pc1] to [Experiment: emulab-ops/hwdown]
TIMESTAMP: 20:05:51:246287 nfree finished
Resetting named maps.
TIMESTAMP: 20:05:51:255901 named started
TIMESTAMP: 20:05:52:246354 named finished
Resetting email lists.
TIMESTAMP: 20:05:52:251809 genelists started
TIMESTAMP: 20:05:53:182196 genelists finished
Trying again...
Mapping to physical reality ...
TIMESTAMP: 20:05:53:196686 assign_wrapper started
assign_wrapper improved started
TIMESTAMP: 20:05:53:739012 assign_wrapper started
TIMESTAMP: 20:05:53:745435 TOP started
Reserved pnodes = 9
Resetting DB before updating.
opened topfile
Minimum nodes = 10
Maximum nodes = 10
TIMESTAMP: 20:05:53:896226 TOP finished
TIMESTAMP: 20:05:53:897601 assign_loop started
Assign Run 1
TIMESTAMP: 20:05:53:900261 ptopgen started
ptopargs -p XenHack -e LargeTest
TIMESTAMP: 20:05:54:892052 ptopgen finished
TIMESTAMP: 20:05:54:895724 assign started
assign -P XenHack-LargeTest-34705.ptop XenHack-LargeTest-34705.top
BEST SCORE: 3.92 in 17000 iters and 1.75685 seconds
TIMESTAMP: 20:05:58:919921 assign finished
TIMESTAMP: 20:05:58:922135 Moving Old Reserved nodes to emulab-ops/oldreserved and back started
[Node: pc14] already reserved in holding reservation.
[Node: pc5] already reserved in holding reservation.
[Node: pc19] already reserved in holding reservation.
[Node: pc16] already reserved in holding reservation.
[Node: pc20] already reserved in holding reservation.
[Node: pc12] already reserved in holding reservation.
[Node: pc7] already reserved in holding reservation.
[Node: pc18] already reserved in holding reservation.
[Node: pc11] already reserved in holding reservation.
TIMESTAMP: 20:06:00:747435 Moving Old Reserved nodes to emulab-ops/oldreserved and back finished
TIMESTAMP: 20:06:00:750892 reserving started
TIMESTAMP: 20:06:01:629320 reserving finished
Successfully reserved all physical nodes we needed.
TIMESTAMP: 20:06:01:639431 assign_loop finished
TIMESTAMP: 20:06:01:644875 LoadPhysResources started
TIMESTAMP: 20:06:01:668451 LoadPhysResources finished
TIMESTAMP: 20:06:01:691428 interpreting started
TIMESTAMP: 20:06:01:699186 interpreting finished
TIMESTAMP: 20:06:01:701142 uploading started
TIMESTAMP: 20:06:10:928878 uploading finished
TIMESTAMP: 20:06:10:931595 assign_wrapper finished
TIMESTAMP: 20:06:11:352810 assign_wrapper finished
Mapped to physical reality!
Fetching tarballs and RPMs (if any) ...
TIMESTAMP: 20:06:11:360773 tarfiles_setup started
TIMESTAMP: 20:06:12:394989 tarfiles_setup finished
Setting up mountpoints.
TIMESTAMP: 20:06:12:406339 mountpoints started
TIMESTAMP: 20:06:17:237299 mountpoints finished
TIMESTAMP: 20:06:17:240119 named started
Setting up named maps.
TIMESTAMP: 20:06:18:215230 named finished
Marking nodes for reboot.
TIMESTAMP: 20:06:18:229522 gentopofile started
Generating ltmap (again) ...
TIMESTAMP: 20:06:19:283013 gentopofile finished
Resetting OS and rebooting.
TIMESTAMP: 20:06:19:291059 launching os_setup
Setting up VLANs.
TIMESTAMP: 20:06:19:321911 snmpit started
TIMESTAMP: 20:06:20:1195 os_setup started
TIMESTAMP: 20:06:20:43837 rebooting/reloading nodes started
Creating VLAN 19 as VLAN #72 on c4506-netlab ... Succeeded.
Applying VLAN changes on c4506-netlab ... Succeeded
Creating VLAN 19 as VLAN #72 on c4006-netlab ... Succeeded.
Applying VLAN changes on c4006-netlab ...reboot (pc11): Attempting to reboot ...
reboot (pc12): Attempting to reboot ...
reboot (pc14): Attempting to reboot ...
reboot (pc16): Attempting to reboot ...
reboot (pc17): Attempting to reboot ...
reboot (pc18): Attempting to reboot ...
reboot (pc19): Attempting to reboot ...
reboot (pc20): Attempting to reboot ...
reboot (pc5): Attempting to reboot ...
reboot (pc7): Attempting to reboot ...
Succeeded
Creating VLAN 18 as VLAN #73 on c4506-netlab ... Succeeded.
Applying VLAN changes on c4506-netlab ... Succeeded
TIMESTAMP: 20:06:29:936388 snmpit finished
Setting up email lists.
TIMESTAMP: 20:06:29:944681 genelists started
TIMESTAMP: 20:06:30:893555 genelists finished
Clearing port counters.
TIMESTAMP: 20:06:30:899003 portstats started
ep: pc17:1;pc18:1;pc5:0;pc7:0;pc16:3;pc19:3;pc20:3;pc12:3;pc14:3;pc11:3;pc17:3
TIMESTAMP: 20:06:32:392983 portstats finished
reboot (pc11): Successful!
reboot (pc12): Successful!
reboot (pc14): Successful!
reboot (pc16): Successful!
reboot (pc17): Successful!
reboot (pc18): Successful!
reboot (pc19): Successful!
reboot (pc20): Successful!
reboot (pc5): Successful!
reboot (pc7): Successful!
reboot: Done. There were 0 failures.
reboot (pc14): child returned 0 status.
reboot (pc5): child returned 0 status.
reboot (pc19): child returned 0 status.
reboot (pc16): child returned 0 status.
reboot (pc17): child returned 0 status.
reboot (pc20): child returned 0 status.
reboot (pc12): child returned 0 status.
reboot (pc7): child returned 0 status.
reboot (pc11): child returned 0 status.
reboot (pc18): child returned 0 status.
TIMESTAMP: 20:06:39:924405 rebooting/reloading finished
Waiting for local testbed nodes to finish rebooting ...
TIMESTAMP: 20:06:39:931535 Local node waiting started
Still waiting for pc14 - it's been 1 minute(s).
Still waiting for pc14 - it's been 2 minute(s).
pc14 is alive and well
pc5 is alive and well
pc19 is alive and well
pc16 is alive and well
Still waiting for pc17 - it's been 2 minute(s).
Still waiting for pc17 - it's been 3 minute(s).
Still waiting for pc17 - it's been 4 minute(s).
Still waiting for pc17 - it's been 5 minute(s).
Still waiting for pc17 - it's been 6 minute(s).
Still waiting for pc17 - it's been 7 minute(s).
Still waiting for pc17 - it's been 8 minute(s).
*** Giving up on pc17 - it's been 8 minute(s).
*** os_setup: Rebooting pc17 and waiting again ...
reboot (pc17): Attempting to reboot ...
*** node_reboot-reboot_node: pc17 appears dead; will power cycle.
pc17 now rebooting
reboot: Done. There were 0 failures.
pc20 is alive and well
pc12 is alive and well
pc7 is alive and well
pc18 is alive and well
pc11 is alive and well
Still waiting for pc17 - it's been 1 minute(s).
Still waiting for pc17 - it's been 2 minute(s).
Still waiting for pc17 - it's been 3 minute(s).
Still waiting for pc17 - it's been 4 minute(s).
Still waiting for pc17 - it's been 5 minute(s).
Still waiting for pc17 - it's been 6 minute(s).
Still waiting for pc17 - it's been 7 minute(s).
Still waiting for pc17 - it's been 8 minute(s).
*** Giving up on pc17 - it's been 8 minute(s).
*** WARNING: os_setup:
*** pc17 may be down. This has been reported to testbed-ops.
TIMESTAMP: 20:22:58:26970 Local node waiting finished
OS Setup Done.
*** ERROR: os_setup:
*** There were 1 failed nodes.
***
*** 1/10 pc2800's with a system osid of "FC6-STD" failed to boot: tbdelay0(pc17)
TIMESTAMP: 20:22:58:38624 os_setup finished
*** ERROR: tbswap: Failed to reset OS and reboot nodes.
Cleaning up after errors; will try again.
Stopping the event system
TIMESTAMP: 20:23:00:507184 snmpit started
Removing VLANs.
Removing VLAN #72 on c4506-netlab ... Succeeded.
Applying VLAN changes on c4506-netlab ... Succeeded
Removing VLAN #73 on c4506-netlab ... Succeeded.
Applying VLAN changes on c4506-netlab ... Succeeded
Removing VLAN #72 on c4006-netlab ... Succeeded.
Applying VLAN changes on c4006-netlab ... Succeeded
TIMESTAMP: 20:23:13:342456 snmpit finished
Freeing failed nodes.
TIMESTAMP: 20:23:13:353466 nfree started
Moving [Node: pc17] to [Experiment: emulab-ops/hwdown]
TIMESTAMP: 20:23:15:111176 nfree finished
Resetting named maps.
TIMESTAMP: 20:23:15:120689 named started
TIMESTAMP: 20:23:16:112461 named finished
Resetting email lists.
TIMESTAMP: 20:23:16:117908 genelists started
TIMESTAMP: 20:23:17:154099 genelists finished
Trying again...
Mapping to physical reality ...
TIMESTAMP: 20:23:17:168464 assign_wrapper started
assign_wrapper improved started
TIMESTAMP: 20:23:17:722132 assign_wrapper started
TIMESTAMP: 20:23:17:728781 TOP started
Reserved pnodes = 9
Resetting DB before updating.
opened topfile
Minimum nodes = 10
Maximum nodes = 10
TIMESTAMP: 20:23:17:878296 TOP finished
TIMESTAMP: 20:23:17:880401 assign_loop started
Assign Run 1
TIMESTAMP: 20:23:17:883767 ptopgen started
ptopargs -p XenHack -e LargeTest
TIMESTAMP: 20:23:18:878090 ptopgen finished
TIMESTAMP: 20:23:18:881813 assign started
assign -P XenHack-LargeTest-35232.ptop XenHack-LargeTest-35232.top
BEST SCORE: 3.92 in 17000 iters and 1.76006 seconds
TIMESTAMP: 20:23:22:905615 assign finished
TIMESTAMP: 20:23:22:907938 Moving Old Reserved nodes to emulab-ops/oldreserved and back started
[Node: pc14] already reserved in holding reservation.
[Node: pc5] already reserved in holding reservation.
[Node: pc19] already reserved in holding reservation.
[Node: pc16] already reserved in holding reservation.
[Node: pc20] already reserved in holding reservation.
[Node: pc12] already reserved in holding reservation.
[Node: pc7] already reserved in holding reservation.
[Node: pc18] already reserved in holding reservation.
[Node: pc11] already reserved in holding reservation.
TIMESTAMP: 20:23:24:721234 Moving Old Reserved nodes to emulab-ops/oldreserved and back finished
TIMESTAMP: 20:23:24:726087 reserving started
TIMESTAMP: 20:23:25:614820 reserving finished
Successfully reserved all physical nodes we needed.
TIMESTAMP: 20:23:25:621611 assign_loop finished
TIMESTAMP: 20:23:25:624423 LoadPhysResources started
TIMESTAMP: 20:23:25:642900 LoadPhysResources finished
TIMESTAMP: 20:23:25:662589 interpreting started
TIMESTAMP: 20:23:25:669765 interpreting finished
TIMESTAMP: 20:23:25:671814 uploading started
TIMESTAMP: 20:23:34:850240 uploading finished
TIMESTAMP: 20:23:34:852882 assign_wrapper finished
TIMESTAMP: 20:23:35:273310 assign_wrapper finished
Mapped to physical reality!
Fetching tarballs and RPMs (if any) ...
TIMESTAMP: 20:23:35:280862 tarfiles_setup started
TIMESTAMP: 20:23:36:312095 tarfiles_setup finished
Setting up mountpoints.
TIMESTAMP: 20:23:36:319109 mountpoints started
TIMESTAMP: 20:23:41:129350 mountpoints finished
TIMESTAMP: 20:23:41:132118 named started
Setting up named maps.
TIMESTAMP: 20:23:42:106374 named finished
Marking nodes for reboot.
TIMESTAMP: 20:23:42:120533 gentopofile started
Generating ltmap (again) ...
TIMESTAMP: 20:23:43:245637 gentopofile finished
Resetting OS and rebooting.
TIMESTAMP: 20:23:43:253754 launching os_setup
Setting up VLANs.
TIMESTAMP: 20:23:43:282931 snmpit started
TIMESTAMP: 20:23:43:949340 os_setup started
TIMESTAMP: 20:23:43:993256 rebooting/reloading nodes started
Creating VLAN 22 as VLAN #72 on c4506-netlab ... Succeeded.
Applying VLAN changes on c4506-netlab ... Succeeded
Creating VLAN 22 as VLAN #72 on c4006-netlab ... Succeeded.
Applying VLAN changes on c4006-netlab ...reboot (pc11): Attempting to reboot ...
reboot (pc12): Attempting to reboot ...
reboot (pc14): Attempting to reboot ...
reboot (pc16): Attempting to reboot ...
reboot (pc18): Attempting to reboot ...
reboot (pc19): Attempting to reboot ...
reboot (pc20): Attempting to reboot ...
reboot (pc5): Attempting to reboot ...
reboot (pc6): Attempting to reboot ...
reboot (pc7): Attempting to reboot ...
. Succeeded
Creating VLAN 21 as VLAN #73 on c4506-netlab ... Succeeded.
Applying VLAN changes on c4506-netlab ... Succeeded
TIMESTAMP: 20:23:57:692697 snmpit finished
Setting up email lists.
TIMESTAMP: 20:23:57:700871 genelists started
TIMESTAMP: 20:23:58:649645 genelists finished
Clearing port counters.
TIMESTAMP: 20:23:58:655092 portstats started
ep: pc6:1;pc18:1;pc5:0;pc7:0;pc16:3;pc19:3;pc20:3;pc12:3;pc14:3;pc11:3;pc6:0
TIMESTAMP: 20:24:00:343293 portstats finished
reboot (pc11): Successful!
reboot (pc12): Successful!
reboot (pc14): Successful!
reboot (pc16): Successful!
reboot (pc18): Successful!
reboot (pc19): Successful!
reboot (pc20): Successful!
reboot (pc5): Successful!
reboot (pc6): Successful!
reboot (pc7): Successful!
reboot: Done. There were 0 failures.
reboot (pc14): child returned 0 status.
reboot (pc5): child returned 0 status.
reboot (pc19): child returned 0 status.
reboot (pc16): child returned 0 status.
reboot (pc6): child returned 0 status.
reboot (pc20): child returned 0 status.
reboot (pc12): child returned 0 status.
reboot (pc7): child returned 0 status.
reboot (pc11): child returned 0 status.
reboot (pc18): child returned 0 status.
TIMESTAMP: 20:24:03:971058 rebooting/reloading finished
Waiting for local testbed nodes to finish rebooting ...
TIMESTAMP: 20:24:03:978306 Local node waiting started
Still waiting for pc14 - it's been 1 minute(s).
Still waiting for pc14 - it's been 2 minute(s).
pc14 is alive and well
pc5 is alive and well
pc19 is alive and well
Still waiting for pc6 - it's been 2 minute(s).
Still waiting for pc6 - it's been 3 minute(s).
Still waiting for pc6 - it's been 4 minute(s).
Still waiting for pc6 - it's been 5 minute(s).
Still waiting for pc6 - it's been 6 minute(s).
Still waiting for pc6 - it's been 7 minute(s).
Still waiting for pc6 - it's been 8 minute(s).
*** Giving up on pc6 - it's been 8 minute(s).
*** os_setup: Rebooting pc6 and waiting again ...
reboot (pc6): Attempting to reboot ...
*** node_reboot-reboot_node: pc6 appears dead; will power cycle.
pc6 now rebooting
reboot: Done. There were 0 failures.
pc16 is alive and well
pc20 is alive and well
pc12 is alive and well
pc7 is alive and well
pc18 is alive and well
pc11 is alive and well
Still waiting for pc6 - it's been 1 minute(s).
Still waiting for pc6 - it's been 2 minute(s).
Still waiting for pc6 - it's been 3 minute(s).
Still waiting for pc6 - it's been 4 minute(s).
Still waiting for pc6 - it's been 5 minute(s).
Still waiting for pc6 - it's been 6 minute(s).
Still waiting for pc6 - it's been 7 minute(s).
Still waiting for pc6 - it's been 8 minute(s).
*** Giving up on pc6 - it's been 8 minute(s).
*** WARNING: os_setup:
*** pc6 may be down. This has been reported to testbed-ops.
TIMESTAMP: 20:40:21:788743 Local node waiting finished
OS Setup Done.
*** ERROR: os_setup:
*** There were 1 failed nodes.
***
*** 1/10 pc2800's with a system osid of "FC6-STD" failed to boot: tbdelay0(pc6)
TIMESTAMP: 20:40:21:800884 os_setup finished
*** ERROR: tbswap: Failed to reset OS and reboot nodes.
Cleaning up after errors.
Stopping the event system
TIMESTAMP: 20:40:24:303659 snmpit started
Removing VLANs.
Removing VLAN #72 on c4506-netlab ... Succeeded.
Applying VLAN changes on c4506-netlab ... Succeeded
Removing VLAN #73 on c4506-netlab ... Succeeded.
Applying VLAN changes on c4506-netlab ... Succeeded
Removing VLAN #72 on c4006-netlab ... Succeeded.
Applying VLAN changes on c4006-netlab .... Succeeded
TIMESTAMP: 20:40:39:819179 snmpit finished
Tearing down virtual nodes.
TIMESTAMP: 20:40:39:827359 vnode_setup -k started
vnode_setup running at parallelization: 10 wait_time: 120
Vnode teardown finished.
TIMESTAMP: 20:40:41:739472 vnode_setup finished
Freeing nodes.
TIMESTAMP: 20:40:41:744844 nfree started
Releasing all nodes from experiment [Experiment: XenHack/LargeTest].
Moving [Node: pc18] to [Experiment: emulab-ops/reloadpending]
Moving [Node: pc12] to [Experiment: emulab-ops/reloadpending]
Moving [Node: pc11] to [Experiment: emulab-ops/reloadpending]
Moving [Node: pc16] to [Experiment: emulab-ops/reloadpending]
Moving [Node: pc20] to [Experiment: emulab-ops/reloadpending]
Moving [Node: pc14] to [Experiment: emulab-ops/reloadpending]
Moving [Node: pc19] to [Experiment: emulab-ops/reloadpending]
Moving [Node: pc7] to [Experiment: emulab-ops/reloadpending]
Moving [Node: pc5] to [Experiment: emulab-ops/reloadpending]
Moving [Node: pc6] to [Experiment: emulab-ops/hwdown]
TIMESTAMP: 20:40:43:632295 nfree finished
Resetting named maps.
TIMESTAMP: 20:40:43:638965 named started
TIMESTAMP: 20:40:44:621253 named finished
Resetting email lists.
TIMESTAMP: 20:40:44:629371 genelists started
TIMESTAMP: 20:40:45:577918 genelists finished
Resetting DB.
Failingly finished swap-in for XenHack/LargeTest. 20:40:45:660855
TIMESTAMP: 20:40:45:662472 tbswap in finished (failed)
*** ERROR: swapexp: tbswap in failed!
Cleaning up and exiting with status 1 ...
**** Experimental information, please ignore ****
Session ID = 11640
Likely Cause of the Problem:
There were 1 failed nodes.
1/10 pc2800's with a system osid of "FC6-STD" failed to boot: tbdelay0(pc1)
Cause: unknown
Confidence: 0.7
Script: os_setup
**** End experimental information ****
|