[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Testbed-admins] experiment failed to set up VLANs
One of my users tried submitting a job several times and got the
following each time. It seems like emulab is timing out waiting for a
response from the experimental switch, but this is the only job that has
produced the error. We've started up several others before and since
with no problems.
Failed to set up VLANs.
--------- /usr/testbed/expwork/RouteStudy/RouteViews/swapexp.ZkvKGc --------
Running 'tbswap in RouteStudy RouteViews'
Beginning swap-in for RouteStudy/RouteViews (84). 01/24/2009 18:09:50
TIMESTAMP: 18:09:50:382440 tbswap in started
Checking with Admission Control ...
Mapping to physical reality ...
TIMESTAMP: 18:09:50:398346 assign_wrapper started
assign_wrapper improved started
TIMESTAMP: 18:09:50:786042 assign_wrapper started
TIMESTAMP: 18:09:50:790132 TOP started
Resetting DB before updating.
opened topfile
Minimum nodes = 4
Maximum nodes = 4
TIMESTAMP: 18:09:50:835527 TOP finished
TIMESTAMP: 18:09:50:836293 assign_loop started
Assign Run 1
TIMESTAMP: 18:09:50:837767 ptopgen started
ptopargs -p RouteStudy -e RouteViews
TIMESTAMP: 18:09:51:672451 ptopgen finished
TIMESTAMP: 18:09:51:675024 assign started
assign -P RouteStudy-RouteViews-74955.ptop RouteStudy-RouteViews-74955.top
BEST SCORE: 2.38 in 17000 iters and 0.164085 seconds
TIMESTAMP: 18:09:53:751367 assign finished
TIMESTAMP: 18:09:53:752588 reserving started
TIMESTAMP: 18:09:54:540210 reserving finished
Successfully reserved all physical nodes we needed.
TIMESTAMP: 18:09:54:546278 assign_loop finished
TIMESTAMP: 18:09:54:548425 LoadPhysResources started
TIMESTAMP: 18:09:54:556689 LoadPhysResources finished
TIMESTAMP: 18:09:54:562014 interpreting started
TIMESTAMP: 18:09:54:564161 interpreting finished
TIMESTAMP: 18:09:54:565063 uploading started
TIMESTAMP: 18:09:57:669665 uploading finished
TIMESTAMP: 18:09:57:671002 assign_wrapper finished
TIMESTAMP: 18:09:58:97843 assign_wrapper finished
Mapped to physical reality!
Fetching tarballs and RPMs (if any) ...
TIMESTAMP: 18:09:58:101936 tarfiles_setup started
TIMESTAMP: 18:09:58:957249 tarfiles_setup finished
Setting up mountpoints.
TIMESTAMP: 18:09:58:961491 mountpoints started
TIMESTAMP: 18:10:03:352264 mountpoints finished
TIMESTAMP: 18:10:03:354355 named started
Setting up named maps.
TIMESTAMP: 18:10:04:178290 named finished
TIMESTAMP: 18:10:04:181526 gentopofile started
Generating ltmap (again) ...
TIMESTAMP: 18:10:05:3365 gentopofile finished
Resetting OS and rebooting.
TIMESTAMP: 18:10:05:6764 launching os_setup
Setting up VLANs.
TIMESTAMP: 18:10:05:15888 snmpit started
TIMESTAMP: 18:10:05:474665 os_setup started
TIMESTAMP: 18:10:05:487308 rebooting/reloading nodes started
reboot (pc24): Attempting to reboot ...
reboot (pc25): Attempting to reboot ...
reboot (pc27): Attempting to reboot ...
reboot (pc28): Attempting to reboot ...
reboot (pc24): Successful!
reboot (pc25): Successful!
reboot (pc27): Successful!
reboot (pc28): Successful!
reboot: Done. There were 0 failures.
reboot (pc28): child returned 0 status.
reboot (pc24): child returned 0 status.
reboot (pc27): child returned 0 status.
reboot (pc25): child returned 0 status.
TIMESTAMP: 18:10:07:563452 rebooting/reloading finished
Waiting for local testbed nodes to finish rebooting ...
TIMESTAMP: 18:10:07:565236 Local node waiting started
pc28 is alive and well
pc24 is alive and well
pc27 is alive and well
pc25 is alive and well
TIMESTAMP: 18:11:04:380621 Local node waiting finished
OS Setup Done.
TIMESTAMP: 18:11:04:383277 os_setup finished
*** ERROR: snmpit:
*** SNMP GET failed - In /usr/testbed/bin/snmpit
*** SNMPIT get failed for device emu-exp-switch (try 30 of 30)
*** Variable was [sysDescr,0,(undefined),(undefined)]
*** Returned (undefined), ErrorNum was -24
*** Error string is: Timeout
*** ERROR: tbswap: Failed to set up VLANs.
Cleaning up after errors.
Waiting for os_setup to finish
Stopping the event system
TIMESTAMP: 18:14:00:441530 snmpit started
Removing VLANs.
*** ERROR: snmpit:
*** SNMP GET failed - In /usr/testbed/bin/snmpit
*** SNMPIT get failed for device emu-exp-switch (try 30 of 30)
*** Variable was [sysDescr,0,(undefined),(undefined)]
*** Returned (undefined), ErrorNum was -24
*** Error string is: Timeout
*** ERROR: tbswap: Failed to reset VLANs
TIMESTAMP: 18:17:53:156064 snmpit finished
Tearing down virtual nodes.
TIMESTAMP: 18:17:53:158599 vnode_setup -k started
vnode_setup running at parallelization: 10 wait_time: 120
Vnode teardown finished.
TIMESTAMP: 18:17:54:946446 vnode_setup finished
Freeing nodes.
TIMESTAMP: 18:17:54:949818 nfree started
Releasing all nodes from experiment [Experiment: RouteStudy/RouteViews].
Moving [Node: pc28] to [Experiment: emulab-ops/reloadpending]
Moving [Node: pc24] to [Experiment: emulab-ops/reloadpending]
Moving [Node: pc27] to [Experiment: emulab-ops/reloadpending]
Moving [Node: pc25] to [Experiment: emulab-ops/reloadpending]
TIMESTAMP: 18:17:56:470358 nfree finished
Resetting named maps.
TIMESTAMP: 18:17:56:474576 named started
TIMESTAMP: 18:17:57:314173 named finished
Resetting email lists.
TIMESTAMP: 18:17:57:320068 genelists started
TIMESTAMP: 18:17:58:129642 genelists finished
Resetting DB.
Failingly finished swap-in for RouteStudy/RouteViews. 18:17:58:149623
TIMESTAMP: 18:17:58:150786 tbswap in finished (failed)
*** ERROR: swapexp: tbswap in failed!
Cleaning up and exiting with status 1 ...
**** Experimental information, please ignore ****
Session ID = 349621
Likely Cause of the Problem:
Failed to set up VLANs.
Cause: unknown
Confidence: 0.7
Script: tbswap
**** End experimental information ****