[Testbed-admins] problem with bge driver?

["David A. Kotz" <dkotz@cs.utexas.edu>]

I've recently added some brand-new Dell PowerEdge r200 machines with 
NetFPGA cards (to be dealt with later) to our testbed.  Users reported 
that their experiments failed to swap in when they set lan speed to 
1000Mbs, so I did some testing:

Hardware:
control switch: Cisco 3750 stack
experimental: Cisco 6509 w/ 48 port Gb blades
pcr200: Dell PowerEdge r200, onboard Broadcom Gb NICs (bge driver), 
NetFPGA (no drivers yet)
pc3001: Dell PowerEdge 2850, Intel NICs (em drivers)


r200 <---> 1000Mbs lan <---> r200 fails
r200 <---> 100Mbs lan <---> r200 works
r200 <--- 1000Mbs ---> r200 (no lan defined, link set to 1000) fails

pc3001 <---> 1000Mbs lan <---> pc3001 works


Looking at the database definitions for the NICs in both pc3001 and 
pcr200, I see no reason for the 100Mbs limitation.  Both have max speed 
set to 1000.  These are the first testbed machines we're used with the 
bge driver.

I allocated an r200 with default settings and got FC8 with a 100/full 
experimental link.  I forced the switchport to 1000, bounced the 
interface, and ifconfig in FC8 showed that I was now at 1000/full.

I allocated an r200 specifying FBSD-STD and got a 100/full experimental 
link.  I forced the switchport to 1000, bounced the link, and ifconfig 
from BSD showed "no carrier".  This would seem to indicate that the bge 
driver for BSD doesn't work for 1000 except that bge0, the control 
interface, is connected to the 3750 at 1000/full.

The hunch I was attempting to test was that maybe the control interface 
was running at 100, and there was either a hardware or software problem 
preventing one of the ports on the NIC (recall the r200 has a single 
dual-port onboard NIC) from running 1000 while the other was running 
100, but that's actually the *working* configuration in BSD.

What seems to be preventing the use of gigabit links is that emulab is 
BSD-centric, and when emulab creates the node it is using BSD, which is 
autonegotiating that link to a maximum of 100Mbs.

I'd greatly appreciate any insight anyone can provide on this issue.

- dave