When uncongested, the real and simulated throughput were equal. When congested, however, simulated throughput differed slightly to significantly: 3.5% to 18.2% for the two congested experiments.
Even in these simple experiments, when congestion was present the simulator slowed to 6 and 7 times the specified runtime (5 secs.).
The above suggests that if we're interested in finding differences between simulated results and testbed results, we should look for experiments that introduce congestion into the environment (our experience suggests that this will result in different delay and bandwidth numbers) and are relatively complicated (our experience suggests this will highlight slow simulator runtime).
(Note that our testbed has uses other than as a complement to network simulators, but that's our emphasis here.)
Results for the real machine experiments were averaged over 10 runs (not 30, but enough to collect an average). Simulations were only run once.
Experiments:
In experiments #2-#7, a delay had to be set for the links.
155.99.213.184/29 155.99.213.190/29
(falcon) <-- 100Mbs --> (chukar) <-- 10Mbs --> (vulture)
155.99.213.198/29 155.99.213.193/29
Chukar routes packets between Falcon and Vulture on two
otherwise unused subnets thus eliminating cross-traffic.
Copies of the rc.conf files required for this configuration are
in
~kwright/src/vint/ns/tcl/kw/etc-[falcon,chukar,vulture].tar.
(Note: There is one thing that the rc.conf files don't reflect:
I modified the 10Mbs endpoints to be full-duplex by hand:
On chukkar:
ifconfig fxp2 media 10baseT/UTP mediaopt full-duplex
On vulture:
ifconfig fxp1 media 10baseT/UTP mediaopt full-duplex
These configurations could be added into the rc.conf ifconfig
lines.
| Experiment | Machine | Simulator |
| One-way 1M, Uncongested | 10.04 | 10.00 |
| One-way 2M, Uncongested | 10.03 | 10.00 |
| One-way 3M, Uncongested | 10.00 | 10.00 |
| RTT 3M, Uncongested | 10.01 | 10.00 |
The estimated send intervals are omitted for the congested experiments because, since packets were lost due to congestion, the difference in send times between consecutive packets received does not reflect accurately the difference in send times since dropped packets would serve to spuriously increase the interval.
The real-time delay intervals tended to be slightly longer than the simulator's. This did have an effect on throughput, but only slightly. In one-way 1M, the delay was on average 10.04. This could make a difference in throughput: 498 packets rather than 500. As we see below, this was exactly the number observed: the simulator reported a throughput of 500 packets where the real machines generated only 498 packets.
$ns duplex-link $n0 $n1 10Mb 0ms DropTail
| 1M | S with 10Mbs | S with 100Mbs |
| 0.12 | 0.82 | 0.082 |
From the relationship between the 10Mbs and 100Mbs S delays, we can see the bw is still being taken into account in spite of a 0 delay. A more correct way to do this would be to change the ns script to have two attached agents assigned to a single node.
We can conclude something about ns overhead from the other experiments. I ran the unix command time on the ns run. Below are the results:
| Experiment | time output |
| One-way 1M, Uncongested | 1.733u 0.031s 0:01.86 94.6% 2463+2982k 0+12io 0pf+0w |
| One-way 2M, Uncongested | |
| One-way 3M, Uncongested | 2.046u 0.038s 0:02.32 89.2% 2508+3140k 0+61io 0pf+0w |
| One-way 3M, Congested | 8.293u 0.438s 0:28.51 30.5% 2383+3333k 27+605io 202pf+0w |
| RTT 3M, Uncongested | 2.365u 0.086s 0:04.19 58.2% 2378+3085k 9+122io 10pf+0w |
| RTT 3M, Congested | 12.240u 0.442s 0:35.02 36.2% 2354+3352k 0+1150io 0pf+0w |
All of the uncongested experiments ran in less than the 5 second simulation length. (This is possible in ns because the simulated time is not real-time so that if there are no events in the event loop to process, the scheduler moves the clock ahead to the next event time and proceeds.)
When the experiments induce congestion, however, the simulator slows significantly: to 28.51 and 35.02 seconds for the one-way and round-trip 5-second simulations, respectively.
Though we expected the simulator to slow down when executing complicated simulations, I was surprised to see the runtime increase by so much for such a simple experiment (few nodes, little traffic, no route changes or calculations, etc.). It would be interesting to see how much the simulator slows down in a larger, more complex experiment.
The two uncongested test cases' results are a different matter: they show slight to significant differences: from 3.5% to 18.2%.
All results are in the table below.
| Experiment | M Throughput | S Throughput | % Off of Real Throughput |
| One-way 1M, Uncongested | (498/509952) | (500/512000) | +00.4% |
| One-way 2M, Uncongested | (498.3/510259.2) | (500/512000) | +00.3% |
| One-way 3M, Uncongested | (499.9/511897.6) | (500/512000) | +00.0% |
| One-way 3M, Congested | (4312.9/6038060) | (4463/6248200) | +03.5% |
| RTT 3M, Uncongested | (500.2/700280) | (500/700000) | -00.0% |
| RTT 3M, Congested | (3775.17/5285233.33) | (4463/6248200) | +18.2% |
The packet delays measured for each experiment are listed below. Note that the packet delay can only be considered reliable in the round-trip time experiments due to the possibility of coarse-grained NTP. These experiments are highlighted in pink.
| Experiment | M Delay (secs) | S Delay (secs) | % Off of Real Delay |
| One-way 1M, Uncongested | .00012 | .00008 | N/A |
| One-way 2M, Uncongested | .00047 | .00020 | -57.4% |
| One-way 3M, Uncongested | .00097 | .00136 | +86.0% |
| One-way 3M, Congested | .06958 | .05056 | -27.3% |
| RTT 3M, Uncongested | .00284 | .00231 | -18.7% |
| RTT 3M, Congested | .14415 | .05117 | -64.5% |
It is important to look at how the link delay (versus the packet delay) is set in ns. The link delays for one-way 2M was set to be equal to half the RTT of an ICMP request sent from the source to sink on the 2M platform. This is the ping output from Kamas to Eureka (the real machines):
30 packets transmitted, 30 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.226/0.239/0.255/0.008 msThe ns delay for the 2M experiment was set like so:
$ns duplex-link $n0 $n1 100Mb .1195ms DropTail
The ping results for the 3M experiments are:
source to router:
30 packets transmitted, 30 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.130/0.140/0.226/0.016 ms
PING 155.99.213.193 (155.99.213.193): 56 data bytes
router to bouncer:
30 packets transmitted, 30 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.311/0.320/0.407/0.018 ms
The corresponding 3M delays were set in ns as follows:
$ns duplex-link $n0 $r 100Mb .14ms DropTail
$ns duplex-link $r $n1 10Mb .32ms DropTail
If you compare the delay times derived from ping with the
actual delay times, it appears that the ping delay times are
way too long for these experiments -- that is, it appears that
ping isn't a good estimater of link delay. If this is so,
then since we used ping delays to determine the simulator's
link delays, then we would expect to see the delays reported
by the simulator to be much higher. However, the
simulated delays for RTT 3M, Uncongested are very close
to the real delays. In fact, they are less than the
combined one-way link delay specified (.14 + .32 = .46ms).
This leaves me wondering exactly what this delay is used
for in ns.
On the other hand, since the delays are relatively close in the
uncongested RTT case yet so disparate in the congested RTT case,
this leads me to believe that ns is not modeling our congestion
scenario accurately. That is, it doesn't appear that the
different results for RTT 3M, Congested are caused
merely by bad delay numbers in the simulator.
Graphs
In spite of the fact that we're probably not modeling the same thing in ns as in our real environment, it is still interesting to quantify the differences between platforms and identify the reasons for the differences. Doing so will allow us to continue with bigger experiments with greater intuition about where the two platforms will differ.
It takes each packet a minimum of 1.12 ms to cross the second link and .0112ms to cross the first link. The total one-way delay is 1.12ms + .0112ms = 1.23ms. Round-trip time is then a minimum of 2.46ms. We know from the delay section that the RTT 3M, Uncongested experiment had an average RTT of 2.84ms. The graph visually shows this.
Compare with the graph above the following phase plot of RTT 3M, Congested.
Temporary increase: Recall that we send 1400 byte packets every 10ms. Relative to the The lines make a little more obvious the following: Recall that 10 1400 byte packets every 10ms and the delay for a packet to be transmitted across the 10Mb/s link is 1.12 ms. To send 10 packets takes at least 11.2ms. This means that after each 10ms sending interval, one packet remains at the router node waiting to be transmitted across the bottleneck link. As the experiment progresses, the bottleneck link's queue grows at one packet per 10ms.
a In times of little congestion, each packet in each group of 10 packets increases it's RTT by approximately 1ms. This means that when no congestion is present the 10th packet has an RTT of approximately 13ms. The lower line (x = y + 10)
Talking to Alastair, he thinks this is not so and that the queue that is filling is the outgoing queue. He guesses that the incoming queue stays relatively small.
