Validation Experiences with the NS Simulator

Summary. The author defines validation as ensuring that the models used in a simulation are reasonable and verification as ensuring that a model's implementation is correct. What isn't addressed is ensuring that the simulator and model together reflect reality because (1) sometimes simulations are targeted to future network scenarios and (2) the Internet is inherently difficult to model due to its heterogeneity and constant evolution and (3) the real world implementations can have bugs (the former two points are listed as the two key challenges in defining models).

More Detail

• The main validation and verification of a model come not from validation tests (of which there are many in ns) but from "the continued exploration of the protocols and mechanisms under many different simulation scenarios, and the detailed examination of traces to understand the behavior of the simulations."

• While confident of her own validation of her research, she is less confident of the effectiveness of the existing validation tests for the arbitrary NS user. It is up to the NS user to validate that the underlying models in their simulation are appropriate.

• The paper stresses that it is each individual researcher's responsibility to verify that a model implemented in the simulator is the model they are expecting and assuming. For example, ns's one-way TCP implementation reflects a simplified model of one-way data xfer of fixed-size packets with no SYN or FIN packets. (A "richer" two-way implementation is available, also, but the one-way implementation is used as an example.)

Other Notes