A performance comparison of recent network
simulators
Elias Weingärtner, Hendrik vom Lehn and Klaus Wehrle
Distributed Systems Group
RWTH Aachen University
Aachen, Germany
Email: {weingaertner,vomlehn,wehrle}@cs.rwth-aachen.de
Abstract—A widespread methodology for performance analysis with hundreds of thousands of nodes. In order to face those
in the field of communication systems engineering is network challenges, a couple of enhancements of ns-2 have been
simulation. While ns-2 has established itself as virtually the proposed, for instance the incorporation of parallelization [16].
standard network simulation tool, other network simulators have
gained more and more attention during the last years. In this However, ns-2 is currently undergoing a major redesign [6].
paper, we briefly survey new developments in the field of network One of the main development goals of its successor, ns-3, is
simulation and conduct a performance comparison study by the improvement of simulation performance.
implementing an identical simulation set-up in five simulators, Besides ns-2, over a dozen network simulators are presently
namely ns-2, OMNet++, ns-3, SimPy and JiST/SWANS. Our
used in academia and in the industry. Prominent examples
results reveal large differences according to both run-time per-
formance and memory usage. include OMNeT++ [19], the Java-based JiST [3] and com-
mercial tools such as the OPNet modeler [14]. In addition,
I. I NTRODUCTION specialized simulation tools, such as the WSN simulator
Network simulation is without a doubt one of the most TOSSIM [9], serve dedicated research domains. This leaves
predominant evaluation methodologies in the area of computer many researchers and graduate students with the question of
networks. It is widely used for the development of new which network simulator to use, especially if one is interested
communication architectures and network protocols. So-called in achieving a high simulation performance.
network simulators allow one to model an arbitrary computer In this paper, we focus on current developments regarding
network by specifying both the behavior of the network nodes open source simulators. In Section II, we provide a brief
and the communication channels. For example, in order to overview of network simulators which have recently gained
investigate the characteristics of a new routing protocol, it is attention in the research community. The main contribution
usually implemented in a network simulator. Afterwards, the of this paper is a performance comparison study incorporat-
routing behavior can be easily studied in different topologies, ing five different open source simulation tools, namely ns-
given the fact that the network topology is merely a set 2 [13], ns-3 [6], OMNeT++ [19], JiST [3] and SimPy [11].
of simulation parameters. Most available network simulation By implementing the same simulation and equal simulation
toolkits are based on the paradigm of discrete event-based models from scratch for all of them, we are able to compare
simulation [5] (DES). Here, the simulated network nodes the simulator performance itself without any distortions caused
trigger events, for instance, when a packet is sent to another by different implementations of simulation models. The design
node. The simulator maintains an event queue sorted by and the outcome of this performance comparison study are
the scheduled event execution time. The simulation itself is discussed in Section III. As a matter of fact, this is not the
performed by successively processing the events in the queue. first performance comparison of network simulators. However,
The first approaches where DES was applied to the simula- to our knowledge none of those studies include recent con-
tion of computer networks were published about two decades tributions like ns-3. In Section IV, we discuss such related
ago [4, 8]. ns-2 [13] is a direct successor of those early performance evaluation studies and compare their results with
efforts and since then, it has become virtually the standard the ones presented in this paper. We conclude in Section V
for network simulation. This can be attributed to the fact that with the lessons we learned from this performance comparison.
numerous models, e.g. protocol models and traffic generators,
are publicly available for ns-2. They can be used off-the- II. I NVESTIGATED S IMULATION T OOLS
shelf, thus eliminating the need of implementing them by
hand. However, a major shortcoming of ns-2 is its limited In this section, we concisely introduce the network simula-
scalability [6, 20] in terms of memory usage and simulation tors considered in the performance comparison. We emphasize
run-time. This is especially a problem as new research domains that ns-3, OMNeT++ and JiST are all gaining more and
in the field of computer networks, such as wireless sensor more prevalence compared to the long-established ns-2. We
networks (WSNs), peer-to-peer networks or grid architectures, include ns-2 here to form a baseline. In addition, SimPy
require the simulation of very large networks, potentially was incorporated in the comparison as it represents a modern
implementation of a process-oriented simulator in the popular Like the aforementioned ns-2 and ns-3, OMNet++ rests
Python language. upon C++ for the implementation of simple modules. How-
ever, the composition of these simple modules into compound
A. ns-2 modules and thus the set-up of network simulation takes place
Network simulations for ns-2 are composed of C++ code, in NED, the network description language of OMNeT++. NED
which is used to model the behavior of the simulation nodes, is transparently rendered into C++ code when the simulation
and oTcl scripts that control the simulation and specify further is compiled as a whole. Moreover, NED supports the specifi-
aspects, for instance the network topology. This design choice cation of variable parameters in the network description: For
was originally made to avoid unnecessary recompilations if example the number of nodes in a network can be marked to be
changes are made to the simulation set-up [6]. Back in dynamic and later on be configured at runtime. In this case, the
1996 when the first version of ns-2 was released, this was modules representing the nodes are dynamically instantiated
a reasonable intent, as the frequent recompilation of C++ by the simulator during execution. This feature is a direct
programs was indeed time-consuming and slowed down the consequence of the simulator’s strict object-oriented design.
research cycle. However, from today’s perspective, the design
of ns-2 trades off simulation performance for the saving of D. JiST
recompilations, which is questionable if one is interested in A fresh approach to network simulation is JiST (“Java in
conducting scalable network simulations. Simulation Time”), which in compliance with its name allows
the implementation of network simulations in standard Java.
B. ns-3
It is mostly used in conjunction with SWANS1 , a simulator
Like its predecessor, ns-3 relies on C++ for the imple- for mobile ad hoc networks built on top of JiST.
mentation of the simulation models. However, ns-3 no longer Network simulations in JiST are made up of entities which
uses oTcl scripts to control the simulation, thus abandoning represent the network elements, for example nodes, with simu-
the problems which were introduced by the combination lation events being formed by method invocations among those
of C++ and oTcl in ns-2. Instead, network simulations in entities. The entities advance the simulation time indepen-
ns-3 can be implemented in pure C++, while parts of the dently by notifying the simulation core. While the code inside
simulation optionally can be realized using Python as well. an entity is executed like any arbitrary Java program, only
Moreover, ns-3 integrates architectural concepts and code from the interactions between the individual entities are carried out
GTNetS [17], a simulator with good scalability characteristics. in simulation time. Thus, these interactions between entities
These design decisions were made at expense of compatibility. correspond to synchronization points and facilitate the parallel
In fact, ns-2 models need to be ported to ns-3 in a manual execution of code at different entities, resulting in a potential
way. Besides performance improvements, the feature set of performance gain. In order to execute the implementation in
the simulator is also about to be extended. For example, ns-3 simulation time, JiST utilizes a custom dynamic Java class
is slated to support the integration of real implementations’ loader which dynamically rewrites the application’s byte code.
code by providing standard APIs, such as Berkeley sockets Unfortunately, the official development of JiST has stalled,
or POSIX threads, which are transparently mapped to the as it is no longer maintained by its original author, Rimon Barr.
simulation [1]. However, a couple of enhancements and improvements have
C. OMNeT++ recently been released by Ulm University2. We incorporate
those enhancements in our performance analysis of JiST in
In contrast to ns-2 and ns-3, OMNeT++ is not a network Section III.
simulator by definition, but a general purpose discrete event-
based simulation framework. Yet it is mostly applied to the E. SimPy
domain of network simulation, given the fact that with its
With SimPy, we include a process-oriented discrete-event
INET package it provides a comprehensive collection of
simulator in this performance comparison. Unlike the other
Internet protocol models. In addition, other model packages
simulators, no public available network models exist for
such as the OMNeT++ Mobility Framework and Castalia [15]
SimPy. Instead, it is a bare simulation API written in Python.
facilitate the simulation of mobile ad hoc networks or wireless
In SimPy, the basic simulation entities are processes. They
sensor networks.
are executed in parallel and may exchange Python objects
OMNeT++ simulations consist of so-called simple modules
among each other. Most processes include an infinite loop
which realize the atomic behavior of a model, e.g. a particular
in which the main actions of the process are performed.
protocol. Multiple simple modules can be linked together
Besides abstractions for processes and the related exchange of
and form a compound module. For instance, multiple simple
objects, SimPy provides instructions for the synchronization
modules which provide protocol models can be combined into
of simulation processes and commands for the monitoring of
a compound module representing a host node. A network
simulation data.
simulation in OMNeT++ is implemented itself as a compound
module which comprehends other compound modules, like the 1 JiST/SWANS Website: http://jist.ece.cornell.edu/
ones which model host nodes. 2 Ulm University’s JiST portal: http://www.vanet.info/jist-swans/
OMNeT++
Sender 0 1 2 3 ns−3
JiST
SimPy
ns−2
1
0.8
0.6
Loss 0.4
4 5 6 7
0.2
1200
1000
1 800
8 9 10 11 0.8
0.6 400
600
Network Size
0.4
Drop Probability 0.2 200
0 0
Fig. 2. End-to-End Packet Loss
12 13 14 15 Receiver 1400
OMNeT++
ns−3
1200 JiST
Fig. 1. Sample Network Topology (size=16) SimPy
ns−2
1000
Computation Time [s]
III. P ERFORMANCE C OMPARISON
800
This section describes the methodology and the outcome of
the performance study which includes the simulation tools in- 600
troduced in the previous section. The comparison is based on a 400
benchmark scenario and discloses large differences according
200
to simulation run-time and memory usage.
With the goal of comparing the simulators’ core perfor- 0
0 500 1000 1500 2000 2500 3000 3500
mance, we first implemented a reference simulation in all Network Size
simulation toolkits from scratch3 . Our benchmarking simu-
lation does not rely on any existing simulation model for Fig. 3. Simulation runtime vs. Network size
any simulator. This decision was made because a network
between 0 and 1 with the square topology size ranging from
simulation’s performance is largely dependent on the code of
4 to 1024 nodes. The simulation time was set to 600 seconds.
the network models and their computational complexity.
Figure 2 depicts the end-to-end packet loss retrieved from the
The reference simulation models a basic network, where
five simulations, given the drop probability and a network size.
the nodes are arranged in a square topology as illustrated
Only SimPy produces slightly higher loss rates on average,
in Figure 1. One sending node generates one packet every
however still within the limits of tolerance. From these results,
second and broadcasts it to its neighbors. The neighboring
we conclude that our independent implementations of the
nodes relay unseen messages after a delay of one second, thus
reference simulation in fact produce equivalent results.
flooding the entire network. The propagation delay is directly
implemented by delaying the simulation events’ execution, and B. Performance comparison
the nodes do not implement a explicit queueing policy. With
a fixed probability which is equal on every link, packets are Given the fact that our simulation set-ups produce equal
dropped on the channel. The receiver is located at the corner results, we now compare the individual simulation tools re-
opposite to the receiver. We chose this simulation scenario for garding two performance metrics: effective simulation run-
its simplicity, not aiming at a simulation of a real network. time and memory usage. In order to evaluate the simulators’
scalability, we conducted two series of different runs using the
All simulation runs were conducted on a AMD Athlon 64
reference simulation. In the first series, the drop probability is
3800+ workstation with 2GB of RAM, running Ubuntu Linux
set to a fixed value of 0.10 with the network size ranging
8.04 LTS. Our measurements were taken using ns-2 version
from 4 to 3025 nodes. The second series uses a fixed network
2.33, OMNeT 3.4b2, ns-3.1, SimPy 1.9.1 and JiST 1.06 with
size of 3025 nodes, given drop probabilities between 0.0 and
the extensions from Ulm University. We made use of SUN
1.0. All results provided in the following are averages over
Java 1.5.0.11 for the execution of JiST, and SimPy was run
five executions of each simulation series. In both series, the
with Python 2.5.1.
simulation time was set to 600 seconds.
A. Model equality 1) Simulation run-time: Figure 3 shows the measured sim-
ulation runtime at different network sizes for the compared
As we implemented the same simulation set-up in five
simulation tools. First of all, these results reveal that SimPy
different simulators, we first checked if our implementations
does not scale well and hence is not applicable to large-scale
yield results which are on par with each other. For this purpose,
network simulations: For a network size of 3025 nodes, it
we ran the simulation in all simulators for drop probabilities
needs 1225 seconds on average to complete the simulation run.
3 The respective source code is available at In contrast to that, JiST finishes the same task about 14 times
http://ds.cs.rwth-aachen.de/research/projects/simcompare/ faster, resulting in an average execution time of 86 seconds.
140
OMNeT++
ns−3
120 JiST
SimPy
ns−2
100
Memory Usage [MB]
80
60
40
20
0
0 500 1000 1500 2000 2500 3000 3500
Network Size
Fig. 5. Memory usage vs. Network size
Fig. 4. Simulation run-time vs. Drop probability 160
OMNeT++
ns−3
140 JiST
SimPy
The overall run-time performance of JiST is astonishing at 120 ns−2
the first glance, given the fact that it is based on Java and
Memory Usage [MB]
100
still outperforms OMNeT++ and ns-3, which are executed
in a native manner. We attribute this winning margin to the 80
architecture of JiST: Besides the parallel execution at different 60
entities, JiST performs different run-time optimizations based
40
on the analysis of the executed byte code. Moreover, it has
been shown that the slowness of Java is merely a myth, and 20
that recent Java run-time environments can keep up with the 0
0 0.2 0.4 0.6 0.8 1
execution speed of compiled C++ code [10]. According to Drop Probability
the run-time performance of ns-3, the architectural improve-
ments, especially the abolishment of the oTCL/C++ duality, Fig. 6. Memory usage vs. Drop probability
are clearly reflected in our results, as ns-3 is considerably
faster than its predecessor. While the run-time performance additional memory requirements of the Java virtual machine
of OMNeT++ is slightly inferior to ns-3 and JiST, all three do not suffice as single explanation. The memory usage
simulation tools exhibit almost the same scalability according performances of ns-2, OMNeT++ and SimPy share a similar
to simulation run-time. linear growth of memory usage, with ns-3 being the most
Additional insight about the run-time behavior of the dif- efficient simulation tool in this regard.
ferent simulators can be derived from the results in Figure Figure 6 shows the memory usage measured during the
4. Here, we picture the averaged run-time from the second second simulation series. As mentioned earlier, for larger drop
simulation series for a fixed network size of 3025 nodes and probabilities the number of events prevalent in the simulation
a varying drop probability. With increasing drop probabilities, is considerably small and thus the memory usage here remains
the simulation run-time naturally decreases in a quick fashion almost constant. This almost “constant” memory footprint is
for all simulators, as more and more packets are removed mostly constituted by the simulation core and a potential run-
from the simulation, thus resulting in fewer events to be time environment, such as the JavaVM in the case of JiST
processed. In other words, the drop probability directly reflects or Python, which is required for the execution of SimPy.
the quantity of events prevalent in the simulation. We notice According to JiST, the footprint is slightly larger than that of
that SimPy’s simulation run-time increases much faster at low ns-2 and the other simulators, but far smaller than the one of
drop probabilities than any one of the other simulators. From SimPy. However, for lower drop probabilities and hence more
our results we conclude that SimPy in fact has a lower event simulation events, the memory usage of JiST grows faster than
throughput than the other simulation cores. with any other simulation tool.
2) Memory usage: Similar to our analysis of the simulation
run-time, we measured the maximum memory usage of the IV. R ELATED W ORK
individual simulators during the two series of simulation runs. A couple of network simulator performance comparisons
The outcome is depicted in Figure 5. Surprisingly, JiST uses have been published in recent years. Most of the more recent
up much more memory resources than the other simulation ones compare ns-2 with other simulation tools. One example is
tools. We first attributed this behavior to the garbage collection the work presented in [12], where the performance of a TCP-
mechanism, but the amount of used memory does not decrease based reference simulation implemented in ns-2 is checked
if the garbage collection is manually triggered at times. In against SSFNet and JavaSim (now known J-Sim), two older
addition, the difference in memory usage between JiST and simulators. In their work, the authors also observe large dif-
the other tools increases at larger network sizes, and hence, the ferences regarding memory consumption and simulation run-
time, with ns-2 performing best according to computational R EFERENCES
demands and worst according to memory consumption. [1] ns-3 Overview (June 2008).
A performance comparison, which in addition to ns-2, also http://www.nsnam.org/docs/ns-3-overview.pdf, June 2008.
includes earlier versions of SimPy and OMNeT is presented [2] D. Albeseder and M. Fuegger. Small PC-Network Simulation
- a comprehensive performance case study. Research Report
in [2]. The authors only provide results concerning the run-
77/2005, TU Wien, Institut für Technische Informatik, 2005.
time performance in their paper, and the used simulation [3] R. Barr, Z. J. Haas, and R. van Renesse. JiST: an efficient
is small in terms of network size. However, the outcome approach to simulation using virtual machines. Softw, Pract.
is similar to ours, with OMNeT++ outperforming ns-2 and Exper, 35(6):539–576, 2005.
SimPy. Regarding SimPy, the authors also note its sluggish [4] A. Dupuy, J. Schwartz, Y. Yemini, and D. Bacon. Nest: a
network simulation and prototyping testbed. Commun. ACM,
performance.
33(10):63–74, 1990.
Two recent publications [7, 18] analyze the characteristics [5] G. S. Fishman. Principles of Discrete Event Simulation. John
of JiST/SWANS in contrast to ns-2. Unlike our work, which Wiley & Sons, Inc., New York, NY, USA, 1978.
focuses mainly on the performance of the simulation cores, [6] T. R. Henderson, S. Roy, S. Floyd, and G. F. Riley. ns-3 project
the authors compare ns-2 and JiST/SWANS in a complex goals. In WNS2 ’06: Proceeding from the 2006 workshop on
simulation of a mobile ad hoc network, using available imple- ns-2: the IP network simulator, page 13, New York, NY, USA,
2006. ACM.
mentations of routing protocols bundled with both simulators. [7] F. Kargl and E. Schoch. Simulation of manets: a qualitative
Given the same parameters, the authors observe that ns-2 comparison between JiST/SWANS and ns-2. In MobiEval
and JiST/SWANS require about the same time to finish the ’07: Proceedings of the 1st international workshop on System
simulation run, with ns-2 exhibiting much higher memory evaluation for mobile platforms, pages 41–46, New York, NY,
demands than JiST in the given scenario. At first sight, USA, 2007. ACM.
[8] S. Keshav. Real: A network simulator. Technical report,
this seems to contradict our results presented in Section III. University of California at Berkeley, Berkeley, CA, USA, 1988.
However, this phenomenon can be explained by the fact that [9] P. Levis, N. Lee, M. Welsh, and D. Culler. TOSSIM: accurate
the radio models of ns-2 duplicate messages in memory, e.g. and scalable simulation of entire TinyOS applications. In
if a packet is broadcasted to other nodes. On the contrary, the In Proceedings of the 1st ACM Conference on Embedded
radio models implemented in SWANS pass solely references Networked Sensor Systems (SenSys 2003)., 2003.
[10] J. Lewis and U. Neumann. Performance of Java versus C++.
to static packet data among the entities, resulting in a much http://www.idiom.com/∼zilla/Computer/javaCbenchmark.html
smaller memory consumption. These results, in combination (accessed 08/25/2008), 2004.
with ours, affirm that the scalability of simulations and related [11] K. Mueller. SimPy documentation.
performance matters are in fact heavily influenced by the http://simpy.sourceforge.net/discuss.htm.
simulation models. [12] D. M. Nicol. Scalability of network simulators revisited. In
Proceedings of the Communication Networks and Distributed
V. C ONCLUSION Systems Modeling and Simulation Conference, Orlando, FL,
February 2003.
In this paper, we investigated the performance requirements [13] The network simulator ns-2. http://www.isi.edu/nsnam/ns/.
and the scalability of five different simulation tools. Our results [14] OPNET Technologies Inc. OPNET modeler website.
show that three of them, ns-3, OMNeT++ and JiST are all http://www.opnet.com/solutions/network\ rd/modeler.html.
capable of carrying out large-scale network simulations in [15] H. N. Pham, D. Pediaditakis, and A. Boulis. From simulation
an efficient way. JiST has proven to be the fastest simulator to real deployments in wsn and back. Proceedings of the 2007
IEEE International Symposium on a World of Wireless, Mobile
by far in our experiments, however the exhaustive memory and Multimedia Networks (WoWMoM 2007)., pages 1–6, June
consumption may limit its applicability in some simulation 2007.
scenarios. In our performance comparison, ns-3 demonstrated [16] G. Riley. PDNS project website.
the best overall performance. Although it was surpassed by http://www.cc.gatech.edu/computing/compass/pdns/.
JiST in terms of simulation run-time, it still shows both low [17] G. Riley. Large scale network simulations with GTNetS. In
Proceedings of the 2003 Winter Simulation Conference, 2003.
computational and less memory demands. However, at present [18] E. Schoch, M. Feiri, F. Kargl, and M. Weber. Simulation of ad
ns-3 still is in the early stages, and just a few simulation hoc networks: ns-2 compared to JiST/SWANS. In Proceedings
models exist which one can use off the shelf. As the rich of the First International Conference on Simulation Tools and
collection of models for ns-2 still needs to be ported from ns- Techniques for Communications, Networks and Systems (SIMU-
2 to ns-3, OMNeT++ can be considered as viable alternative. Tools 2008’), March 2008.
[19] A. Varga and R. Hornig. An overview of the OMNeT++
While its performance is slightly inferior than that of ns-3 simulation environment. In Proceedings of the First Inter-
and JiST, over the last few years a very comprehensive set national Conference on Simulation Tools and Techniques for
of models has been developed for this simulator. Moreover, Communications, Networks and Systems (SIMUTools 2008’),
OMNeT++ provides a rich graphical user interface and an March 2008.
abstract modeling language, while JiST and ns-3 rely on pure [20] Y. Xue, H. S. Lee, M. Yang, P. Kumarawadu, H. Ghenniwa, and
W. Shen. Performance evaluation of ns-2 simulator for wireless
source code for the development of the entire simulation. In sensor networks. Proceedings of the Canadian Conference on
conclusion, the question of which simulator to use is a difficult Electrical and Computer Engineering (CCECE 2007), pages
one, and the answer is largely dependent on the specific use 1372–1375, April 2007.
case. However, if scalability is the main concern, JiST, ns-3
and OMNeT++ are smart choices.
