a network intrusion detection system test suite

© 1999 Anzen Computing


Nidsbench is a toolkit for testing network intrusion detection systems (NIDS).

A NIDS is a computer security system which detects misuse, attacks against, or compromise of computers connected to a network. They operate by passively examining network packets as they travel over the wire and alerting administrators when they see something unusual or malicious. [1]

Network intrusion detection is still something of a black art - while it is intuitively easy to understand (analogies to burglar alarms, traffic speed traps, etc. abound), the implementation details are often overlooked. For example, in a seminal paper on network intrusion detection published last year, Ptacek and Newsham demonstrated that the vast majority of commercially-available NIDSs are trivially defeated. Fundamental problems in passive monitoring of TCP/IP limit the ability of a NIDS to correctly determine what's actually happening at the endpoint of a traffic stream, and most NIDSs actually do nothing to correct for them. [2]

The goal of the nidsbench project is to provide better tools for evaluating NIDS products and to help standardize a testing methodology for the purpose of objective comparison. Other groups are already working toward the same goal - some industry magazines have their own security test labs (such as InfoWorld and DataComm), IDS shootouts are being featured at industry conferences, and a few research groups have made much headway in the areas of NIDS taxonomy, formal testing environments, reference network attack corpora, etc. [3]


Nidsbench provides tools to evaluate two measurable NIDS characteristics: performance and correctness. Of course, there are many other features that could be evaluated, but none as objective or easily quantifiable.

Nidsbench includes the following programs to do this:

Tcpreplay is aimed at testing the performance of a NIDS by replaying real background network traffic in which to hide attacks. Tcpreplay allows you to control the speed at which the traffic is replayed, and can replay arbitrary tcpdump traces. Unlike programmatically-generated artificial traffic which doesn't exercise the application/protocol inspection that a NIDS performs, and doesn't reproduce the real-world anomalies that appear on production networks (asymmetric routes, traffic bursts/lulls, fragmentation, retransmissions, etc.), tcpreplay allows for exact replication of real traffic seen on real networks.

Fragrouter is aimed at testing the correctness of a NIDS, according to the specific TCP/IP attacks listed in the Secure Networks NIDS evasion paper. [2] Other NIDS evasion toolkits which implement these attacks are in circulation among hackers or publically available, and it is assumed that they are currently being used to bypass NIDSs. [4]

Idstest is aimed at testing the correctness of a NIDS by actually performing the attacks such systems are supposed to detect. In theory, this is no different from what several commercially-available vulnerability scanners do, except that many of them only look for vulnerability symptoms (ex. versions reported in software banners) instead of actually attempting exploits.

Nidsbench does not include a corpus of data to run tests with, nor does it specify a procedure or methodology to use in evaluating NIDSs. We hope that our software is useful to those looking for the tools to instrument such tests, but we have refrained from trying to specify how they should actually be conducted - the rest is up to you!


Tcpreplay and fragrouter are available now for downloading.

File Description Supported Platforms
fragrouter-1.1.tar.gz Fragrouter version 1.1 BSD, Linux, others?
tcpreplay-1.0.1.tar.gz Tcpreplay version 1.0.1 BSD, Linux, Solaris, others?


Please send all questions, comments, and bug reports to <>.


  1. General NIDS information:

  2. Problems in passive network monitoring:

  3. Related NIDS evaluation projects:

  4. Publically-available NIDS evasion toolkits:
© 1999 Anzen Computing. All rights reserved.