CommBench Homepage

Overview

CommBench is a benchmark for use in evaluating and designing telecommunications network processors. The benchmark applications focus on small, computationally intense program kernels typical of the network processor environment. CommBench is composed of eight programs, four of them oriented towards packet header processing and four oriented towards data stream processing.

CommBench has been successfully used for the evaluation of requirements in network processor systems. A paper with detailed evaluations of a RISC processor running CommBench is available:

Tilman Wolf and Mark A. Franklin, "CommBench - a telecommunications benchmark for network processors," in Proc. of IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), Austin, TX, Apr. 2000, pp. 154-162.

What workload does CommBench model?

The main philosophy behind CommBench was to identify applications that can be used to model a workload for network processors. At this point in time, there is a wide variety of network processors ranging from data link level processors to application layer processing engines. Thus, it was important to capture a broad variety of applications, from simple header processing to more complex applications, like payload encryption. Particular significance was put on the computational kernel of an application, which corresponds to the part of the program contributes to the majority of instructions executed (i.e., the 'inner loop'). While it is clear that different network processor systems exhibit notable differences in how programs are started and initialized, it can also be expected that there is much overlap in how the computational kernel is processed. Thus, results that are obtained from benchmark runs on a given system can be extrapolated to architecturally similar systems even though there might be differences in components that do not affect the computational kernel.

As with all benchmarks, care has to be taken in interpreting the obtained result. If you use CommBench, it is important that you understand for what it is useful and where its limits are:

What can be done with it?

CommBench applications represent various layers of 'network processing'. The workload can be adapted to certain systems by weighing applications differently (i.e. only consider header processing applications for traditional routers).
CommBench can be used to measure basic program execution performance on a given processor (i.e., cache hit rates, instruction mix)
CommBench can be used to estimate the required processing capacity of a network processor system. By measuring the number of instructions/cycles spent per byte of packet data, one can figure out how fast the processor has to be to keep up with the link rate.

What are its limitations?

CommBench does not model a network processor. There is no network processor architecture, nor a concept of packets streaming through the system. 'Packets' are solely reflected by correlating input data to the benchmark measurements
CommBench cannot be 'just compiled' to any arbitrary system. CommBench relies on library functions, like read from standard in, that are provided by a full-blown operating system. So far it has only been compiled and run successfully on Solaris SunOS 5.7.
CommBench does not include any tools for measuring the performance of a system. You have to provide profilers and such by yourself.

How to obtain CommBench?

CommBench License Agreement

Permission to use this software and its documentation is hereby granted for non commercial use only, provided that this copyright notice appears in all copies of the software and any portions thereof and that credit is given to Washington University and the authors in all publications reporting on direct or indirect use of this software.

ALL COMMERCIAL USE OF THIS SOFTWARE IS STRICTLY PROHIBITED.

ALL REDISTRIBUTION OF THIS SOFTWARE TO OTHER PARTIES FOR COMMERCIAL OR NON COMMERCIAL USE IS STRICTLY PROHIBITED.

THIS SOFTWARE IS EXPERIMENTAL AND IS KNOWN TO HAVE BUGS, SOME OF WHICH MAY HAVE SERIOUS CONSEQUENCES. WASHINGTON UNIVERSITY AND THE AUTHORS PROVIDE THIS SOFTWARE IN ITS ``AS IS'' CONDITION, AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL WASHINGTON UNIVERSITY OR THE AUTHORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

If you find this software useful, please consider a contribution to the Computer Communications Research Center, Campus Box 1115, School of Engineering and Applied Sciences at Washington University, One Brookings Dr., St. Louis, MO 63130, USA, to defray development and distribution expenses and to further support this and related research.

This copyright notice DOES NOT include the SSL implementation used in the CommBench CAST application. See the copyright for the SSL package.

This copyright notice DOES NOT include the JPEG implementation used in the CommBench JPEG application. See the copyright for the JPEG package.

This copyright notice DOES NOT include the libpcap packet capture interface and the tcpdump network monitoring application, which was developed by the Network Research Group at Lawrence Berkeley National Laboratory.

This copyright notice DOES NOT include the ZIP compression/decompression application, which is covered under the GNU Public License.

SSL Copyright

Copyright (C) 1997 Eric Young (eay@cryptsoft.com)
All rights reserved.

This package is an SSL implementation written by Eric Young (eay@cryptsoft.com). The implementation was written so as to conform with Netscapes SSL.

This library is free for commercial and non-commercial use as long as the following conditions are aheared to. The following conditions apply to all code found in this distribution, be it the RC4, RSA, lhash, DES, etc., code; not just the SSL code. The SSL documentation included with this distribution is covered by the same copyright terms except that the holder is Tim Hudson (tjh@cryptsoft.com).

Please note that MD2, MD5 and IDEA are publically available standards that contain sample implementations, I have re-coded them in my own way but there is nothing special about those implementations. The DES library is another mater :-).

JPEG Copyright

The authors make NO WARRANTY or representation, either express or implied, with respect to this software, its quality, accuracy, merchantability, or fitness for a particular purpose. This software is provided "AS IS", and you, its user, assume the entire risk as to its quality and accuracy.

This software is copyright (C) 1991-1998, Thomas G. Lane. All Rights Reserved except as specified below.

Permission is hereby granted to use, copy, modify, and distribute this software (or portions thereof) for any purpose, without fee, subject to these conditions:
(1) If any part of the source code for this software is distributed, then this README file must be included, with this copyright and no-warranty notice unaltered; and any additions, deletions, or changes to the original files must be clearly indicated in accompanying documentation.
(2) If only executable code is distributed, then the accompanying documentation must state that "this software is based in part on the work of the Independent JPEG Group".
(3) Permission for use of this software is granted only if the user accepts full responsibility for any undesirable consequences; the authors accept NO LIABILITY for damages of any kind.

These conditions apply to any software derived from or based on the IJG code, not just to the unmodified library. If you use our work, you ought to acknowledge us.

Permission is NOT granted for the use of any IJG author's name or company name in advertising or publicity relating to this software or products derived from it. This software may be referred to only as "the Independent JPEG Group's software".

We specifically permit and encourage the use of this software as the basis of commercial products, provided that all warranty or liability claims are assumed by the product vendor.

ansi2knr.c is included in this distribution by permission of L. Peter Deutsch, sole proprietor of its copyright holder, Aladdin Enterprises of Menlo Park, CA. ansi2knr.c is NOT covered by the above copyright and conditions, but instead by the usual distribution terms of the Free Software Foundation; principally, that you must include source code if you redistribute it. (See the file ansi2knr.c for full details.) However, since ansi2knr.c is not needed as part of any program generated from the IJG code, this does not limit you more than the foregoing paragraphs do.

The Unix configuration script "configure" was produced with GNU Autoconf. It is copyright by the Free Software Foundation but is freely distributable. The same holds for its supporting scripts (config.guess, config.sub, ltconfig, ltmain.sh). Another support script, install-sh, is copyright by M.I.T. but is also freely distributable.

It appears that the arithmetic coding option of the JPEG spec is covered by patents owned by IBM, AT&T, and Mitsubishi. Hence arithmetic coding cannot legally be used without obtaining one or more licenses. For this reason, support for arithmetic coding has been removed from the free JPEG software. (Since arithmetic coding provides only a marginal gain over the unpatented Huffman mode, it is unlikely that very many implementations will support it.) So far as we are aware, there are no patent restrictions on the remaining code.

The IJG distribution formerly included code to read and write GIF files. To avoid entanglement with the Unisys LZW patent, GIF reading support has been removed altogether, and the GIF writer has been simplified to produce "uncompressed GIFs". This technique does not use the LZW algorithm; the resulting GIF files are larger than usual, but are readable by all standard GIF decoders.

We are required to state that "The Graphics Interchange Format(c) is the Copyright property of
CompuServe Incorporated. GIF(sm) is a Service Mark property of CompuServe Incorporated."

Download

By downloading this software you agree to abide by the above license agreement.

Solaris/Linux: CommBench-05.tar.gz (11.3MB)

Errata

It was pointed out that there is an error in frag/data/gen/frag_gen.c. Lines 41 and 49 should be:

   ip.ip_len = sizemin + (random() % (sizemax-sizemin+1));

If you have any questions regarding CommBench, please contact Tilman Wolf.