Benchmark suites


  • Berkeley Multimedia Workload This benchmark suite follows the MediaBench approach of providing full applications with extensive data sets. The BMW focuses on image/video processing and graphics applications and consists of: adpcm, djvu, doom, ghostscript, gsm, jpeg, lame, mesa, mpeg-2, mpg123, povray, rasta, rsynth, and timidity.
  1. Nathan T. Slingerland and Alan Jay Smith, "Design and Characterization of the Berkeley Multimedia Workload," ACM Transactions on Multimedia Systems, Vol. 8, No. 4, pp. 315-327, July 2002.

 

  • DSPstone The main goal of the DSPstone project was the efficiency evaluation of state-of-the-art DSP C compilers. The suite consists of:
  1. application benchmarks: adpcm
  2. DSP-kernel benchmarks: real_updates, n_real_updates, complex_updates, n_complex_updates, dot_product, matrix_1x3, matrix, convolution, fir, fir2d, iir_biquad_1, iir_biquad_n, lms, fft_input_scaled, fft_stage_scaled
  3. HLL-kernels that evaluate the performance of the compiler on standard C constructs. 
  1. V. Zivojnovic, J. M. Velarde, C. Schlager and H. Meyr, "DSPstone: A DSP-oriented benchmarking methodology," Proceedings of the International Conference on Signal Processing Applications and Technology, pp. 715-720, October 1994.

 

  • UTDSP is a DSP oriented suite consisting of applications and kernels. 
  1. application benchmarks: adpcm, compress, edge_detect, g721 (variation 1), g721 (variation b), g722, histogram, jpeg, lpc, spectral, trellis, v32.modem.
  2. DSP kernels (each one has two variations): fft, fir, iir, latnrm, lmsfir, mult.
  1. Mazen A. R. Saghir, Application-Specific Instruction-Set Architectures for Embedded DSP Applications, Ph.D. Thesis, Department of Electrical and Computer Engineering, University of Toronto, Canada, 1998.

 

  • MediaBench Widely used and referenced media-oriented benchmark suite. MediaBench is composed of complete applications originating from the image/video processing, communications and DSP fields. The applications of MediaBench 1.0 are (an updated version, MediaBenchII, is expected to be released soon): jpeg, mpeg-2, gsm, g721, pgp, pegwit, ghostscript, mesa, rasta, epic, adpcm.
  1. Chunho Lee, Miodrag Potkonjak and William H. Mangione Smith, " MediaBench: A Tool for Evaluating and Synthesizing Multimedia and Communications Systems," Proceedings of the 30th IEEE/ACM Symposium on Microarchitecture, pp. 330-335, December 1997.
  2. Jason Fritts, Architecture and Compiler Design Issues in Programmable Media Processors, Ph.D. Thesis, Princeton University, June 2000.
  3. Benjamin Bishop, Thomas P. Kelliher and Mary Jane Irwin, "A Detailed Analysis of MediaBench," Proceedings of the IEEE International Workshop on Signal Processing Systems, pp. 448-455, Taipei, Taiwan, October 1999.

 

  • MiBench is a benchmark suite targeting embedded processing environments. The developers of MiBench specifically had in mind a SimpleScalar/ARM evaluation framework. It can be considered as an open-source counterpart of the EEMBC benchmark suite. The MiBench applications are:
  1. automotive: basicmath, bitcount, qsort, susan (3 variations).
  2. consumer: jpeg, lame, mad, tiff (4 variations), typeset.
  3. office: ghostscript, ispell, rsynth, sphinx, stringsearch.
  4. network: dijkstra, patricia, (crc32, sha, blowfish are regarded as security applications).
  5. security: blowfish encoder, blowfish decoder, pgp sign, pgp verify, rijndael encoder, rijndael decoder, sha.
  6. telecommunications: crc32, fft, ifft, adpcm encoder, adpcm decoder, gsm encoder, gsm decoder.
  1. Matthew R. Guthaus, Jeffrey S. Ringenberg, Dan Ernst, Todd M. Austin, Trevor Mudge and Richard B. Brown, "MiBench: A Free, Commercially Representative Embedded Benchmark Suite," Proceedings of the IEEE 4th Annual Workshop on Workload Characterization, December 2001.

 

  • NetBench consists of 9 applications that are representative of commercial applications for network processors. These can be distinguished into 3 subcategories: micro-level programs, IP-level programs and application-level programs as following:
  1. micro-level programs: crc, tl (table lookup routine).
  2. IP-level programs: route (IPv4 routing), drr (deficit round-robin scheduling), nat (network address translation for IP address simplification and conservation), ipchains (firewall application).
  3. application-level programs: url (URL-based switching), dh (Diffie-Hellman public key encryption decryption mechanism), md5 (message digest algorithm).
  1. Gokhan Memik, William H. Mangione-Smith and Wendong Hu, "NetBench: A Benchmarking Suite for Network Processors".

 

  • SPLASH and SPLASH-2 benchmarks (Stanford Parallel Applications for Shared Memory aim the workload characterization of multiprocessor applications. SPLASH-2 consists of:
  1. applications: 
  • barnes: Barnes-Hut method for N-body simulation
  • fmm: a parallel adaptive Fast Multipole Method for N-bosy simulation
  • ocean (2 variations): simulates large-scale ocean movements based on eddy and boundary currents.
  • radiosity: computes the equilibrium distribution of light in a scene using the sequential hierarchical radiosity method.
  • raytrace: renders a three-dimensional scene onto a two-dimensional image plane using optimized ray tracing
  • volrend: renders a three-dimensional volume onto a two-dimensional image plane using an optimized ray casting technique.
  • water-nsquared:  evaluates forces and potentials that occur over time in a system of water molecules. 
  • water-spatial: a different algorithm for solving the water molecule problem.
  1. kernels: 
  • cholesky: blocked Cholesky Factorization on a sparse matrix. 
  • fft: a complex, one-dimensional version of the "Six-Step" FFT.
  • lu: factors a dense matrix into the product of a lower triangular and an upper triangular matrix.
  • radix: integer radix sort algorithm
  1. Jaswinder Pal Singh, Wolf-Dietrich Weber and Anoop Gupta, "SPLASH: Stanford Parallel Applications for Shared-Memory," In Computer Architecture News, Vol. 20, No. 1, pp. 5-44. 
  2. Steven Cameron Woo, Moriyoshi Ohara, Evan Torrie, Jaswinder Pal Singh and Anoop Gupta,"The SPLASH-2 Programs: Characterization and Methodological Considerations," In Proceedings of the 22nd International Symposium on Computer Architecture, pp. 24-36, Santa Margherita Ligure, Italy, June 1995.

 

  • The RAW benchmark suite has been designed to facilitate benchmarking and comparing reconfigurable computing systems. The benchmarks are provided in the form of behavioral Verilog and the corresponding C versions. intmatmul (integer matrix multiply), jacobi (differential equation solver), life (Conway's game of life), mergesort, nqueens (N-queens problem).
  • bheap: binary heap 
  • bubblesort
  • des: Data Encryption Standard
  • fft
  • graph: semiring graph benchmark
  • intmatmul: integer matrix multiply
  • jacobi: differential equation solver
  • life: Conway's game of life
  • mergesort
  • nqueens: N-queens problem
  1. J. Babb, M. Frank, V. Lee, E. Waingold, M. Taylor, J. Kim, S. Devabhaktuni and A. Agarwal, "The RAW Benchmark Suite: Computation Structures for General Purpose Computing," IEEE Symposium for Custom Computing Machines, Napa Valley, CA, USA, April 1997.

 

  • VersaBench is a benchmark suite for versatile architectures. Its goal is to provide sufficient test cases for measuring a single metric, Versatility, in order to quantify the ability of a versatile architecture to effectively execute a broad set of applications. The VersaBench suite consists of desktop integer, desktop floating point, server, embedded streaming and embedded bit-level applications.
  1. desktop integer
  • mcf: combinatorial optimizer
  • parser: link grammar parser
  • twolf: place and route simulator
  1. desktop floating point
  • bmm: blocked matrix multiplication
  • vpenta: pentadiagonal matrix inverter
  • tomcatv: vectorized mesh generator
  1. server
  • mesa: 3D-gaphics library
  • mgrid: multi-grid solver
  • dbms: database management system
  1. embedded streaming
  • corner: large matrix transposer
  • fm: software FM radio
  • beam: multi-channel beamformer
  1. embedded bit-level
  • 80211a: convolutional encoder from IEEE 802.11a standard
  • 8b10b: IBM 8bit/10bit block encoder
  • des: ECB encoder of the Data Encryption standard
  1. Rodric M. Rabbah, Ian Bratt, Krste Asanovic and Anant Agarwal, "Versatility and VersaBench: A New Metric and a Benchmark Suite for Flexible Architectures," Technical report MIT-LCS-TM-646, Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA 02139, USA, June 2004.

Updated by Nikolaos Kavvadias on June 22, 2013 (first version: 2004)