Software Testing Tools Blog – Testertools

HATE is a test harness, a program that is intended to take the tedium out of testing and evaluating programs.

Some effort has been taken to ensure that HATE is totally independent of both the program being tested and the series of tests to be applied to the test program.  The intention is to provide a single tool that may be used for both testing and evaluation; to encourage well-designed objective testing; and, by allowing researchers to apply the same sets of tests to their own programs, to make it easy for researchers to compare techniques on an informed basis.

The major features of HATE are:

HATE is written in a high-level scripting language and may be used unchanged on any platform that the language supports.  This includes all flavours of Unix; Linux; Windows NT, 95, 3. n ; and the Macintosh.

Test scripts may be used unchanged on all platforms.

Test scripts are independent of the software being tested or evaluated.

Programs are interfaced to HATE by means of a short interface procedure ; thereafter, all relevant existing test scripts may be used with them.

The decomposition into test script and interface procedure, and the portablity of HATE, encourage the sharing of scripts between developers or researchers.

HATE is able to generate its output in a range of forms, including HTML and LaTeX tables.

HATE was developed principally to aid the evaluation of computer vision algorithms . There is increasing realization within the vision research community that this type of work is essential in converting computer vision from a “black art“, with algorithms being tested on only a few images, to sound engineering practice. It is hoped that the vision community will develop test scripts that allow comprehensive comparisons of existing algorithms to be performed, and that new algorithms are subsequently compared with the existing body of results.

View TesterTools dedicated page for this tool.

Bonnie is a benchmark which measures the performance of Unix file system operations. Bonnie is concerned with identifying bottlenecks; the name is a tribute to Bonnie Raitt, who knows how to use one.Bonnie is not the biggest nor most comprehensive of such programs, but is unequaled (though I say so myself) in its combination of ease-of-use and usefulness-of-output.

Why Bonnie?

I believe that:

* memory is in short supply, so caches max out, thus
* many I/O operations end up really doing I/O, thus
* it’s worthwhile to try to measure real I/O speeds, and
* random seeks on Unix filesystems are appallingly slow.

What Bonnie Does

Bonnie performs a series of tests on a file of known size. If the size is not specified, Bonnie uses 100 Mb; but that probably isn’t enough for a big modern server – you your file to be a lot bigger than the available RAM

Bonnie works with 64-bit pointers if you have them.

For each test, Bonnie reports the bytes processed per elapsed second, per CPU second, and the % CPU usage (user and system).

In each case, an attempt is made to keep optimizers from noticing it’s all bogus. The idea is to make sure that these are real transfers between user space and the physical disk. The tests are:
1. Sequential Output

1.1 Per-Character

The file is written using the putc() stdio macro. The loop that does the writing should be small enough to fit into any reasonable I-cache. The CPU overhead here is that required to do the stdio code plus the OS file space allocation.

1.2 Block

The file is created using write(2). The CPU overhead should be just the OS file space allocation.

1.3 Rewrite

Each Chunk (currently, the size is 16384) of the file is read with read(2), dirtied, and rewritten with write(2), requiring an lseek(2). Since no space allocation is done, and the I/O is well-localized, this should test the effectiveness of the filesystem cache and the speed of data transfer.

2. Sequential Input

2.1 Per-Character

The file is read using the getc() stdio macro. Once again, the inner loop is small. This should exercise only stdio and sequential input.

2.2 Block

The file is read using read(2). This should be a very pure test of sequential input performance.

3. Random Seeks

This test runs SeekProcCount (currently 4) processes in parallel, doing a total of 4000 lseek()s to locations in the file computed using by random() in bsd systems, drand48() on sysV systems. In each case, the block is read with read(2). In 10% of cases, it is dirtied and written back with write(2).

The idea behind the SeekProcCount processes is to make sure there’s always a seek queued up.

AXIOM: For any unix filesystem, the effective number of lseek(2) calls per second declines asymptotically to near 30, once the effect of caching is defeated. [ I wrote the previous sentence in about 1988, and it's a bit better now, but not much ]

The size of the file has a strong nonlinear effect on the results of this test. Many Unix systems that have the memory available will make aggressive efforts to cache the whole thing, and report random I/O rates in the thousands per second, which is ridiculous. As an extreme example, an IBM RISC 6000 with 64 Mb of memory reported 3,722 per second on a 50 Mb file. Some have argued that bypassing the cache is artificial since the cache is just doing what it’s designed to. True, but in any application that requires rapid random access to file(s) significantly larger than main memory which is running on a system which is doing significant other work, the caches will inevitably max out.

For more information on this tool click here

View TesterTools dedicated page for this tool.