Testability Through Audibility

I was working with a client today who complained that there were hidden errors buried in a log file produced by the product he was testing. So, I wrote him a tool that continuously monitors any text file, such as a server log (as long as it is accessible through the file system, as in the case of a test server running locally) and plays WAV files whenever certain string patterns appear in the stream.

With this little tool, a streaming verbose log can be rendered as a stream of clicks and whirrs, if you want, or you can just have it yell “ERROR!” when an error pops up in the log. All this in real time without taking your eyes off the application. Using this, I found a bug in a browser based app whereby perfectly ordinary looking HTML displayed on the screen coincided with a Java null pointer exception in the log.

I released this bit of code with the GPL 2.0 license and you can find it here:

http://www.satisfice.com/tools/log-watch.zip

By the way, this is an example of what I call agile test tooling. I paired with a tester. I heard a complaint. I offered a tool idea. The tester said “yes, please.” I delivered the tool the next day. As we were playing with it, I added a couple of features. I don’t believe you have to be a programmer to be a great tester, but it helps to have a programmer or two on the testing staff. It’s nice work for programmers like me, who get bored with long term production coding.

5 thoughts on “Testability Through Audibility

  1. I’ll keep this in mind for the big test effort we have coming up. We have a small team working on a large problem, and the visual channel will get totally overloaded, just looking at numbers, stats, logs, and specs all day. I look forward to using an oracle that communicates with me in a different mode – I’ll probably listen for bugs at the same time as I look for them. You just found me some free bandwidth, Jim!

    This technique also reminds me of the transistor radio trick that Greg Pope showed us at one of your classes.

  2. That reminds me.

    Currently, I am testing and supporting a system where the end result is ‘recited’ information. This information is customized as per the request of the user and it includes live data and static data. Static data is in the form of audio/wave files.

    Sometimes during the doldrums of listening to repetitive wave files, it is very easy to overlook, I should say ‘overhear’, the playing of an incorrect or incomplete wave file.

    If a tool was available to verify the correct playing or order of the wavefiles, it will be very handy and great but it might not be cost effective to develop. The verification of these wave files is done over a phone line through a regular phone

    The most effective tool that I found is when my forehead hits the table and the mild concussion accompanied by a slight bang sound. This is usually enough of a warning for me to realize that I probably missed an error.

    YMMV
    Eugene Lee

  3. Comments to Geordie and Eugene:

    Geordie: if that “transistor radio trick” is what I think it is, you are now connected by one more gossamer thread to the programmers of the Paleolithic. Congratulations!

    I remember people using an AM portable radio set near the front panel console of an IBM 1620 in the 1960s. And I used that same sort of “news of difference” tool occasionally with a KIM-1 single-board microcomputer, ca. 1976. Homo programmus was adept at hacks like these back when electrons were large enough to see individually as they traveled down the wires. Interesting to see a renaissance of this in the era of the multiple-gigahertz (and even multiple, gigahertz) CPU.

    Eugene: That forehead-drop–>bang effect might be a consequence of your using children’s lullabies as your .wav files. 🙂 But yes, even multiple debug output modalities are not a sure defense. Thanks for reminding us that “MEGO” can also stand for “My EARS Glaze Over”.

    ytf

  4. I am thinking of extending this idea to use “Expect” (mostly as an excuse to learn more about that product: http://expect.nist.gov/ says “Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect really makes this stuff trivial.”

    I think it might be fun to monitor this sort of activity with an audio stream.

    I’ll let you know if I cook up anything useful.

    ytf

  5. Thats an hats off to James 🙂 , Hey guys , yes I had implemented some of similar kind using expect, it works pretty ok… nto a professional one 😉 ..agile testing really does make testing a fun.

Leave a Reply

Your email address will not be published. Required fields are marked *