Even a chimp can write code

Friday, October 22, 2004

Why you should take statistics with a pinch of salt

There are lies, damned lies and statistics. Earlier this week, I was changing the implementation of a frequently called method which used a simple boolean flag to denote status for collection of objects. My aim was to add status indicators at the lowest level of granularity. This would mean the evaluation mechanism would have to iterate over this collection to provide status at a global level. My chief concern with the design was that it may introduce an overhead the transaction -- of which our method in question is a part -- could not sustain. I then proceeded to write a simple test to time the method.

I occasionally use the arithmetic mean approach to timing methods. Its quite simple and goes something like this:


long startTime = System.currentTimeMillis();
for (int i=0; i<1000; i++) {
methodUnderTest();
}
System.out.println("Avg. method execution time=" +
((System.currentTimeMillis() - startTime)/1000));


Now this may not be the best approach, but it works for me. I had assumed that if you used a large enough sample, you could get pretty accurate results. Except, this time, while I was demonstrating the result to Bill Hubertus, a disturbing though struck me. Statistical approaches such as this could so easily be misused by people, if they so desired. This was hardly an epiphany though. In an instant I recalled all those J2EE v/s .NET showdowns and imagined crooked old nerds concocting their slimy recipes to skew the results one way or the other. But no, this action hero is not out to rid the earth of its scum. So back to our story...

In fact the arithmetic mean approach itself is awfully flawed. For those that took a photography class in place of statistics, here's a refresher: the arithmetic mean is the standard average, in that you take a sum of all the numbers in a sequence and divide them by the number of items in that sequence. If you have a sequence of {1, 2, 3, 4, 5}, the mean is 3. Looks good, eh? But if you calculated the net worth of say, Bill Gates, Warren Buffet and Ashish Shetty, you'd be misled into believing all three were members of the Forbes 500 club. For the record, it would be money lost for Mr Gates and Mr Buffet if they even bent down to pick up a bag of cash equivalent to my annual paycheck from the pavement. Using the arithmetic mean then skews the results higher than they actually are. Which itself is not a problem for my test above. All it does it show a higher execution time; not quite as evil but misleading nevertheless.

The median is another way to look at statistics, perhaps a better one. It is the number that is exactly midway in a sequence of numbers. It is the value below which 50% of the scores fall (and need I add, the value above which 50% of the scores fall. ). When there is an even number of scores, the median is the mean of the two centermost scores.

And then there is the mode. It is a measure of frequency of occurence. So the most frequent occuring number in a sequence is the mode. I do not know if this is practical in timing methods.

So the next time you see a report throwing statistical inswinging yorkers your way, think of this: 98% of all heroin addicts started out by drinking milk. Go figure! And my personal favorite (I use this to discredit most claims): 47.5% of all statistics are made on the spot.

If you have any pet tricks for measuring method execution times reliably, please comment. But keep them honest.

Email this | Bookmark this

Thursday, October 21, 2004

Testers: Are they Vegetable or Mineral?

This incisive essay is a must read for testers everywhere so they don't annoy their developer colleagues to the point of homicide.

Here's a checklist of undesirable behavior exhibited by testers, as compiled by Mr. Ed [www.hacknot.info]:

  • Abbreviating Instructions for Reproducing the Bug


  • Not Identifying the Erroneous Behavior


  • Not Identifying the Expected Behavior


  • Not Justifying the Expected Behavior


  • Re-opening Old Bug Reports for New Bugs with Similar Symptoms


  • Testing an Old Version of the Software


  • Inventing Requirements Based Upon Personal Preference


  • Omitting Screen Shots


  • Using Vague or Ambiguous Wording


  • Diagnosing Instead of Reporting


  • Exaggerating the Priority of a Bug


  • Justifying Partial Coverage with Appeals to Bad Assumptions



Testers of the world, mend your ways and we just might get along.

Email this | Bookmark this

Thursday, October 14, 2004

Who wouldn't love the Google Desktop Search?

While it silently indexes my emails and documents in the background, I figured I'd take the time to say "Thank you!" to the Google team for this excellent contraption. I love it!

Email this | Bookmark this

Sunday, October 03, 2004

How to determine the calling class in Java

I once tried a dirty hack that involved parsing the stack trace of an exception instance, but this would return the calling class name much efficiently:




String callingClassName =
sun.reflect.Reflection.getCallerClass(2).getName();



The getCallerClass method takes an int argument where a value of 0 (zero) returns sun.reflect.Reflection, 1 returns the class of the currently executing method and every increment looks up the call stack.

If you are bothered by the prospect of using a sun.* API, you may (write a utility class to) sub-class the SecurityManager and use its protected native Class[] getClassContext() method which returns the call stack. The element at index 0 is the class of the currently executing method, the element at 1 is the caller and so on up the stack.

Email this | Bookmark this