Even if you’re not a statistician, you may one day find yourself in the position of reviewing
astatistical analysis that was done by someone else. It may be an associate, someone who works
for you, or even a competitor. Don’t panic. Critiquing someone else’s work
has got to be one of the easiest jobs in the world. After all, your boss does it all the time(http://statswithcats.wordpress.com/2010/11/14/you-can-lead-a-boss-to-data-but-you-can%e2%80%99t-make-him-think/). Doing it in a constructive manner is another story.
Don’t expect to find a major flaw in a multivariate
analysis of variance, or a neural network, or afactor analysis. Look for the simple and fundamental
errors of logic and performance. It’s
probably what you are best suited for and will be most usefulto the report writer who can no longer see the statistical forestthrough the numerical trees.
So here’s the deal. I’ll give you some bulletproof leads onwhat criticisms to level on that statistical report you’re
reading. In exchange, you must promise to be gracious,forgiving, understanding, and, above all, constructive in your
remarks. If you don’t, you will be forever cursed to receive
the same manner of comments that you dish out.With that said, here are some things to look for.
The Red-Face Test
Start with an overall look at the calculations and findings. Not infrequently, there is a glaringerror that is invisible to all the poor folks who have been living with the analysis 24/7 for the lastseveral months. The error is usually simple, obvious once detected, very embarrassing, andenough to send them back to their computers. Look for:
Wrong number of samples
Either samples were unintentionally omitted or replicateswere included when the
y shouldn’t have been.
Calculated means look too high or low, sometimes by a lot. Thecause may be a mistaken data entry, an incorrect calculation, or an untreated outlier.
A stated conclusion seems counterintuitive or unlikely givenknown conditions. This may be caused by a lost sign on a correlation or regressioncoefficient, a misinterpreted test probability, or an inappropriate statistical design oranalysis.
Nobody Expects the Sample Inquisition
Start with the samples. If you can cast doubt on the representativeness of the samples, everything
else done after that doesn’t matter. If you are reviewing a product from a mathematically
probably the only place to look for difficulties is in the samples. There are a fewreasons for this. First, a statistician may not be familiar with some of the technical complexitiesof sampling the medium or population being investigated. Second, he or she may have beenhanded the dataset with little or no explanation of the methods used to generate the data. Third,he or she will probably get everything else right. Focus on what the data analyst knows the leastabout.
I feel like I'm being watched.