
Originally Posted by
Beyond
Just a comment on this, since I've done some work on measurement theory. Doing a measurement "in a rigorous way" is harder than you may think. It isn't just "being careful," and contrary to popular belief, using precision instruments doesn't mean measurements are worth a damn. All measurements, from a carpenter's guestimate of a straight cut to a surveyor's laser, have error. Both human and instrumental.
More formally, a measurement is made of a "true" measurement that points at a real physical referent, and error. Error that points at everything from unconsciously moving your hand a fraction of a mm, to misreading the number on the dial, to precision instruments fluctuating as temperature, humidity, or voltage changes. Then there's bias: When you file an edge, do you make the same tiny error the same way each time (systematic), or do you mess up a new way each time (random)? Lot of noise to account for.
So when you publish, you specify your reliability. Eg, how much and what kind of error do your measurements or treatments have?
What kind means accuracy (how close you are to a "gold standard") and consistency (how well you can reproduce the same measurement or outcome over and over), plus the systematic vs random issue. There are standard test-retest methods for getting numbers on all this and then doing a statistical analysis. Let's say that this fellow did all his measurements himself, so no need to assess interobserver error (having different people working on the same measurements). But unless he does reliability tests on his own measurements, and for each of his various base treatments, and for his machines, if they're non-standard, his numbers are hard to interpret.
Well-known research teams, with proven track records and earlier statements of reliability, can get away with just talking about the specific instruments and references to techniques. But newbies have to earn that right by really spelling it out.
You get the idea. Don't worship "careful measurements." They may be carefully wrong. And "superior measurements" may involve finer increments (say a laser instead of a ruler), but still be just as wrong. There is no default position in all this. No one in science takes outcomes seriously unless we know something about the measurements' reliability and applicability. That's why papers have such long "Methods and Materials" sections.
Bookmarks