Friday, November 12, 2010

"Research Must Pursue The Truth Wherever It Leads"

Can't recall whether the hat tip I owe should be directed at Instapundit or Marginal Revolution. Here's British economist John Kay on a practice that really chaps my ass:

The studies I have cited are carefully referenced and use advanced statistical techniques. But sophistication of method is used to torture data to reveal conclusions that do not obviously follow from them, but which fit either the researchers’ preconceptions or the sponsor’s policy objectives, or both.

Bad arguments do not necessarily invalidate the causes in which they are deployed. People should not drink and drive. Smoking is unpleasant and perhaps harmful to non-smokers. But these observations do not justify blurring the distinction between genuine scientific analysis and propaganda disguised as science. Policy should follow evidence, not evidence policy. It is time to reassert the principle that research must pursue the truth wherever it leads: the principle on which the social and economic progress of the past few centuries has depended.

Amen, brother.  Every time I hear another innumerate nitwit dismiss data with the insufferable "lies, damn lies, and statistics" line, I have the urge to declare douchnozzle clobberin' time. But when so many skilled practitioners of research and data analysis torture data in the way Kay describes, one can hardly blame careless observers from dismissing the entire enterprise of collecting data to, you know, learn something true.

Of course, this never really changes. So count this as a reminder to everyone who feels Kay's pain: keep fighting.


  1. The real frustration is when the majority of people believe the spin without ever assessing the data. Education does not guarantee the ability to have critical thought. In many cases the data is so complex it's difficult for any individual outside of that area of expertise to understand it. Even within the group of experts, there can be conflicting conclusions of the same set of data. For these situations the average person rejects the data and relies on pundits, talking heads and other sources that have their own agendas to push. How does a democracy avoid being dumbed-down given this set of conditions?

  2. That's a great question Michael. I think maybe we need a critical thinking tv network. You know, a real one. Part of it's work would involve making fun of shows about ghost and such. Faux research. Torturing data. There's a brand to build there, IMO. Ever see Penn and Teller's Bullshit? Like that.

    And it could also dig into that problem of only the experts getting it by helping them to explain. You know, spend some time on it instead of just repeating the headline.

    I know that I seek out folks on blogs who can help me out in such matters. And that I seek the opinions of folks I disagree with. But that's not a common practice.

    One consoling thought is that more often than not the truth tends to emerge. In the world of science, Cathedrals built on false premises are prone to collapses. Roads built on such grounds end up as dead ends. Or so we can hope, anyway.

    Thanks for stopping by. --Cranky

  3. Preaching to the converted here. Or is that the perverted? Anyay, you know where I stand on such practices. And how long I've been publicly deconstructing them.