After learning experimental design, and statistical analysis, oh, and actually doing ‘research’ and publishing, I can pick apart many a journal article, good as any other with my credentials and as well as many with less credential. These days experimental design is taught to school children. Ever judge a science fair?
If experimental design is such common knowledge, why is there so much crap research published and promoted as science?
Where the p really meets the bell curve is in behavioral and social science (an oxymoron?). Behavioral and social researchers really try to be like researchers in the hard sciences. They really try, but translating experimental design from a Petri-dish-controlled-environment to a group of humans is, well, unscientific.
Great effort goes into an ‘experimental’ design where outcomes are behaviors under specified circumstances. Operational definitions of behavior are necessarily laborious. Efforts to create controlled circumstances and environments (medium) require pages of methodical description. Not to mention the incredible variability within each person.
Groups of people are not homogeneous even if they are the same race, the same marital status, the same age, live in the same geographic location, same socioeconomic class, and the same family! Well, maybe if they are in the same family – but even there the variance is wide, and you can rarely get them to be the same age – except in twin studies, which reinforces the heavy effects of genetics.
Genetics – now that’s scientific – studied in a lab and unsubjective and unchangeable data on each and every person.
Other than providing tissue samples and images of internal body parts, people make poor Petri dishes. Behavior, seated in the less-than-fully-understood-brain, is the result of multiple, complex and in-the-moment changes that are difficult to correlate, much less show cause and effect or prediction.
How big of a sample does one need to characterize Americans on eating habits, smoking, drinking alcohol, substance use, sexual behavior or parenting? Nary a research report includes conclusive statements for resolving one the big questions in human existence. The cliché research report outcome – more research needs to be done on this topic.
So let’s say you have some pretty good data – a big number of data-points of a discreetly defined behavior gathered by data-collectors who are blind to the point of the study (and therefore objective) and perfect in gathering the data in a reliable way. The data is appropriately crunched and the completely unbiased researchers interpret the data to say….. write and submit a report to a journal where reviewers consider the paper un-vacuously and after recommended edits, recommend the article for publication.
Late in the game, some reporter, journalist or media-type picks-up the conclusion section on the research report, paraphrases portions considered interesting, and voila! Headline reads: Teens Having More Sex. Puleease!
Good and bad science is bought and sold in the media market.
Much-abused is survey research, perhaps effective in commerce, but even focus group responses are only meaningful if correlated with purchases (discreetly defined behavior).
Survey, the bane of all research and the majority of all social research, is subjective at its core and entirely weak for predicting behavior.
Not buying my message? Are you a glass-half-full person or a glass-half-empty person? Depending on which type of person interprets the data....depending on the life-view of the reporter of the research...that is the essence of subjectivity.
Comments