{"id":38322,"date":"2017-04-30T03:00:43","date_gmt":"2017-04-30T07:00:43","guid":{"rendered":"http:\/\/quotulatiousness.ca\/blog\/?p=38322"},"modified":"2020-03-04T21:08:40","modified_gmt":"2020-03-05T02:08:40","slug":"p-hacking-is-one-of-the-many-questionable-research-practices-responsible-for-the-replication-crisis-in-the-social-sciences","status":"publish","type":"post","link":"https:\/\/quotulatiousness.ca\/blog\/2017\/04\/30\/p-hacking-is-one-of-the-many-questionable-research-practices-responsible-for-the-replication-crisis-in-the-social-sciences\/","title":{"rendered":"[p-hacking] “is one of the many questionable research practices responsible for the replication crisis in the social sciences”"},"content":{"rendered":"

What happens when someone digs into the statistics of highly influential health studies and discovers oddities? We’re in the process of finding out in the case of “rockstar researcher” Brian Wansink<\/a> and several of his studies under the statistical microscope:<\/p>\n

Things began to go bad late last year when Wansink posted some advice for grad students on his blog. The post, which has subsequently been removed (although a cached copy is available<\/a>), described a grad student who, on Wansink\u2019s instruction, had delved into a data set to look for interesting results. The data came from a study that had sold people coupons for an all-you-can-eat buffet. One group had paid $4 for the coupon, and the other group had paid $8.<\/p>\n

The hypothesis had been that people would eat more if they had paid more, but the study had not found that result. That\u2019s not necessarily a bad thing. In fact, publishing null results like these is important \u2014 failure to do so leads to publication bias<\/a>, which can lead to a skewed public record that shows (for example) three successful tests of a hypothesis but not the 18 failed ones. But instead of publishing the null result, Wansink wanted to get something more out of the data.<\/p>\n

\u201cWhen [the grad student] arrived,\u201d Wansink wrote, \u201cI gave her a data set of a self-funded, failed study which had null results… I said, \u2018This cost us a lot of time and our own money to collect. There\u2019s got to be something here we can salvage because it\u2019s a cool (rich & unique) data set.\u2019 I had three ideas for potential Plan B, C, & D directions (since Plan A had failed).\u201d<\/p>\n

The responses to Wansink\u2019s blog post from other researchers were incredulous, because this kind of data analysis is considered an incredibly bad idea. As this very famous xkcd strip<\/a> explains, trawling through data, running lots of statistical tests, and looking only for significant results is bound to turn up some false positives. This practice of \u201cp-hacking<\/a>\u201d \u2014 hunting for significant p-values in statistical analyses \u2014 is one of the many questionable research practices responsible for the replication crisis in the social sciences.<\/p><\/blockquote>\n

H\/T to Kate at Small Dead Animals<\/em><\/a> for the link.<\/p>\n","protected":false},"excerpt":{"rendered":"

What happens when someone digs into the statistics of highly influential health studies and discovers oddities? We’re in the process of finding out in the case of “rockstar researcher” Brian Wansink and several of his studies under the statistical microscope: Things began to go bad late last year when Wansink posted some advice for grad […]<\/p>\n","protected":false},"author":1,"featured_media":35193,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[66,28,16],"tags":[39,1361,1233,513,290],"class_list":["post-38322","post","type-post","status-publish","format-standard","hentry","category-health-science","category-media","category-science","tag-junkscience","tag-p-hacking","tag-replicationcrisis","tag-research","tag-statistics"],"jetpack_featured_media_url":"https:\/\/quotulatiousness.ca\/blog\/wp-content\/uploads\/2016\/06\/favicon.png","jetpack_shortlink":"https:\/\/wp.me\/p2hpV6-9Y6","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/quotulatiousness.ca\/blog\/wp-json\/wp\/v2\/posts\/38322","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/quotulatiousness.ca\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/quotulatiousness.ca\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/quotulatiousness.ca\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/quotulatiousness.ca\/blog\/wp-json\/wp\/v2\/comments?post=38322"}],"version-history":[{"count":2,"href":"https:\/\/quotulatiousness.ca\/blog\/wp-json\/wp\/v2\/posts\/38322\/revisions"}],"predecessor-version":[{"id":55467,"href":"https:\/\/quotulatiousness.ca\/blog\/wp-json\/wp\/v2\/posts\/38322\/revisions\/55467"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/quotulatiousness.ca\/blog\/wp-json\/wp\/v2\/media\/35193"}],"wp:attachment":[{"href":"https:\/\/quotulatiousness.ca\/blog\/wp-json\/wp\/v2\/media?parent=38322"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/quotulatiousness.ca\/blog\/wp-json\/wp\/v2\/categories?post=38322"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/quotulatiousness.ca\/blog\/wp-json\/wp\/v2\/tags?post=38322"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}