Quotulatiousness

February 7, 2024

A disturbing proportion of scientific publishing is … bullshit

Filed under: Economics, Media, Science — Tags: , , , , , , , — Nicholas @ 04:00

Tim Worstall on a few of the more upsetting details of how much we’ve been able depend on truth and testability in the scientific community and how badly that’s been undermined in recent years:

The Observer tells us that science itself is becoming polluted by journal mills. Fools — intellectual thieves perhaps — are publishing nonsense in scientific journals, this then pollutes the conclusions reached by people surveying science to see what’s what.

This is true and is a problem. But it’s what people publish as supposedly real science that is the real problem here, not just those obvious cases they’re complaining about:

    The startling rise in the publication of sham science papers has its roots in China, where young doctors and scientists seeking promotion were required to have published scientific papers. Shadow organisations – known as “paper mills” – began to supply fabricated work for publication in journals there.

    The practice has since spread to India, Iran, Russia, former Soviet Union states and eastern Europe, with paper mills supplying ­fabricated studies to more and more journals as increasing numbers of young ­scientists try to boost their careers by claiming false research experience. In some cases, journal editors have been bribed to accept articles, while paper mills have managed to establish their own agents as guest editors who then allow reams of ­falsified work to be published.

Indeed, an actual and real problem:

    The products of paper mills often look like regular articles but are based on templates in which names of genes or diseases are slotted in at random among fictitious tables and figures. Worryingly, these articles can then get incorporated into large databases used by those working on drug discovery.

    Others are more bizarre and include research unrelated to a journal’s field, making it clear that no peer review has taken place in relation to that article. An example is a paper on Marxist ideology that appeared in the journal Computational and Mathematical Methods in Medicine. Others are distinctive because of the strange language they use, including references to “bosom peril” rather than breast cancer and “Parkinson’s ailment” rather Parkinson’s disease.

Quite. But the problem is worse, much, much, worse.

Let us turn to something we all can agree is of some importance. Those critical minerals things. We all agree that we’re going to be using more of them in the future. Largely because the whole renewables thing is changing the minerals we use to power the world. We’re — to some extent, perhaps enough, perhaps not enough — moving from using fossil fuels to power the world to using rare earths, silicon, copper and so on to power the world. How much there is, how much useable, of those minerals is important. Because that’s what we’re doing, we’re changing which minerals — from fossil to metallic elements — we use to power the world.

Those estimates of how much there is out there are therefore important. The European Union, for example, has innumerable reports and task forces working on the basis that there’s not that much out there and therefore we’ve got to recycle everything. One of those foundational blocks of the circular economy is that we’ve got to do it anyway. Because there’s simply not enough to be able to power society without the circular economy.

This argument is nads*. The circular economy might be desirable for other reasons. At least in part it’s very sensible too – if it’s cheaper to recycle than to dig up new then of course we should recycle. But that we must recycle, regardless of the cost, because otherwise supply will vanish is that nads*.

But, folk will and do say, if we look at the actual science here we are short of these minerals and metals. Therefore etc. But it’s the science that has become infected. Wrongly infected, infested even.

Here’s the Royal Society of Chemistry and their periodic table. You need to click around a bit to see this but they have hafnium supply risk as “unknown”. That’s at least an advance from their previous insistence that it was at high supply risk. It isn’t, there’s more hafnium out there than we can shake a stick at. At current consumption rates — and assuming no recycling at all which, with hafnium, isn’t all that odd an idea — we’re going to run out sometime around the expected date for the heat death of the universe. No, not run out of the universe’s hafnium, run out of what we’ve got in the lithosphere of our own Earth. To a reasonable and rough measure the entirety of Cornwall is 0.01% hafnium. We happily mine for gold at 0.0001% concentrations and we use less hafnium annually than we do gold.

The RSC also says that gallium and germanium have a high supply risk. Can you guess which bodily part(s) such a claim should be associated with? For gallium we already have a thousand year supply booked to pass through the plants we normally use to extract our gallium for us. For germanium I — although someone competent could be a preference — could build you a plant to supply 2 to 4% of global annual germanium demand/supply. Take about 6 months and cost $10 million even at government contracting rates to do it too. The raw material would be fly ash from coal burning and there’s no shortage of that — hundreds of such plants could be constructed that is.

The idea that humanity is, in anything like the likely timespan of our species, going to run short in absolute terms of Hf, Ga or Ge is just the utmost nads*

But the American Chemistry Society says the same thing:

    * As ever, we are polite around here. Therefore we use the English euphemism “nads”, a shortening of “nadgers”, for the real meaning of “bollocks”.

January 18, 2024

QotD: Scientific fraud

Filed under: Quotations, Science — Tags: , , , — Nicholas @ 01:00

In sorting out these feelings, I start from the datum that scientific fraud feels to me like sacrilege. Plausible reports of it make me feel deeply angry and disgusted, with a stronger sense of moral indignation than I get about almost any other sort of misbehavior. I feel like people who commit it have violated a sacred trust.

What is sacred here? What are they profaning?

The answer to that question seemed obvious to me immediately when I first formed the question. But in order to explain it comprehensibly to a reader, I need to establish what I actually mean by “science”. Science is not a set of answers, it’s a way of asking questions. It’s a process of continual self-correction in which we form theories about what is, check them by experiment, and use the result to improve our theories. Implicitly there is no end to this journey; anything we think of as “truth” is merely a theory that has had predictive utility so far but could be be falsified at any moment by further evidence.

When I ask myself why I feel scientific fraud is like sacrilege, I rediscover on the level of emotion something I have written from an intellectual angle: Sanity is the process by which you continually adjust your beliefs so they are predictively sound. I could have written “scientific method” rather than “sanity” there, and that is sort of the point. Scientific method is sanity writ large and systematized; sanity is science in the small personal domain of one’s own skull.

Science is sanity is salvation – it’s how we redeem ourselves, individually and collectively, from the state of ignorance and sin into which we were born. “Sin” here has a special interpretation; it’s the whole pile of cognitive biases, instinctive mis-beliefs, and false cultural baggage we’re wired with that obstruct and weigh down our attempts to be rational. But my emotional reaction to this is, I realize, quite like that of a religious person’s reaction to whatever tribal superstitious definition of “sin” he has internalized.

I feel that scientists have a special duty of sanity that is analogous to a priest’s special duty to be pious and virtuous. They are supposed to lead us out of epistemic sin, set the example, light the way forward. When one of them betrays that trust, it is worse than ordinary stupidity. It damages all of us; it feeds the besetting demons of ignorance and sloppy thinking, and casts discredit on scientists who have remained true to their sacred vocation.

Eric S. Raymond, “Maybe science is my religion, after all”, Armed and Dangerous, 2011-05-18.

January 12, 2024

“… normal people no longer trust experts to any great degree”

Filed under: Education, Media, Politics, Science, USA — Tags: , , , , , , — Nicholas @ 05:00

Theophilus Chilton explains why the imprimatur of “the experts” is a rapidly diminishing value:

One explanation for the rise of midwittery and academic mediocrity in America directly connects to the “everybody should go to college” mantra that has become a common platitude. During the period of America’s rise to world superpower, going to college was reserved for a small minority of higher IQ Americans who attended under a relatively meritocratic regime. The quality of these graduates, however, was quite high and these were the “White men with slide rules” who built Hoover Dam, put a man on the moon, and could keep Boeing passenger jets from losing their doors halfway through a flight. As the bar has been lowered and the ranks of Gender and Queer Studies programs have been filled, the quality of college students has declined precipitously. One recent study shows that the average IQ of college students has dropped to the point where it is basically on par with the average for the general population as a whole.

Another area where this comes into play is with the replication crisis in science. For those who haven’t heard, the results from an increasingly large number of scientific studies, including many that have been used to have a direct impact on our lives, cannot be reproduced by other workers in the relevant fields. Obviously, this is a problem because being able to replicate other scientists’ results is sort of central to that whole scientific method thing. If you can’t do this, then your results really aren’t any more “scientific” than your Aunt Gertie’s internet searches.

As with other areas of increasing sociotechnical incompetency, some of this is diversity-driven. But not wholly so, by any means. Indeed, I’d say that most of it is due to the simple fact that bad science will always be unable to be consistently replicated. Much of this is because of bad experimental design and other technical matters like that. The rest is due to bad experimental design, etc., caused by overarching ideological drivers that operate on flawed assumptions that create bad experimentation and which lead to things like cherry-picking data to give results that the scientists (or, more often, those funding them) want to publish. After all, “science” carries a lot of moral and intellectual authority in the modern world, and that authority is what is really being purchased.

It’s no secret that Big Science is agenda-driven and definitely reflects Regime priorities. So whenever you see “New study shows the genetic origins of homosexuality” or “Latest data indicates trooning your kid improves their health by 768%,” that’s what is going on. REAL science is not on display. And don’t even get started on global warming, with its preselected, computer-generated “data” sets that have little reflection on actual, observable natural phenomena.

“Butbutbutbut this is all peer-reviewed!! 97% of scientists agree!!” The latter assertion is usually dubious, at best. The former, on the other hand, is irrelevant. Peer-reviewing has stopped being a useful measure for weeding out spurious theories and results and is now merely a way to put a Regime stamp of approval on desired result. But that’s okay because the “we love science” crowd flat out ignores data that contradict their presuppositions anywise, meaning they go from doing science to doing ideology (e.g. rejecting human biodiversity, etc.). This sort of thing was what drove the idiotic responses to COVID-19 a few years ago, and is what is still inducing midwits to gum up the works with outdated “science” that they’re not smart enough to move past.

If you want a succinct definition of “scientism,” it might be this – A belief system in which science is accorded intellectual abilities far beyond what the scientific method is capable of by people whose intellectual abilities are far below being able to understand what the scientific method even is.

September 23, 2023

More on the history field’s “reproducibility crisis”

Filed under: Books, Britain, History, USA — Tags: , , , , , — Nicholas @ 04:00

In the most recent edition of the Age of Invention newsletter, Anton Howes follows up on his earlier post about the history field’s efforts to track down and debunk fake history:

The concern I expressed in the piece is that the field of history doesn’t self-correct quickly enough. Historical myths and false facts can persist for decades, and even when busted they have a habit of surviving. The response from some historians was that they thought I was exaggerating the problem, at least when it came to scholarly history. I wrote that I had not heard of papers being retracted in history, but was informed of a few such cases, including even a peer-reviewed book being dropped by its publisher.

In 2001/2, University of North Carolina Press decided to stop publishing the 1999 book Designs against Charleston: The Trial Record of the Denmark Vesey Slave Conspiracy of 1822 when a paper was published showing hundreds of cases where its editor had either omitted or introduced words to the transcript of the trial. The critic also came to very different conclusions about the conspiracy. In this case, the editor did admit to “unrelenting carelessness“, but maintained that his interpretation of the evidence was still correct. Many other historians agreed, thinking the critique had gone too far and thrown “the baby out with the bath water“.

In another case, the 2000 book Arming America: The Origins of a National Gun Culture — not peer-reviewed, but which won an academic prize — had its prize revoked when found to contain major errors and potential fabrications. This is perhaps the most extreme case I’ve seen, in that the author ultimately resigned from his professorship at Emory University (that same author believes that if it had happened today, now that we’re more used to the dynamics of the internet, things would have gone differently).

It’s somewhat comforting to learn that retraction in history does occasionally happen. And although I complained that scholars today are rarely as delightfully acerbic as they had been in the 1960s and 70s in openly criticising one another, they can still be very forthright. Take James D. Perry in 2020 in the Journal of Strategy and Politics reviewing Nigel Hamilton’s acclaimed trilogy FDR at War. All three of Perry’s reviews are critical, but that of the second book especially forthright, including a test of the book’s reproducibility:

    This work contains numerous examples of poor scholarship. Hamilton repeatedly misrepresents his sources. He fails to quote sources fully, leaving out words that entirely change the meaning of the quoted sentence. He quotes selectively, including sentences from his sources that support his case but ignoring other important sentences that contradict his case. He brackets his own conjectures between quotes from his sources, leaving the false impression that the source supports his conjectures. He invents conversations and emotional reactions for the historical figures in the book. Finally, he fails to provide any source at all for some of his major arguments

Blimey.

But I think there’s still a problem here of scale. It’s hard to tell if these cases are signs that history on the whole is successfully self-correcting quickly, or are stand-out exceptions. I was positively inundated with other messages — many from amateur historical investigators, but also a fair few academic historians — sharing their own examples of mistakes that had snuck past the careful scholars for decades, or of other zombies that refused to stay dead.

August 31, 2023

The sciences have replication problems … historians face similar issues

Filed under: Britain, History, Media, Politics — Tags: , , , , , — Nicholas @ 03:00

In the latest Age of Invention newsletter, Anton Howes considers the history profession’s closest equivalent to the ongoing replication crisis in the sciences:

… I’ve become increasingly worried that science’s replication crises might pale in comparison to what happens all the time in history, which is not just a replication crisis but a reproducibility crisis. Replication is when you can repeat an experiment with new data or new materials and get the same result. Reproducibility is when you use exactly the same evidence as another person and still get the same result — so it has a much, much lower bar for success, which is what makes the lack of it in history all the more worrying.

Historical myths, often based on mere misunderstanding, but occasionally on bias or fraud, spread like wildfire. People just love to share unusual and interesting facts, and history is replete with things that are both unusual and true. So much that is surprising or shocking has happened, that it can take only years or decades of familiarity with a particular niche of history in order to smell a rat. Not only do myths spread rapidly, but they survive — far longer, I suspect, than in scientific fields.

Take the oft-repeated idea that more troops were sent to quash the Luddites in 1812 than to fight Napoleon in the Peninsular War in 1808. Utter nonsense, as I set out in 2017, though it has been cited again and again and again as fact ever since Eric Hobsbawm first misled everyone back in 1964. Before me, only a handful of niche military history experts seem to have noticed and were largely ignored. Despite being busted, it continues to spread. Terry Deary (of Horrible Histories fame), to give just one of many recent examples, repeated the myth in a 2020 book. Historical myths are especially zombie-like. Even when disproven, they just. won’t. die.

[…]

I don’t think this is just me being grumpy and pedantic. I come across examples of mistakes being made and then spreading almost daily. It is utterly pervasive. Last week when chatting to my friend Saloni Dattani, who has lately been writing a piece on the story of the malaria vaccine, I shared my mounting paranoia healthy scepticism of secondary sources and suggested she check up on a few of the references she’d cited just to see. A few days later and Saloni was horrified. When she actually looked closely, many of the neat little anecdotes she’d cited in her draft — like Louis Pasteur viewing some samples under a microscope and having his mind changed on the nature of malaria — turned out to have no actual underlying primary source from the time. It may as well have been fiction. And there was inaccuracy after inaccuracy, often inexplicable: one history of the World Health Organisation’s malaria eradication programme said it had been planned to take 5-7 years, but the sources actually said 10-15; a graph showed cholera pandemics as having killed a million people, with no citation, while the main sources on the topic actually suggest that in 1865-1947 it killed some 12 million people in British India alone.

Now, it’s shockingly easy to make these mistakes — something I still do embarrassingly often, despite being constantly worried about it. When you write a lot, you’re bound to make some errors. You have to pray they’re small ones and try to correct them as swiftly as you can. I’m extremely grateful to the handful of subject-matter experts who will go out of their way to point them out to me. But the sheer pervasiveness of errors also allows unintentionally biased narratives to get repeated and become embedded as certainty, and even perhaps gives cover to people who purposefully make stuff up.

If the lack of replication or reproducibility is a problem in science, in history nobody even thinks about it in such terms. I don’t think I’ve ever heard of anyone systematically looking at the same sources as another historian and seeing if they’d reach the same conclusions. Nor can I think of a history paper ever being retracted or corrected, as they can be in science. At the most, a history journal might host a back-and-forth debate — sometimes delightfully acerbic — for the closely interested to follow. In the 1960s you could find an agricultural historian saying of another that he was “of course entitled to express his views, however bizarre.” But many journals will no longer print those kinds of exchanges, they’re hardly easy for the uninitiated to follow, and there is often a strong incentive to shut up and play nice (unless they happen to be a peer-reviewer, in which case some will feel empowered by the cover of anonymity to be extraordinarily rude).

December 10, 2021

QotD: The media and the replication crisis

Filed under: Health, Media, Quotations, Science — Tags: , , , , — Nicholas @ 01:00

Here is the iron law of medical — in fact all scientific — studies in the modern world: most do not replicate. This has always been true of studies that supposedly find some link between doing [thing we enjoy] and cancer. This of course does not stop the media from running with initial study results based on 37 study participants as “fact”. The same is true for studies of new drugs and treatments. Most don’t pan out or are not nearly as efficacious as early studies might indicate. We have seen that over and over during COVID.

Warren Meyer, “A Couple of Thoughts on Medical Studies Given Recent Experience”, Coyote Blog, 2021-08-31.

February 23, 2021

The danger period is when “the coherence of the news breaks” … and it appears to be breaking before our eyes

Filed under: History, Media, Politics, Russia, USA — Tags: , , , , , , — Nicholas @ 03:00

In the latest Libertarian Enterprise, Sarah Hoyt remembers the breakdown of the Soviet Union and the disturbing parallels we can see today:

[Click to see full-size flowchart]

Our likelihood of coming out of this a constitutional republic is still high. Why? Well, because societies under stress become more themselves.

I remember when the USSR fell. And out of the ashes Tzar Putin emerged, who is despicable, but not particularly out of keeping with Russian monarchy.

So, yeah, the pull of our culture will be towards the reestablishment of who and what we are and were: a constitutional republic.

But on the way …

Look, I remember when the USSR fell.

The people in the USSR knew they were being treated like mushrooms: kept in the dark and fed on crap. They knew there was truth in Pravda. But they were used to having certain information, and interpreting it.

And there is something worse than reading the news in totalitarianism. You can get used to interpreting the news, and knowing the shape of the hole of what they’re not reporting.

But once you realize it’s all nonsense, once the coherence of the news breaks — and it’s doing so now, earlier than I expected, with the Times article, with the New York Times admitting the protesters at the capitol didn’t kill the police officer — once there are holes, but they’re not consistent, or they’re consistent, but then contradicted; once the narrative changes almost by the week, to the point it can’t be ignored, that’s the dangerous period.

I know I joke that by the end of this year I’ll have to apologize to the lizard-people conspiracy theorists. But the problem is that the lizard people conspiracy theorists can acquire respectability and a strange new respect. Or something even crazier. Heck, a lot of crazier things.

To an extent the 9/11 troofer conspiracies, which yes, are crazy and also anti-scientific were our warning shot. That they flourished and that to this day a lot of people believe them means that there was already a sense that the news made no sense, that there were other things going on behind the scenes that we weren’t aware of.

It’s going to get far, far worse than that, as the actual elites, the top of various fields fall like struck trees in a thunder storm. There is a good chance that authorities you rely on for your profession, or just for your knowledge have been compromised. A lot of our research is tainted by China paying to get the results it wants, for instance. And there’s probably worse. You already know most research can’t be reproduced, and that’s not even recent.

As all this stuff comes out, the problem is that people won’t stop believing. Instead they’ll believe in all and everything.

I don’t know how much was reported here, as the USSR collapsed. but I remember what I read in European magazines and journals. All of a sudden it was all new age mysticism and spoon bending and only the good Lord knew what else.

And that’s what we’re going to head into. So, when you find yourself in the middle of an elaborate explanation that someone constructed, well …

First find the facts. Pace Heinlein: Again, and again what are the facts. Never mind if your ideology demands they be something else. Establish the facts to the extent you can. Facts and math don’t lie. (Statistics do. So be aware you can lie with them. And any metrics that involve intangibles, like intelligence or performance much less sociability or micro anything? forget about it.)

From the facts, deploy Occam’s razor. What is the simplest explanation?

Then remember that humans run at the mouth, and the more humans in the conspiracy, the more facts are likely to leak out somewhere.

And while we’ve seen a lot of Omerta among leftists, note that they’re all afflicted by evil villain syndrome. Sooner or later, they brag about how clever they were in deceiving us. So, if your conspiracy theory requires perfect silence forever, it’s probably not true.

December 13, 2020

QotD: The statistical “Rule of Silicone Boobs”

Filed under: Health, Humour, Quotations, Science — Tags: , , , , — Nicholas @ 01:00

If it’s sexy, it’s probably fake.

“Sexy” means “likely to get published in the New York Times and/or get the researcher on a TEDx stage”. Actual sexiness research is not “sexy” because it keeps running into inconvenient results like that rich and high-status men in their forties and skinny women in their early twenties tend to find each other very sexy. The only way to make a result like that “sexy” is to blame it on the patriarchy, and most psychologist aren’t that far gone (yet).

[…]

Anything counterintuitive is also sexy, and (according to Rule 2) less likely to be true. So is anything novel that isn’t based on solid existing research. After all, the Times is the newspaper business, not in the truthspaper one.

Finding robust results is very hard, but getting sexy results published is very easy. Thus, sexy results generally lack robustness. I personally find a certain robustness quite sexy, but that attitude seems to have gone out of fashion since the Renaissance.

Jacob Falkovich, “The Scent of Bad Psychology”, Put a Number On It!, 2018-09-07.

November 4, 2020

The replication crisis in all fields is worse than you imagine

Filed under: Media, Science — Tags: , , , — Nicholas @ 03:00

It may sound like a trivial issue, but it absolutely is not: scientific studies that can’t be replicated are worthless, yet our lives are often impacted by these failed studies, especially when politicans are guided by junk science results:

More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments. Those are some of the telling figures that emerged from Nature‘s survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research.

The data reveal sometimes-contradictory attitudes towards reproducibility. Although 52% of those surveyed agree that there is a significant ‘crisis’ of reproducibility, less than 31% think that failure to reproduce published results means that the result is probably wrong, and most say that they still trust the published literature.

Data on how much of the scientific literature is reproducible are rare and generally bleak. The best-known analyses, from psychology and cancer biology, found rates of around 40% and 10%, respectively. Our survey respondents were more optimistic: 73% said that they think that at least half of the papers in their field can be trusted, with physicists and chemists generally showing the most confidence.

The results capture a confusing snapshot of attitudes around these issues, says Arturo Casadevall, a microbiologist at the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland. “At the current time there is no consensus on what reproducibility is or should be.” But just recognizing that is a step forward, he says. “The next step may be identifying what is the problem and to get a consensus.”

Failing to reproduce results is a rite of passage, says Marcus Munafo, a biological psychologist at the University of Bristol, UK, who has a long-standing interest in scientific reproducibility. When he was a student, he says, “I tried to replicate what looked simple from the literature, and wasn’t able to. Then I had a crisis of confidence, and then I learned that my experience wasn’t uncommon.”

The challenge is not to eliminate problems with reproducibility in published work. Being at the cutting edge of science means that sometimes results will not be robust, says Munafo. “We want to be discovering new things but not generating too many false leads.”

February 28, 2018

Psychology’s replication failures – “many casualties of the replication crisis do indeed bear a strong resemblance to voodoo”

Filed under: Health — Tags: , , , , — Nicholas @ 03:00

In Psychology Today, Lee Jussim says that once upon a time in the benighted, ignorant past, we generally believed in magic. In these more scientific, advanced, de-mystified days, we believe in psychology. He wonders if there’s actually much of a difference between the two:

Are some of the most famous effects in psychology – priming, stereotype threat, implicit bias – based on smoke and mirrors? Does the widespread credibility given such effects in the face of very weak evidence have a weird kinship with supernatural beliefs?

Many findings in psychology are celebrated in part because they were shocking and seemed almost magical. So magical that psychiatrist Scott Alexander argued that many casualties of the replication crisis do indeed bear a strong resemblance to voodoo — the main difference being an appeal to mysterious unobserved unconscious forces rather than mysterious unobserved supernatural ones.

Belief in the efficacy of voodoo itself can by psychologized: Curses work, some say, because believing you are hexed can kill you. There are similar mind-over-matter tales involving implausibly strong effects from placebos and self-affirmations. Priming studies claim that even thinking about the word “retirement” can transfer the weakness of old age into a young body, and make young people walk slower. The idea that people gravitate toward occupations that sound like their names bears a strange resemblance to sympathetic magic: If you name your daughter Suzie, she’s now more likely to wind up selling shells by the seashore, whereas your son Brandon will be a banker.

Supernatural beliefs are a universal feature of human societies. For people in many tribal societies, magic is serious business — a matter of life and death. Sorcerers can make good money by selling their services, while those accused of sorcery might be killed. The same was true in medieval Europe, where many were executed for supposedly using evil magic against their neighbors.

Belief in magic has retreated in modern times. Science has rendered the world less mysterious, technology has given us more effective control over it, and bureaucratic rules make life more predictable. Magic retreated.

Or did it? Any belief as universal as magic may be marvelously adapted to well-worn ruts in the human brain and encouraged by common structures and rhythms of human interaction.

April 30, 2017

[p-hacking] “is one of the many questionable research practices responsible for the replication crisis in the social sciences”

Filed under: Health, Media, Science — Tags: , , , , — Nicholas @ 03:00

What happens when someone digs into the statistics of highly influential health studies and discovers oddities? We’re in the process of finding out in the case of “rockstar researcher” Brian Wansink and several of his studies under the statistical microscope:

Things began to go bad late last year when Wansink posted some advice for grad students on his blog. The post, which has subsequently been removed (although a cached copy is available), described a grad student who, on Wansink’s instruction, had delved into a data set to look for interesting results. The data came from a study that had sold people coupons for an all-you-can-eat buffet. One group had paid $4 for the coupon, and the other group had paid $8.

The hypothesis had been that people would eat more if they had paid more, but the study had not found that result. That’s not necessarily a bad thing. In fact, publishing null results like these is important — failure to do so leads to publication bias, which can lead to a skewed public record that shows (for example) three successful tests of a hypothesis but not the 18 failed ones. But instead of publishing the null result, Wansink wanted to get something more out of the data.

“When [the grad student] arrived,” Wansink wrote, “I gave her a data set of a self-funded, failed study which had null results… I said, ‘This cost us a lot of time and our own money to collect. There’s got to be something here we can salvage because it’s a cool (rich & unique) data set.’ I had three ideas for potential Plan B, C, & D directions (since Plan A had failed).”

The responses to Wansink’s blog post from other researchers were incredulous, because this kind of data analysis is considered an incredibly bad idea. As this very famous xkcd strip explains, trawling through data, running lots of statistical tests, and looking only for significant results is bound to turn up some false positives. This practice of “p-hacking” — hunting for significant p-values in statistical analyses — is one of the many questionable research practices responsible for the replication crisis in the social sciences.

H/T to Kate at Small Dead Animals for the link.

June 10, 2016

A breakthrough in our understanding of the causes of depression

Filed under: Health, Science — Tags: , , , , — Nicholas @ 02:00

Scott Alexander takes a quick look at a recent discovery in medication for depression:

A few weeks ago, Nature published a bombshell study showing that ketamine’s antidepressant effects were actually caused by a metabolite, 2S,6S;2R,6R-hydroxynorketamine (don’t worry about the name; within ten years it’ll be called JOYVIVA™®© and you’ll catch yourself humming advertising jingles about it in the shower). Unlike ketamine, which is addictive and produces scary dissociative experiences, the metabolite is pretty safe. This is a big deal clinically, because it makes it easier and safer to prescribe to depressed people.

It’s also a big deal scientifically. Ketamine is a strong NMDA receptor antagonist; the metabolite is an AMPA agonist – they have different mechanisms of action. Knowing the real story behind why ketamine works will hopefully speed efforts to understand the nature of depression.

But I’m also interested in it from another angle. For the last ten years, everyone has been excited about ketamine. In a field that gets mocked for not having any really useful clinical discoveries in the last thirty years, ketamine was proof that progress was possible. It was the Exciting New Thing that everybody wanted to do research about.

Given the whole replication crisis thing, I wondered. You’ve got a community of people who think that NMDA antagonism and dissociation are somehow related to depression. If the latest study is true, all that was false. This is good; science is supposed to be self-correcting. But what about before it self-corrected? Did researchers virtuously say “I know the paradigm says NMDA is essential to depression, and nobody’s come up with a better idea yet, but there are some troubling inconsistencies in that picture”? Or did they tinker with their studies until they got the results they expected, then triumphantly declare that they had confirmed the dominant paradigm was right about everything all along?

September 5, 2015

The subtle lure of “research” that confirms our biases

Filed under: Health, Science — Tags: , , , , — Nicholas @ 02:00

Megan McArdle on why we fall for bogus research:

Almost three years ago, Nobel Prize-winning psychologist Daniel Kahneman penned an open letter to researchers working on “social priming,” the study of how thoughts and environmental cues can change later, mostly unrelated behaviors. After highlighting a series of embarrassing revelations, ranging from outright fraud to unreproducible results, he warned:

    For all these reasons, right or wrong, your field is now the poster child for doubts about the integrity of psychological research. Your problem is not with the few people who have actively challenged the validity of some priming results. It is with the much larger population of colleagues who in the past accepted your surprising results as facts when they were published. These people have now attached a question mark to the field, and it is your responsibility to remove it.

At the time it was a bombshell. Now it seems almost delicate. Replication of psychology studies has become a hot topic, and on Thursday, Science published the results of a project that aimed to replicate 100 famous studies — and found that only about one-third of them held up. The others showed weaker effects, or failed to find the effect at all.

This is, to put it mildly, a problem. But it is not necessarily the problem that many people seem to assume, which is that psychology research standards are terrible, or that the teams that put out the papers are stupid. Sure, some researchers doubtless are stupid, and some psychological research standards could be tighter, because we live in a wide and varied universe where almost anything you can say is certain to be true about some part of it. But for me, the problem is not individual research papers, or even the field of psychology. It’s the way that academic culture filters papers, and the way that the larger society gets their results.

August 29, 2015

We need a new publication called The Journal of Successfully Reproduced Results

Filed under: Media, Science — Tags: , , , , , — Nicholas @ 04:00

We depend on scientific studies to provide us with valid information on so many different aspects of life … it’d be nice to know that the results of those studies actually hold up to scrutiny:

One of the bedrock assumptions of science is that for a study’s results to be valid, other researchers should be able to reproduce the study and reach the same conclusions. The ability to successfully reproduce a study and find the same results is, as much as anything, how we know that its findings are true, rather than a one-off result.

This seems obvious, but in practice, a lot more work goes into original studies designed to create interesting conclusions than into the rather less interesting work of reproducing studies that have already been done to see whether their results hold up.

Everyone wants to be part of the effort to identify new and interesting results, not the more mundane (and yet potentially career-endangering) work of reproducing the results of older studies:

Why is psychology research (and, it seems likely, social science research generally) so stuffed with dubious results? Let me suggest three likely reasons:

A bias towards research that is not only new but interesting: An interesting, counterintuitive finding that appears to come from good, solid scientific investigation gets a researcher more media coverage, more attention, more fame both inside and outside of the field. A boring and obvious result, or no result, on the other hand, even if investigated honestly and rigorously, usually does little for a researcher’s reputation. The career path for academic researchers, especially in social science, is paved with interesting but hard to replicate findings. (In a clever way, the Reproducibility Project gets around this issue by coming up with the really interesting result that lots of psychology studies have problems.)

An institutional bias against checking the work of others: This is the flipside of the first factor: Senior social science researchers often actively warn their younger colleagues — who are in many cases the best positioned to check older work—against investigating the work of established members of the field. As one psychology professor from the University of Southern California grouses to the Times, “There’s no doubt replication is important, but it’s often just an attack, a vigilante exercise.”

[…]

Small, unrepresentative sample sizes: In general, social science experiments tend to work with fairly small sample sizes — often just a few dozen people who are meant to stand in for everyone else. Researchers often have a hard time putting together truly representative samples, so they work with subjects they can access, which in a lot of cases means college students.

A couple of years ago, I linked to a story about the problem of using western university students as the default source of your statistical sample for psychological and sociological studies:

A notion that’s popped up several times in the last couple of months is that the easy access to willing test subjects (university students) introduces a strong bias to a lot of the tests, yet until recently the majority of studies disregarded the possibility that their test results were unrepresentative of the general population.

August 14, 2015

QotD: When “the science” shows what you want it to show

Filed under: Media, Quotations, Science — Tags: , , , , , — Nicholas @ 01:00

To see what I mean, consider the recent tradition of psychology articles showing that conservatives are authoritarian while liberals are not. Jeremy Frimer, who runs the Moral Psychology Lab at the University of Winnipeg, realized that who you asked those questions about might matter — did conservatives defer to the military because they were authoritarians or because the military is considered a “conservative” institution? And, lo and behold, when he asked similar questions about, say, environmentalists, the liberals were the authoritarians.

It also matters because social psychology, and social science more generally, has a replication problem, which was recently covered in a very good article at Slate. Take the infamous “paradox of choice” study that found that offering a few kinds of jam samples at a supermarket was more likely to result in a purchase than offering dozens of samples. A team of researchers that tried to replicate this — and other famous experiments — completely failed. When they did a survey of the literature, they found that the array of choices generally had no important effect either way. The replication problem is bad enough in one subfield of social psychology that Nobel laureate Daniel Kahneman wrote an open letter to its practitioners, urging them to institute tougher replication protocols before their field implodes. A recent issue of Social Psychology was devoted to trying to replicate famous studies in the discipline; more than a third failed replication.

Let me pause here to say something important: Though I mentioned bias above, I’m not suggesting in any way that the replication problems mostly happen because social scientists are in on a conspiracy against conservatives to do bad research or to make stuff up. The replication problems mostly happen because, as the Slate article notes, journals are biased toward publishing positive and novel results, not “there was no relationship, which is exactly what you’d expect.” So readers see the one paper showing that something interesting happened, not the (possibly many more) teams that got muddy data showing no particular effect. If you do enough studies on enough small groups, you will occasionally get an effect just by random chance. But because those are the only studies that get published, it seems like “science has proved …” whatever those papers are about.

Megan McArdle, “The Truth About Truthiness”, Bloomberg View, 2014-09-08.

Older Posts »

Powered by WordPress