In the latest Age of Invention newsletter, Anton Howes considers the history profession’s closest equivalent to the ongoing replication crisis in the sciences:

… I’ve become increasingly worried that science’s replication crises might pale in comparison to what happens all the time in history, which is not just a replication crisis but a reproducibility crisis. Replication is when you can repeat an experiment with new data or new materials and get the same result. Reproducibility is when you use exactly the same evidence as another person and still get the same result — so it has a much, much lower bar for success, which is what makes the lack of it in history all the more worrying.

Historical myths, often based on mere misunderstanding, but occasionally on bias or fraud, spread like wildfire. People just love to share unusual and interesting facts, and history is replete with things that are both unusual and true. So much that is surprising or shocking has happened, that it can take only years or decades of familiarity with a particular niche of history in order to smell a rat. Not only do myths spread rapidly, but they survive — far longer, I suspect, than in scientific fields.

Take the oft-repeated idea that more troops were sent to quash the Luddites in 1812 than to fight Napoleon in the Peninsular War in 1808. Utter nonsense, as I set out in 2017, though it has been cited again and again and again as fact ever since Eric Hobsbawm first misled everyone back in 1964. Before me, only a handful of niche military history experts seem to have noticed and were largely ignored. Despite being busted, it continues to spread. Terry Deary (of Horrible Histories fame), to give just one of many recent examples, repeated the myth in a 2020 book. Historical myths are especially zombie-like. Even when disproven, they just. won’t. die.

[…]

I don’t think this is just me being grumpy and pedantic. I come across examples of mistakes being made and then spreading almost daily. It is utterly pervasive. Last week when chatting to my friend Saloni Dattani, who has lately been writing a piece on the story of the malaria vaccine, I shared my mounting paranoia healthy scepticism of secondary sources and suggested she check up on a few of the references she’d cited just to see. A few days later and Saloni was horrified. When she actually looked closely, many of the neat little anecdotes she’d cited in her draft — like Louis Pasteur viewing some samples under a microscope and having his mind changed on the nature of malaria — turned out to have no actual underlying primary source from the time. It may as well have been fiction. And there was inaccuracy after inaccuracy, often inexplicable: one history of the World Health Organisation’s malaria eradication programme said it had been planned to take 5-7 years, but the sources actually said 10-15; a graph showed cholera pandemics as having killed a million people, with no citation, while the main sources on the topic actually suggest that in 1865-1947 it killed some 12 million people in British India alone.

Now, it’s shockingly easy to make these mistakes — something I still do embarrassingly often, despite being constantly worried about it. When you write a lot, you’re bound to make some errors. You have to pray they’re small ones and try to correct them as swiftly as you can. I’m extremely grateful to the handful of subject-matter experts who will go out of their way to point them out to me. But the sheer pervasiveness of errors also allows unintentionally biased narratives to get repeated and become embedded as certainty, and even perhaps gives cover to people who purposefully make stuff up.

If the lack of replication or reproducibility is a problem in science, in history nobody even thinks about it in such terms. I don’t think I’ve ever heard of anyone systematically looking at the same sources as another historian and seeing if they’d reach the same conclusions. Nor can I think of a history paper ever being retracted or corrected, as they can be in science. At the most, a history journal might host a back-and-forth debate — sometimes delightfully acerbic — for the closely interested to follow. In the 1960s you could find an agricultural historian saying of another that he was “of course entitled to express his views, however bizarre.” But many journals will no longer print those kinds of exchanges, they’re hardly easy for the uninitiated to follow, and there is often a strong incentive to shut up and play nice (unless they happen to be a peer-reviewer, in which case some will feel empowered by the cover of anonymity to be extraordinarily rude).