Quotulatiousness

August 29, 2015

We need a new publication called The Journal of Successfully Reproduced Results

Filed under: Media, Science — Tags: , , , , — Nicholas @ 04:00

We depend on scientific studies to provide us with valid information on so many different aspects of life … it’d be nice to know that the results of those studies actually hold up to scrutiny:

One of the bedrock assumptions of science is that for a study’s results to be valid, other researchers should be able to reproduce the study and reach the same conclusions. The ability to successfully reproduce a study and find the same results is, as much as anything, how we know that its findings are true, rather than a one-off result.

This seems obvious, but in practice, a lot more work goes into original studies designed to create interesting conclusions than into the rather less interesting work of reproducing studies that have already been done to see whether their results hold up.

Everyone wants to be part of the effort to identify new and interesting results, not the more mundane (and yet potentially career-endangering) work of reproducing the results of older studies:

Why is psychology research (and, it seems likely, social science research generally) so stuffed with dubious results? Let me suggest three likely reasons:

A bias towards research that is not only new but interesting: An interesting, counterintuitive finding that appears to come from good, solid scientific investigation gets a researcher more media coverage, more attention, more fame both inside and outside of the field. A boring and obvious result, or no result, on the other hand, even if investigated honestly and rigorously, usually does little for a researcher’s reputation. The career path for academic researchers, especially in social science, is paved with interesting but hard to replicate findings. (In a clever way, the Reproducibility Project gets around this issue by coming up with the really interesting result that lots of psychology studies have problems.)

An institutional bias against checking the work of others: This is the flipside of the first factor: Senior social science researchers often actively warn their younger colleagues — who are in many cases the best positioned to check older work—against investigating the work of established members of the field. As one psychology professor from the University of Southern California grouses to the Times, “There’s no doubt replication is important, but it’s often just an attack, a vigilante exercise.”

[…]

Small, unrepresentative sample sizes: In general, social science experiments tend to work with fairly small sample sizes — often just a few dozen people who are meant to stand in for everyone else. Researchers often have a hard time putting together truly representative samples, so they work with subjects they can access, which in a lot of cases means college students.

A couple of years ago, I linked to a story about the problem of using western university students as the default source of your statistical sample for psychological and sociological studies:

A notion that’s popped up several times in the last couple of months is that the easy access to willing test subjects (university students) introduces a strong bias to a lot of the tests, yet until recently the majority of studies disregarded the possibility that their test results were unrepresentative of the general population.

The bad news about good news

Filed under: Economics, Media, Politics — Tags: , , , — Nicholas @ 03:00

At Strategy Page, an explanation for why most people think the world is going to hell, despite the facts pointing in all kinds of positive and hopeful directions:

One of the ironies of the post-Cold War world is that most people get the impression that things are getting worse and worse while for the majority of people on the planet life is getting better. Worldwide poverty and death rates are plummeting while income and reported (via opinion surveys) satisfaction are way up. Many major diseases (like tetanus and polio) have nearly been eliminated and malaria, the disease that has killed more people than any other throughout history, is in decline because of medical advances. War related deaths have been declining since World War II ended in 1945 and that decline continued after the Cold War eliminated most communist governments in 1991. Why do most people think otherwise? You can blame the mass media and their most effective marketing tool; FUD (Fear, Uncertainty, and Doubt).

Mass media first appeared in the mid-19th century with the development of the steam press, which made cheap-enough-to-reach-a-mass-audience newspapers possible. Editors quickly learned that FUD sells best. Politicians, rebels, and even advertisers found that FUD was a very effective tool to grab attention and change attitudes. Put another way, excitement sells, and the best way to excite readers is to scare them.

Modern terrorism, based on using murderous mass attacks on the public to trigger a flurry of media coverage, came out of this. The 19th century anarchists, followed by the Bolsheviks (communists), several fascist movements (like the Nazis), and many others, all used this media proclivity to jump on terrorist acts in order to scare readers into buying more newspapers, or supporting some extremist cause or another. The terrorists got the publicity and attention they wanted, which sometimes led to acquiring political power as well.

Radio appeared in the 1930s and this made it even easier to reach literate as well as illiterate populations. Combining radio and FUD allowed communism and fascism to spread far and fast in the 1930s. The sad fact is that this situation is not unknown among journalists. Many of them have been complaining about it for over a century. No one has been able to come up with a solution. Good news doesn’t sell. And the pursuit of scary headlines that do has created a race to the bottom.

It’s probably rational for mass media outlets to concentrate on the vivid, shocking bad news … because it grabs the attention and sells more newspapers and encourages more people to watch video reports. Good news? Well, it’s nice to hear, but it’s neither urgent nor compelling (except cat videos on YouTube, of course). You might like to hear it, but it’s not urgent and compelling … you can catch up on that anytime. A flood? An earthquake? A breaking story about a hostage situation? You’ll pay attention whether you want to or not. And that sells newspapers and gets ad revenue for networks.

The US Army/USMC replacement for the Humvee

Filed under: Military, USA — Tags: , — Nicholas @ 02:00

At Breaking Defense, Colin Clark explains why the recent contract award to build the first batch of Joint Light Tactical Vehicles (JLTV) to Oshkosh is kind of a big deal:

You wouldn’t have known it from the way the Army announced it, but the service awarded arguably its most important contract in a decade this evening to build the first 17,000 Joint Light Tactical Vehicles (JLTV) to Oshkosh.

“The JLTV production contract is a historic win for Oshkosh Corporation and more than 300 suppliers in 31 states across the country, and most importantly, for America’s warfighters,” says Charles Szews, Oshkosh CEO. Oshkosh beat back impressive efforts by Lockheed Martin and AM General to win today’s $6.75 billion contract. We’ll find out in the next 10 days if either or both of them file a protest. Many observers expect just that and the program officials at this evening’s briefing were unwilling to say virtually anything about why Oshkosh won or the strengths or weaknesses of any of the three competitors. They clearly feared giving someone grounds for a protest. Scott Davis, head of the Army’s Program Executive Office Combat Support & Combat Service Support, told us “there is no expectation of a protest,” but his language was very carefully chosen. They may not expect a protest, but that doesn’t mean they aren’t worried one will be filed.

Sen. Tom Cotton, in whose state the Lockheed version would have been built, appeared to open the door to political pressure to change the results when he issued a statement this evening that included the pledge that “as Lockheed Martin explores their next steps, we stand ready to assist them however we can.” Since the fixed price low rate initial production contract with eight options has been awarded, about the only next step would be a protest.

How committed was Lockheed Martin to this competition? It bought partner BAE Systems’ entire wheeled vehicle production line and physically moved it from Sealy, Texas to Camden, Ark.

The JLTV will replace most of the US military’s Humvees, the iconic vehicle built by AM General. The Marines are getting 5,500 JLTVs and the rest go to the Army. Up to 40,000 JLTVs will be built through 2040.

QotD: Superstitions

Filed under: Quotations, Religion — Tags: , — Nicholas @ 01:00

If, as the Army tests of conscripts showed, nearly 50 per cent. of American adult males never get beyond the mental development of a twelve-year-old child, then it must be obvious that a much smaller number get beyond the mental development of a youth at the end of his teens. I put that number, at a venture, at 5 per cent. The remaining 95 per cent never quite free themselves from religious superstitions. They may no longer believe it is an act of God every time an individual catches a cold, or sprains his ankle, or cuts himself shaving, but they are pretty sure to see some trace of divine intervention in it if he is struck by lightning, or hanged, or afflicted with leprosy or syphilis.

H.L. Mencken, “The Nature of Faith”, Prejudices, Fourth Series, 1924.

Powered by WordPress