In the Guardian, David Derbyshire takes the modern “science” of wine tasting to the woodshed:
… drawing on his background in statistics, Hodgson approached the organisers of the California State Fair wine competition, the oldest contest of its kind in North America, and proposed an experiment for their annual June tasting sessions.
Each panel of four judges would be presented with their usual “flight” of samples to sniff, sip and slurp. But some wines would be presented to the panel three times, poured from the same bottle each time. The results would be compiled and analysed to see whether wine testing really is scientific.
The first experiment took place in 2005. The last was in Sacramento earlier this month. Hodgson’s findings have stunned the wine industry. Over the years he has shown again and again that even trained, professional palates are terrible at judging wine.
“The results are disturbing,” says Hodgson from the Fieldbrook Winery in Humboldt County, described by its owner as a rural paradise. “Only about 10% of judges are consistent and those judges who were consistent one year were ordinary the next year.
“Chance has a great deal to do with the awards that wines win.”
These judges are not amateurs either. They read like a who’s who of the American wine industry from winemakers, sommeliers, critics and buyers to wine consultants and academics. In Hodgson’s tests, judges rated wines on a scale running from 50 to 100. In practice, most wines scored in the 70s, 80s and low 90s.
Results from the first four years of the experiment, published in the Journal of Wine Economics, showed a typical judge’s scores varied by plus or minus four points over the three blind tastings. A wine deemed to be a good 90 would be rated as an acceptable 86 by the same judge minutes later and then an excellent 94.
Today’s headline is a slightly stronger version of one I ran in May: Is wine tasting bullshit? with this rather amusing caption:
Although that “real” wine “review” illustrates the verbal bullshit side of wine reviewing, the statistical analysis in Robert Hodgson’s tests rather undermines the claims to any kind of actual analysis in most or all wine reviewing.
I’ve said for years that for most people there is a range of wine prices that will satisfy their tastes without emptying their wallets — in Ontario, the range for most people seems to be in the $14-$40 price spectrum. Pay less than that, and you risk buying wine that really isn’t very good (although there are some underpriced gems even there), and pay over $40 and you’re just paying extra for the “prestige” and most of us wouldn’t really be able to detect any flavour differences.
It’s interesting to see what kind of immediate environmental changes seem to be able to directly influence the scores given by reviewers:
More evidence that wine-tasting is influenced by context was provided by a 2008 study from Heriot-Watt University in Edinburgh. The team found that different music could boost tasters’ wine scores by 60%. Researchers discovered that a blast of Jimi Hendrix enhanced cabernet sauvignon while Kylie Minogue went well with chardonnay.
