Wine, critics and science (pseudo, junk and real). Part 1

The Today Programme feature

Is wine assessment science? Pseudo science? Junk science? Are wine judges and critics "bullshit artists"? 

I got sucked into having to consider all these questions when I agreed last week to take part in a brief discussion on the BBC Radio 4 Today Programme.

The producers' decision to tackle the issue was sparked by an article by David Derbyshire in the Observer headlined that wine tasting was "junk science". Robert Hodgson, a US oceanographer and statistician-turned-winemaker, had, according to the piece, "proved" that 

"Only about 10% of [US wine] judges are consistent and those judges who were consistent one year were ordinary the next year....Chance has a great deal to do with the awards that wines win."

Hodgson has been researching wine judging consistency since 2005, saw early results published in the respected Journal of Wine Economics in 2008, and then introduced to the wider world in the Wall Street Journal the following year. The Observer piece, which covers some - but not much - new ground has created a substantial amount of noise since its publication on June 23.

The June 2013 Observer feature that ruffled
so many feathers

The very similar piece from the WSJ in 

November 2009, that went largely unnoticed

Apart from Hodgson's findings, the Observer and WSJ also gleefully picked up on earlier research by a Frenchmen called Frederic Brochet into Gallic tasters' reaction to the same wine when it was presented under different labels. (They rated it more highly - and used more positive terms - when it was served as a Bordeaux than as a Vin de Table). Consumers, it was gleefully revealed, actually often prefer cheap wine to pricier fare.

As a further killer punch to the egos of wordier wine authorities, there was also reference to research questioning the physical ability of the human palate to pick out the large number of specific flavours that frequently appear in some critics' descriptions. 

Putting all of this together, Derbyshire concludes that "human scores of wines are of limited value."

Tim Atkin, in a robust response in Off Licence News and in his Timatkin.com blog, declared that "

"Wine tasting is not a science (or “junk science”, as The Observer would have it) because it is a subjective exercise."

and devoted much to questioning Hodgson's research. 

"We are not told how “expert” the experts who performed badly were. Are we talking a sommelier with a couple of weeks’ work experience in a steakhouse? Or an experienced show judge who assesses wines professionally for a living? The claim that they read like a “who’s who of the American wine industry” sounds questionable. In Sacramento?"

Good competitions - like the IWC of which he is Co-Chair - have, Atkin claimed, 

"tasters [who] are perfectly capable of judging 100 or more wines in a day and of tasting them with insight and knowledge."

In raising doubts over the quality of the Sacramento competition, Atkin is actually in good company. Leading US wine blogger Blake Gray recently wrote a Palate Press piece that was highly critical of that event - and other California competitions.

It's all too easy, however, simply to dismiss allegedly dodgy US wine competitions, cranky US statisticians, and sensationalist UK newspapers. But let's look at the picture from a more neutral perspective. The Observer piece attracted over 400 comments, most of which were favourable to its tone.

The story was widely covered elsewhere and the BBC thought it worth discussing in its flagship morning radio news programme. Maybe, just maybe, we might have to acknowledge that we wine people might be out of tune with rather a lot of the consumers who go out and spend their own hard-earned money on the stuff we care so much about.

Atkin refers to this specifically:

Features bashing wine experts with their “flowery language” seem to appeal for two main reasons: first, they enable some members of the British wine-consuming public to indulge the muddle-headed notion that cheap plonk is “often superior” to more expensive stuff (that deal-driven, lowest-common-denominator mentality that has done so much damage to average wine quality in the UK) and, second, to give vent to a deep-seated insecurity about their own senses, invariably expressed as reverse snobbery. “I know what I like,” etc.

He does not seem to want to accept any responsibility for this state of affairs. It appears to be the British wine-consuming public's fault for failing to understand what we are saying to them. Of course, Atkin is totally correct in talking about UK consumers' tight-fistedness but inconveniently for his argument, most of the elements of this story originated in the US and in France. The Wall Street Journal article was definitely not aimed at British readers.
I was also struck by the sympathetic tone of contributors to more professional forums such as Linkedin's Wine Business Network

Wine tasting may not be a science, but I'll break ranks with many wine writers by saying that I agree with the basic theme of the criticisms: wine assessment and the way wine is written about do lay themselves wide open to accusations of being pseudo-scientific.

Let's start with so-called Parker Points. Giving a wine 89 or 90 or 91 points implies considerable precision, especially when, as we know, the side of the 90-point line on which a wine sits can have a huge impact on its marketability and the price it can command. (17.5/20 implies similar precision - and actually involves it if one considers that users of the 100-point scale rarely dip below 79 and those who prefer rating out of 20 treat 10 as their floor).

To the annoyance of many in the wine world, points have been welcomed both by consumers who find them an invaluable route through the wine jungle, and by producers and distributors who use them to drive sales.

But, as I've often been asked by consumers, how does a critic arrive at a mark of 89 rather than 90, say? The OIV competitions attempt to give structure to their scoring process by requiring judges to indicate the number of points they have allocated for colour, nose, typicity and so on - but they still allow a margin for tasters to show gut-preference. Besides, like many other OIV tasters, I'll admit to coming up with the mark first and filling the category numbers afterwards - just as most critics do when giving their "Parker" point. 

However they arrive at their mark, in theory at least, trained palates should achieve some level of consistency - and they often do. But it's far from absolute. Robert Parker admitted (to the author of the WSJ piece) that his marks for the same wine can deviate by two or three points, and it's a rare taster who'd stake anything of value on greater consistency than that. But three points on a scale that runs from 70-100 sounds very much like an error-rate of 10%. And when the variable is applied to wines getting 85-100 (which is likely to be the case), it's a lot more significant.

As some of the people who responded to the Observer article pointed out, there is a long list of reasonable explanations why even the most skilled and experienced taster might give - possibly widely - different marks to the same wine. Let's consider just a few of them:

• The place and time. A wine may unsurprisingly seem very different when sampled in the producer's cellar to the way it tastes in a blind line-up in a competition. A few weeks can also have an effect on a wine's evolution, especially when it is young. (Think of all the variations between en primeur tasting notes written in early April and when the same wines are presented in June at Vinexpo.)
• The time of day. A wine tasted at 9am, shortly after breakfast might be rated differently by a hungry taster at 1pm, or by one who has just enjoyed lunch at 2.30.
• Physical state. The taster's state of tiredness, stress or physical health. Zinc deficiency, for example, can lead to a condition called Dysgeusia - "a distortion of the sense of taste" - often associated with complaints of foods having a a metallic taste. Patients undergoing chemotherapy frequently complain of their palates going awry. This Japanese piece of research reveals that physical exhaustion lowers the sucrose-perception threshold. The researchers on that occasion worked with athletes who'd run a half marathon, but some of the same effects might well be found in a taster who's had a sleepless night dealing with a young baby, for example. Another - US - study found that stress increases the perception of bitterness. Further causes of "bitter-taste-in-mout" include, acid reflux disease or GERD, Hiatal Hernias (or other hernias), tooth decay or gum disease, Achalasia (a condition that affects the oesophagus), H. Pylori infection (an extremely widespread bacteria that may infect around two-thirds of the people in the world), Oral Cancers, Aspiration Pneumonia and syphilis.
• Mood. Happiness, sadness, fear and anger can affect the way we perceive taste. According to a Los Angeles Times article,
  
  Given a neutral-tasting shot of diluted blue Gatorade, participants in a study in press at the journal PLoS One thought the beverage tasted more delicious after reading about someone being morally virtuous and more disgusting after reading about a moral transgression.
  
  
  

  • 

• Time of the month. In the case of women, their menstrual cycle or pregnancy can affect perceptions of taste (the higher the levels of oestrogen, the lower the sucrose threshold)
• External distractions
• The position in a line-up
• The temperature of the wine
• The length of time the bottle has been open
• The glass from which it is tasted (if you are to believe Georg Riedel's blindfold demonstrations)

• Shipping. The old notion that some wines "don't travel" may have some validity. (New Zealand wine authority Bob Campbell addressed this issue last yearDr Neill McCallum, founder of Dry River Wines in Martinborough, New Zealand. McCallum, who gained his PhD at Oxford studying the relevant branch of chemistry, theorised that movement through travel breaks the hydrogen ions in wine resulting in muted flavour and aroma. Hydrogen ions do re-form, although not completely, according to McCallum, resulting in partial recovery of aroma and flavour when wine has a chance to recover after travel. He also notes that damage to hydrogen ions is significantly lower if the wine is cooler, supporting the case for shipping wine in refrigerated containers.
  
• Lunar cycles. If you believe in biodynamics, the calendar is divided between four different kinds of day, depending on the lunar cycle. 

Some of these are apparently more propitious for planting than for wine tasting. It's easy to dismiss this as superstitious nonsense, but listen to Jo Ahearne MW, a trained winemaker who was employed as a buyer by Marks and Spencer in the UK: 

"I was sceptical at first, but then had a eureka moment," says Jo Aherne, winemaker at Marks and Spencer. "Our wines showed beautifully at a press tasting one day and far less well the next. We couldn't understand it. The wines were all favourites of ours and the bottles were all from the same case. Someone checked the calendar and we found that the first day had been a fruit day, when the wines were expressive, exuberant and aromatic, and the second a root day, when they were closed, tannic and earthy. Further rather unscientific tests have confirmed our view."

Ahearne is not alone in her belief: other UK retailers now avoid root days when inviting critics to taste their wines. Wine competitions have a trickier task, however. The IWC runs across two weeks - that could involve a lot of fruit and root days.

• Altitude. It is acknowledged that foods and drinks taste different in airplanes, but that might be because of the lack of moisture in the cabin air: wines in particular seem to taste more tannic and more alcoholic. As a 2010 German study found,

"Light and fresh flavors decreased, whereas intensive flavors persisted"

But the altitude itself may have a role to play. Another study - published by Maga JA, Lorenz K.in the  US National Library of Medicine National Institutes of Health - suggests that from 5-10,000 feet, people become more tolerant to bitterness.

• Barometric pressure. If altitude affects taste, so, logically, might the weather on the day the wine is tasted.
• Humidity - or lack of it. The dry conditions in airline cabins are widely blamed for numbing tastebuds.

Okay, let's assume that none of these factors applies to the two bottles in front of us. Let's say that they were both bottled from the same tank on the same day, were pulled out of the same carton and poured into a pair of identical glasses to be tasted alongside each other. They should consequently taste the same - and receive the same marks from a competent taster. Shouldn't they?

The day I really lost my faith in the usefulness of specific wine ratings came late in 2004. I had been asked by a big bank to talk at a dinner to a large roomful of its guests about a set of very classy Italian wines. There were a dozen bottles each of examples of Ornelaia, Solaia, Sassicaia, Biondi-Santi, Gaja, Maculan and Jermann. (This was back in the days when banks threw their money around pretty freely.)

As usual, I pre-tasted all of the bottles for cork taint - and found a couple that showed clear signs of TCA. Two litres of sour milk out of 80 or so bottle, or a couple of bad eggs in a dozen cartons would be a cause for concern, but in the crazy world of wine, the sommelier at the venue and I shrugged off this level of faultiness in highly costly reds and whites, as par for the course. Far more concerning to me, however, were the more subtle variations I encountered in the other wines. When tasted really carefully - something wines of this calibre deserved - there was only one set that were truly 100% consistent: the Gajas. All the others included at least one or two bottles that tasted different enough from the others to warrant the addition or removal at least a point or from their score. 

Now - and this is important - I'm not claiming any exceptional tasting skills. Far from it. In fact, I tested my belief in the irregularity of the bottles by setting up a few three-glass tastings for the sommelier and an enthusiastic waiter. Even without any experience of the wines we were looking at, in three of the four instances, they had little difficulty in spotting the odd man out.

All the bottles were sealed with natural corks and I suspect that the poorer examples almost certainly suffered from very low level TCA or random oxidation - both of which flatten flavour, It is harder to explain the outstandingly better bottles. I suspect that particularly "good" natural corks may have provided optimum oxygen ingress plus possibly, that rarely discussed factor: "gout de bouchon" - the subtle flavour of non-tainted cork oak.

I was, naturally, aware of bottle variation before that day, but the experience was, as I say, faith-shaking. How many readers of my recommendations were sipping their freshly-bought bottles and shaking their heads saying "I've got no idea why he was so impressed by this."?

This naturally leads me on to one of the anomalies of wine competitions. I have chaired or co-chaired over 50 International Wine Challenges and other similar events across the globe and judged at dozens more. One of the things all these contests have in common is the encouragement to tasters to call for a second bottle if they have doubts about the first. On occasion, a third bottle can be requested.

Sometimes the call is easy: a wine reeks of TCA, vinegar or sherry. All too often, however, it's more marginal. Tasters discuss whether it's worth pulling another cork (or unscrewing a cap). Sometimes the second bottle is substantially better (or worse). In those cases, it is the good one that is judged and quite possibly given a gold medal. 

It's as though the judges at Wimbledon decided that perhaps Federer and Nadal weren't really playing at their best this year and deserved to be allowed another match to see whether they deserved to stay in the tournament. Or a restaurant critic giving Michelin stars to a restaurant where she has had to return a badly cooked dish.

At this stage of my fairly lengthy career, there is one thing of which I am sure. Every "Parker point" or medal, or media review represents the result of a single event: a single encounter or, possibly, in the case of a competition, a rapid succession of encounters with one or a small number of humans. It is not and cannot be definitive. Was Sabina Lisicki a 100-pointer in her losing-in-straight-sets Wimbledon final? Maybe not. But what if the clock had stopped 
on the day she'd beaten Serena Williams. At the final, it was Marion Bartoli who got the magic score. Are either of these players really greater than either Williams sister?

So yes, critics are bullshitting if they pretend that the mark or medal they give a wine on any given day has the quality of a verdict from the Supreme Court. And the wine world is guilty of pseudo science when it talks of a 95-point or trophy-winner as if that non-scientifically awarded mark or prize defines it in a permanent fashion. The truly great wines - like the truly great sportsmen, actors, singers, artists and authors - are the ones that consistently impress a number of credible critics.