Tuesday, July 09, 2013

Wine, critics and science (pseudo, junk and real). Part 1

The Today Programme feature

Is wine assessment science? Pseudo science? Junk science? Are wine judges and critics "bullshit artists"? 

I got sucked into having to consider all these questions when I agreed last week to take part in a brief discussion on the BBC Radio 4 Today Programme.

The producers' decision to tackle the issue was sparked by an article by David Derbyshire in the Observer headlined that wine tasting was "junk science". Robert Hodgson, a US oceanographer and statistician-turned-winemaker, had, according to the piece, "proved" that 

"Only about 10% of [US wine] judges are consistent and those judges who were consistent one year were ordinary the next year....
Chance has a great deal to do with the awards that wines win."

Hodgson has been researching wine judging consistency since 2005, saw early results published in the respected Journal of Wine Economics in 2008, and then introduced to the wider world in the Wall Street Journal the following year. The Observer piece, which covers some - but not much - new ground has created a substantial amount of noise since its publication on June 23.

The June 2013 Observer feature that ruffled
so many feathers

The very similar piece from the WSJ in 
November 2009, that went largely unnoticed

Apart from Hodgson's findings, the Observer and WSJ also gleefully picked up on earlier research by a Frenchmen called Frederic Brochet into Gallic tasters' reaction to the same wine when it was presented under different labels. (They rated it more highly - and used more positive terms - when it was served as a Bordeaux than as a Vin de Table). Consumers, it was gleefully revealed, actually often prefer cheap wine to pricier fare.

As a further killer punch to the egos of wordier wine authorities, there was also reference to research questioning the physical ability of the human palate to pick out the large number of specific flavours that frequently appear in some critics' descriptions. 

Putting all of this together, Derbyshire concludes that "human scores of wines are of limited value."

Tim Atkin, in a robust response in Off Licence News and in his Timatkin.com blog, declared that "

"Wine tasting is not a science (or “junk science”, as The Observer would have it) because it is a subjective exercise."

and devoted much to questioning Hodgson's research. 

"We are not told how “expert” the experts who performed badly were. Are we talking a sommelier with a couple of weeks’ work experience in a steakhouse? Or an experienced show judge who assesses wines professionally for a living? The claim that they read like a “who’s who of the American wine industry” sounds questionable. In Sacramento?"

Good competitions - like the IWC of which he is Co-Chair - have, Atkin claimed, 
"tasters [who] are perfectly capable of judging 100 or more wines in a day and of tasting them with insight and knowledge."

In raising doubts over the quality of the Sacramento competition, Atkin is actually in good company. Leading US wine blogger Blake Gray recently wrote a Palate Press piece that was highly critical of that event - and other California competitions.

It's all too easy, however, simply to dismiss allegedly dodgy US wine competitions, cranky US statisticians, and sensationalist UK newspapers. But let's look at the picture from a more neutral perspective. The Observer piece attracted over 400 comments, most of which were favourable to its tone.

The story was widely covered elsewhere and the BBC thought it worth discussing in its flagship morning radio news programme. Maybe, just maybe, we might have to acknowledge that we wine people might be out of tune with rather a lot of the consumers who go out and spend their own hard-earned money on the stuff we care so much about.

Atkin refers to this specifically:

Features bashing wine experts with their “flowery language” seem to appeal for two main reasons: first, they enable some members of the British wine-consuming public to indulge the muddle-headed notion that cheap plonk is “often superior” to more expensive stuff (that deal-driven, lowest-common-denominator mentality that has done so much damage to average wine quality in the UK) and, second, to give vent to a deep-seated insecurity about their own senses, invariably expressed as reverse snobbery. “I know what I like,” etc.

He does not seem to want to accept any responsibility for this state of affairs. It appears to be the British wine-consuming public's fault for failing to understand what we are saying to them. Of course, Atkin is totally correct in talking about UK consumers' tight-fistedness but inconveniently for his argument, most of the elements of this story originated in the US and in France. The Wall Street Journal article was definitely not aimed at British readers.
I was also struck by the sympathetic tone of contributors to more professional forums such as Linkedin's Wine Business Network

Wine tasting may not be a science, but I'll break ranks with many wine writers by saying that I agree with the basic theme of the criticisms: wine assessment and the way wine is written about do lay themselves wide open to accusations of being pseudo-scientific.

Let's start with so-called Parker Points. Giving a wine 89 or 90 or 91 points implies considerable precision, especially when, as we know, the side of the 90-point line on which a wine sits can have a huge impact on its marketability and the price it can command. (17.5/20 implies similar precision - and actually involves it if one considers that users of the 100-point scale rarely dip below 79 and those who prefer rating out of 20 treat 10 as their floor).

To the annoyance of many in the wine world, points have been welcomed both by consumers who find them an invaluable route through the wine jungle, and by producers and distributors who use them to drive sales.

But, as I've often been asked by consumers, how does a critic arrive at a mark of 89 rather than 90, say? The OIV competitions attempt to give structure to their scoring process by requiring judges to indicate the number of points they have allocated for colour, nose, typicity and so on - but they still allow a margin for tasters to show gut-preference. Besides, like many other OIV tasters, I'll admit to coming up with the mark first and filling the category numbers afterwards - just as most critics do when giving their "Parker" point. 

However they arrive at their mark, in theory at least, trained palates should achieve some level of consistency - and they often do. But it's far from absolute. Robert Parker admitted (to the author of the WSJ piece) that his marks for the same wine can deviate by two or three points, and it's a rare taster who'd stake anything of value on greater consistency than that. But three points on a scale that runs from 70-100 sounds very much like an error-rate of 10%. And when the variable is applied to wines getting 85-100 (which is likely to be the case), it's a lot more significant.

As some of the people who responded to the Observer article pointed out, there is a long list of reasonable explanations why even the most skilled and experienced taster might give - possibly widely - different marks to the same wine. Let's consider just a few of them:

  • The place and time. A wine may unsurprisingly seem very different when sampled in the producer's cellar to the way it tastes in a blind line-up in a competition. A few weeks can also have an effect on a wine's evolution, especially when it is young. (Think of all the variations between en primeur tasting notes written in early April and when the same wines are presented in June at Vinexpo.)
  • The time of day. A wine tasted at 9am, shortly after breakfast might be rated differently by a hungry taster at 1pm, or by one who has just enjoyed lunch at 2.30.
  • Physical state. The taster's state of tiredness, stress or physical health. Zinc deficiency, for example, can lead to a condition called Dysgeusia - "a distortion of the sense of taste" - often associated with complaints of foods having a a metallic taste. Patients undergoing chemotherapy frequently complain of their palates going awry. This Japanese piece of research reveals that physical exhaustion lowers the sucrose-perception threshold. The researchers on that occasion worked with athletes who'd run a half marathon, but some of the same effects might well be found in a taster who's had a sleepless night dealing with a young baby, for example. Another - US - study found that stress increases the perception of bitterness. Further causes of "bitter-taste-in-mout" include, acid reflux disease or GERDHiatal Hernias (or other hernias), tooth decay or gum disease, Achalasia (a condition that affects the oesophagus), H. Pylori infection (an extremely widespread bacteria that may infect around two-thirds of the people in the world), Oral CancersAspiration Pneumonia and syphilis.
  • Mood. Happiness, sadness, fear and anger can affect the way we perceive taste. According to a Los Angeles Times article,

    iven a neutral-tasting shot of diluted blue Gatorade, participants in a study in press at the journal PLoS One thought the beverage tasted more delicious after reading about someone being morally virtuous and more disgusting after reading about a moral transgression.

  • Time of the month. In the case of women, their menstrual cycle or pregnancy can affect perceptions of taste (the higher the levels of oestrogen, the lower the sucrose threshold)
  • External distractions
  • The position in a line-up
  • The temperature of the wine
  • The length of time the bottle has been open
  • The glass from which it is tasted (if you are to believe Georg Riedel's blindfold demonstrations)
  • Shipping. The old notion that some wines "don't travel" may have some validity. (New Zealand wine authority Bob Campbell addressed this issue last yearDr Neill McCallum, founder of Dry River Wines in Martinborough, New Zealand. McCallum, who gained his PhD at Oxford studying the relevant branch of chemistry, theorised that movement through travel breaks the hydrogen ions in wine resulting in muted flavour and aroma. Hydrogen ions do re-form, although not completely, according to McCallum, resulting in partial recovery of aroma and flavour when wine has a chance to recover after travel. He also notes that damage to hydrogen ions is significantly lower if the wine is cooler, supporting the case for shipping wine in refrigerated containers.
  • Lunar cycles. If you believe in biodynamics, the calendar is divided between four different kinds of day, depending on the lunar cycle. 

Some of these are apparently more propitious for planting than for wine tasting. It's easy to dismiss this as superstitious nonsense, but listen to Jo Ahearne MW, a trained winemaker who was employed as a buyer by Marks and Spencer in the UK: 

"I was sceptical at first, but then had a eureka moment," says Jo Aherne, winemaker at Marks and Spencer. "Our wines showed beautifully at a press tasting one day and far less well the next. We couldn't understand it. The wines were all favourites of ours and the bottles were all from the same case. Someone checked the calendar and we found that the first day had been a fruit day, when the wines were expressive, exuberant and aromatic, and the second a root day, when they were closed, tannic and earthy. Further rather unscientific tests have confirmed our view."

Ahearne is not alone in her belief: other UK retailers now avoid root days when inviting critics to taste their wines. Wine competitions have a trickier task, however. The IWC runs across two weeks - that could involve a lot of fruit and root days.

  • Altitude. It is acknowledged that foods and drinks taste different in airplanes, but that might be because of the lack of moisture in the cabin air: wines in particular seem to taste more tannic and more alcoholic. As a 2010 German study found,
"Light and fresh flavors decreased, whereas intensive flavors persisted"

But the altitude itself may have a role to play. Another study - published by Maga JA, Lorenz K.in the  US National Library of Medicine National Institutes of Health - suggests that from 5-10,000 feet, people become more tolerant to bitterness.

  • Barometric pressure. If altitude affects taste, so, logically, might the weather on the day the wine is tasted.
  • Humidity - or lack of it. The dry conditions in airline cabins are widely blamed for numbing tastebuds.
Okay, let's assume that none of these factors applies to the two bottles in front of us. Let's say that they were both bottled from the same tank on the same day, were pulled out of the same carton and poured into a pair of identical glasses to be tasted alongside each other. They should consequently taste the same - and receive the same marks from a competent taster. Shouldn't they?

The day I really lost my faith in the usefulness of specific wine ratings came late in 2004. I had been asked by a big bank to talk at a dinner to a large roomful of its guests about a set of very classy Italian wines. There were a dozen bottles each of examples of Ornelaia, Solaia, Sassicaia, Biondi-Santi, Gaja, Maculan and Jermann. (This was back in the days when banks threw their money around pretty freely.)

As usual, I pre-tasted all of the bottles for cork taint - and found a couple that showed clear signs of TCA. Two litres of sour milk out of 80 or so bottle, or a couple of bad eggs in a dozen cartons would be a cause for concern, but in the crazy world of wine, the sommelier at the venue and I shrugged off this level of faultiness in highly costly reds and whites, as par for the course. Far more concerning to me, however, were the more subtle variations I encountered in the other wines. When tasted really carefully - something wines of this calibre deserved - there was only one set that were truly 100% consistent: the Gajas. All the others included at least one or two bottles that tasted different enough from the others to warrant the addition or removal at least a point or from their score. 

Now - and this is important - I'm not claiming any exceptional tasting skills. Far from it. In fact, I tested my belief in the irregularity of the bottles by setting up a few three-glass tastings for the sommelier and an enthusiastic waiter. Even without any experience of the wines we were looking at, in three of the four instances, they had little difficulty in spotting the odd man out.

All the bottles were sealed with natural corks and I suspect that the poorer examples almost certainly suffered from very low level TCA or random oxidation - both of which flatten flavour, It is harder to explain the outstandingly better bottles. I suspect that particularly "good" natural corks may have provided optimum oxygen ingress plus possibly, that rarely discussed factor: "gout de bouchon" - the subtle flavour of non-tainted cork oak.

I was, naturally, aware of bottle variation before that day, but the experience was, as I say, faith-shaking. How many readers of my recommendations were sipping their freshly-bought bottles and shaking their heads saying "I've got no idea why he was so impressed by this."?

This naturally leads me on to one of the anomalies of wine competitions. I have chaired or co-chaired over 50 International Wine Challenges and other similar events across the globe and judged at dozens more. One of the things all these contests have in common is the encouragement to tasters to call for a second bottle if they have doubts about the first. On occasion, a third bottle can be requested.

Sometimes the call is easy: a wine reeks of TCA, vinegar or sherry. All too often, however, it's more marginal. Tasters discuss whether it's worth pulling another cork (or unscrewing a cap). Sometimes the second bottle is substantially better (or worse). In those cases, it is the good one that is judged and quite possibly given a gold medal. 

It's as though the judges at Wimbledon decided that perhaps Federer and Nadal weren't really playing at their best this year and deserved to be allowed another match to see whether they deserved to stay in the tournament. Or a restaurant critic giving Michelin stars to a restaurant where she has had to return a badly cooked dish.

At this stage of my fairly lengthy career, there is one thing of which I am sure. Every "Parker point" or medal, or media review represents the result of a single event: a single encounter or, possibly, in the case of a competition, a rapid succession of encounters with one or a small number of humans. It is not and cannot be definitive. Was Sabina Lisicki a 100-pointer in her losing-in-straight-sets Wimbledon final? Maybe not. But what if the clock had stopped 
on the day she'd beaten Serena Williams. At the final, it was Marion Bartoli who got the magic score. Are either of these players really greater than either Williams sister?

So yes, critics are bullshitting if they pretend that the mark or medal they give a wine on any given day has the quality of a verdict from the Supreme Court. And the wine world is guilty of pseudo science when it talks of a 95-point or trophy-winner as if that non-scientifically awarded mark or prize defines it in a permanent fashion. The truly great wines - like the truly great sportsmen, actors, singers, artists and authors - are the ones that consistently impress a number of credible critics.


  1. Far ranging post Robert, covering a lot of ground. It will be interesting to see wine judges/critics' responses. Another factor to consider is age of taster - development/deterioration of palate over two decades or more. Does one really taste/appreciate the same flavours as 20 years ago any more than have exactly the same opinions? I hope not. Am actively suspicious of 50yrs worth of experience as am doubtful its the same tongue in the same head.

    Am also glad you raised the issue of altitude/pressurised cabin on wines/our palates as we have another Altitude tasting planned for September - will have to save you a spot...

  2. Anonymous8:10 am

    This comment has been removed by a blog administrator.

  3. Thanks Tom. I agree re age being a factor. National/cultural responses to sugar/bitterness is also relevant.
    Thanks for thought re Altitude, though I fear I may be trying it for real in Sept - taking a lot of flights.

  4. Anonymous8:13 am

    Thanks for putting this together. Very helpful post.
    The tennis/Wimbledon analogy is very apt and informative.
    The best player ON THE DAY.

  5. http://winewizard.co.za/

  6. While on the topic....http://winewizard.co.za/


  7. I reckon the lenght of time the wine is poured before being judged has a huge impact too, along with temperature of the wine at time of tasting. Moreso how you choose to 'look at' the wine ... I prefer to nose a lineup first, with plenty of swirling then taste and nose again with plenty of swirling and then nose/taste again. In very lineup there are at least 30% of wines that I change my opinion of substantially between initial nosing and writing notes and tasting. If it is a class of 10 wines, vs the same wine in a class of 50 wines, chances are, my opinion will be swayed differently because of the amount of time the wine has had to open up will differ and because it will likely be preceded by a different wine. I think, ultimately the best way to assess a wine is over 4 or 5 hours with a few friends. it's not very scientific really but its better than deciding on the merits of a library after reading the back covers.

    1. See Catherine Monahan's comment below. I agree with most of what you say and do - but have to say that it all makes our marks and ratings even less relevant to consumers. It's as though we're motoring journalists who think that a day driving round the track is the best way to road-test a car. (Maybe it's worth checking whether there's room for the shopping in the boot, and for a baby-seat in the back)

  8. Marie Calderón Sund wrote on Facebook

    As I always say to consumers at my wine presentations:

    "Trust your own taste" - have faith in your own taste.

    The wine critics may help the consumers to sort out the wines on the huge market/djungle, but we need to remember that this reviewer may not have the same taste on wines as yourself.

    The problem starts in the trade when a wine buyers concentrate too much on the points from one/two critics. If your winery's wines have not been evaluated by this or that wine critic your are not even "a wine to be considered" to compete for the shelf space.

    We should not to forget that wine critics are human beings as ourselves, with bad and good days. Who said that there is only one truth in life, there are always several truths, it just depends from which perspective you look from.

  9. Wink Lorch wrote on Facebook Robert Joseph examines in some detail the subject that's being - quite rightly in my view - done to death at present, and about time too. Unfortunately the industry has been governed ever-increasingly by scores and medals in which many consumers say they don't believe, and yet really they do, because what else do they have to grasp when they choose their wines? Would be great if we could move on, somehow.

    1. The trouble, Wink, is that the very complexity that wine enthusiasts so love creates the need for the kind of simplistic guidance provided by marks and medals.
      If wine were as relatively simple as spirits - and as heavily branded, this issue would more or less disappear.

  10. Ken Payton wrote on Facebook

    Don't really think wine writing will ever move beyond these simple adolescent preoccupations. The thematic repetition is mind-numbing... honestly...

    1. Chris Kassel wrote on Facebook That's pretty cynical, Ken. Any genre is only as good as its practitioners, and new ones are born every day. Of course, we'll have to wait 21 years before we hear what they have to say...

    2. I'm not sure what you mean by "simple adolescent preoccupations", Ken. All I can say is that the wider public (WSJ, Observer, BBC etc) seems to find it interesting. Could it possibly be that you're too close to the subject? Interestingly, this has been one of my more popular posts for a while, to judge by pageviews, favourites and retweets. But there's no accounting for taste.

  11. Nicholas King @beerwest wrote
    great piece, great read and IMO bang on the money.

  12. LJLoch @ljLoch and @ggriffins wrote the art & science of wine tasting. Long essay but @robertjoseph makes it fun and useful

  13. The irony is that the way wine is bought by professional buyers and by the consumers bears no resemblance to how all these competitions taste wine - and that is blind - in all meaning of the word.

    1. True, but there crossover there and paradox too ... judge a wine highly, find identity afterward, then source for business ... rare to radically change my mind on the wine when retasting in the latter scenario. Mind you, if it looks average on opening, previous experience of the wine absolutely ensures you keep coming back to it until it looks as you remembered it.

    2. That's also true for consumers.... But only if they already know the wine. Most people are encountering wine for the first time. Like meeting them at a party. First impressions are the ones that count I'm afraid. In life, you don't often get a second chance, once you've got it badly wrong.

  14. A very fair assessment Robert - once again.

    There are a lot of very expert wine tasters out there, but many of them would prove less consistent than they'd like to think they are, as I think the AWRI have proved.

    Scoring wines is a bit like scoring your friends. Imagine if I am giving a dinner party, and only inviting 90's and above. Luckily, Robert, on the basis of this article, (and some other qualities besides) you qualify!

    I proffer that we like our friends because of their character, more than because of their objective excellence, and much the same is true of wine.....

    1. Good point. And we accept our friends' inconsistencies - and that when they're in a bad mood they may not go down well with some of the other people we know...

  15. Chris Kassel wrote on Facebook Good piece. Mine on the same subject:


    In short, all the articles I quoted refer to the Hodgson California State Fair' experiment' as well.

    Well, access his website and find that his OWN introductory statement is: ‘Fieldbrook Winery is recognized for producing medal winning wines in both national and international wine competitions for over 30 years,’ then proceeds to proudly list over ninety awards and medals his wines have taken at various competitions, including the California State Fair.

    1. Daniel Mayer If I would have spend that money on entering competitions I would certainly point out as well the good sides of them..

    2. Thanks Chris. And I like your piece... Some very good points in there....

  16. Rufus Weston nice piece Robert Joseph

  17. Anonymous12:15 am

    Very good piece Robert. I've had a draft blog on my desktop for a month or two on this subject that I need to finish now. You've found one of my metaphors and I feel like a pint size Darwin needing to publish now....Thanks for the motivation!

    1. Thanks Dudley, I look forward to seeing it.

  18. Stuart Pemble wrote on Facebook - in an answer to a brief post by Wink Lorch about this essay Very interesting Wink, but I think wine consumers are being done a collective dis-service, in part by assuming there's such a thing as the average wine consumer; who is to be distinguished from the 'average' wine expert. I imagine the latter category is quite diverse; the former must be even more so.

    The choices we make about wine or anything else we buy are personal, but there's nothing objectively wrong with buying wine because someone you trust has recommended it. Parker points may be a cliche, and overly-simplistic and inaccurate as Robert Joseph argues brilliantly - but I suspect Mr Parker has risen to prominence in part because people trusted his judgment. Whether his palate is any better than anyone else's or whether his influence on wine style has been a good thing is a whole different debate, but his influence was as a result of market demand and not imposed on unsuspecting public from on high. Likewise, a Decanter or IWC medal does make a difference, as does a Tim Atkin choice on Saturday Kitchen; but I doubt that anyone takes them to be a guarantee of perfection.

    I suspect that most people remain intimidated by wine and the complicated wine world. I certainly used to be. Now, it's my big passion. I'm not sure I understand much of it, but I love reading about it, learning (in a loose sense rather than a formal study sense) about it and, most of all, drinking it with friends and family.

    And it's in debunking the myths and making wine accessible that wine experts come into their own. Most people love a wine tasting and I really value the opinion of you and other experts and am inspired by great wine writing and explanations and stories. I agree that where you taste and the company you taste it with can make a huge difference - but that doesn't stop me being more likely to try a Jura wine you rated as being exceptional than one you told me was ordinary.

    There's nothing wrong with that, is there?

    1. Robert Joseph Great contribution Stuart (and thanks for the compliment). I agree especially with your view that Parker's success (and others') is driven by the market rather than imposed. My only difference with you is that the points and medals ARE somewhat imposed, just as Oscars and book prizes are, by the people who use them for marketing. If fewer merchants and producers crowed about their 95 Parker points, far fewer people would ever have heard of him...