by Elliott Morss, Morss Global Finance
The evidence from blind tastings of wines people like and their ability to tell differences between wines is growing rapidly. In this piece, I look carefully at this evidence and attempt to interpret what it all means.
Humans have five sense organs: eyes, ears, tongue, touch, and nose. Three of these are used when tasting wines: eyes, tongue and nose. “Blind tasting” is normally a misnomer; it simply means people do not know what wines is in each glass they are tasting.
a. Blind Tastings Using Eyes, Tongue and Nose
From tastings in Paris , Princeton , Stellenbosch  and the Lenox Wine Club [2012-14], we have learned there is no correlation between price and scores. This means inexpensive wines taste just as good as expensive ones. And this conclusion holds for most so-called expert tasters and well as for the “hoi polloi”.
Goldstein et al analyzed data from 6,000 blind tastings – a lot of blind tastings! I quote from their findings:
“Individuals who are unaware of the price do not derive more enjoyment from more expensive wine. …we find that the correlation between price and overall rating is small and negative, suggesting that individuals on average enjoy more expensive wines slightly less….”
More recently, Ashton has taken another look at Goldstein findings and provides data that reinforces them. Lecocq and Visser analyzed data from three data sets totaling 1,387 observations on French Bordeaux’s and Burgundies. They report similar findings:
In 9 tastings since 2012, the Lenox Wine Club has rated a 3 liter box wine its Club favorite six times, 2nd twice and 3rd once. The 3 liter boxes cost $16, equivalent to $4 per .75L bottle. And these tastings included bottles costing as much as $85.
b. Blind Tastings Actually Blindfolded (Using Tongue and Nose)
Being somewhat dismayed when members of the Lenox Wine Club thought a Dolcetto was a Barolo, I conducted a blindfolded test to determine whether members could tell the difference between red and white wines. 75% of them got the color right but the chances were pretty good inasmuch as we were only tasting two reds and whites.
Robin Goldstein (personal correspondence):
“I once did a version of the red/white test, also with blindfolds, and got similar results (about 75% accuracy). Just to mess with people, we threw in samples of beer as well, and almost everybody didn’t question that it was wine, and they answered either ‘red’ or ‘white’. Only 1-2 people out of about 40 said ‘hey, this isn’t wine, it’s beer!’”
Also, Calvin Trillin’s New Yorker article on this same subject is a good read.
So What Really Adds Value?
The evidence suggests most people cannot tell one wine from another when only using sensory organs. So what really adds to wine’s value? Like doctors’ recommendations, there appear to be significant wine “placebo effects” from brand, label, price, country of origin, high ratings, and the recommendations of wine store staff.
A couple of years back, I did a two-part series on fashion. I defined fashion to be the premium people will pay over “necessity costs” for a product. In 2009, I estimated global entertainment expenses by category. The leaders were, not surprisingly, drinking, drugs, and sex. Drinking overall was $1.6 trillion with wine purchases $250 billion annually. Wine “fashion” might be defined as the difference between the wine “necessity price” for a good bottle of wine (a good 3-liter box wine sells for $16 ($4 per 750ML equivalent) and the average retail price for a bottle – say $15. So maybe $100 billion of global wine sales are the “fashion”/“placebo” premium.
What Are Expert Tasters Really Scoring?
Robert Hodgson has developed a pretty rigorous and convincing test on human abilities to distinguish between wines. Using 4 tastings of 10 glasses each with 3 wines poured from the same bottle in each tasting, Hodgson concludes from the spread of scores on the same wines that only 10% of tasters can distinguish between wines.
Hodgson’s work does suggest that there are some people who can tell quite minute differences between wines. Differences, yes, but at most tastings, judges are asked to score wines on the basis of what they like. But is this really what “expert tasters” do? Consider the nine Wine Spectator (WS) tasters listed in Table 1 below. Each has a region they cover, and most of them have been rating wines for many years.
Table 1. – Wine Spectator Raters and Regions
Do what wines they like really come into play? Harvey Steiman tastes Australian wines. WS has ratings for 14,150 Australian wines. If Steiman tasted them all over his 31 years at WS, he would average 456 wines per year. James Molesworth is their South African taster. WS has rated 6,220 South African wines. To rate of them, Molesworth would have to average 478 tastings a year over his 17 years at WS. In all likelihood, each WS expert tastes even more wines because I believe WS does not rate wines that are very disappointing to the tasters.
Sidebar on Wine Descriptors
I offer some quotes from WS tasters on their favorite wines below. The words in bold were featured in Richard Quandt’s entertaining article: “On Wine Bullshit: Some New Software?” The words in CAPS are candidates for an updated version of the article.
“Light, racy and aromatic, with depth, offering mineral notes of crushed wet stone before delivering beautifully restrained apple, pineapple and citrus flavors that play against a lean, sinewy frame. The finish doesn’t quit.” – Harvey Steiman
“This is brilliantly focused, tangy and balanced, offering towering flavors of pear, pineapple, cream and floral on a pinpoint of acidity, broadening into an expressive, explosive finish. Delivers tremendous precision, persistence and expression.” – Harvey Steiman
“Openly rich and smoky, yet graceful and polished, with roasted fig, honeysuckle, baked apple, marshmallow and cinnamon-tinged spices, gaining depth and length.” – James Laube
“An unctuous, creamy white, packed with apricot, peach, citrus, floral and spice aromas and flavors. An underlying mineral element leaves a savory impression on the finish, accompanied by notes of spice and honey. Balanced, broad-shouldered and impressive.” – Bruce Sanderson
“This rich and creamy white is laced with lime blossom, peach, citronella, hazelnut and toasted spice aromas and flavors. Balanced, with bright acidity focusing and driving the long finish.” – Bruce Sanderson
“A haunting combination of creamy texture and bright structure sets the pace for floral, peach, apple and hazelnut flavors in this balanced white. Stays fresh, with the aftertaste homing in on orchard fruit, spice and mineral.” – Bruce Sanderson
“Dark, dense and spicy, with dark berry, mineral, cedar, sage, roasted caraway and tobacco flavors….” – James Laube
“Pure, ripe, riveting aromas of blackberry, blueberry, currant, sage, cedar, espresso and mocha lead to a full-bodied palate, firming beautifully on the finish, where the flavors glide along and gain traction.” – James Laube
“This pure, racy red exhibits a dark side, with layers of spice, graphite and maduro tobacco to the crushed raspberry, blackberry and concentrated cassis fruit.” – MaryAnn Worobiec
“A touch of oak in the aroma leads off, masking the cherry, leather, tar and tobacco flavors for now.” – Bruce Sanderson
“Focused, elegant, powerful and layered, with plum, currant, floral, spice and coffee flavors that come together seamlessly and persist…. Has presence and deftness in equal measure….” – Harvey Steiman
“Offers ebullient raspberry and black cherry flavors that are ripe and seductive, with spice, loam and sandalwood details. Notes of cassis, raspberry jam and red licorice crescendo on the long finish, with very refined tannins.” – MaryAnn Worobiec
“Stunning, with aromas of warmed yet fresh Jonagold apple and white peach, followed by creamy notes of papaya and mango, mixed with lively mirabelle plum and green apple fruit. Flashes of verbena, quinine and lemon shortbread fill the lengthy finish, revealing a backdrop of wet stone at the very end.” – James Molesworth
I ask you, how long can you write this stuff before going stark, raving mad? I strongly doubt that WS tasters actually “like” the 400+ wines they have been paid to rate annually. More likely, they are quite sick of them. So what are their ratings based on? My guess is they rate on the basis of how well a wine fits a particular genre – how representative the wine is of that genre?
Molesworth covers the Bordeaux region of France. Does he rate Bordeaux on the basis of one he likes? It is far more probable he rates them against his concept of how a good Bordeaux should taste.
For these reasons, I am always suspicious of the ratings of so-called “experts”. The ratings of “amateurs” probably provide a more accurate picture of what wines they truly like or dislike.
Wine Tastings – The Two Large Elephants in the Rooms
Tastings continue with little attention to a couple of problems with almost all of them.
a. Most Wine is Consumed with Food
We have all seen the pictures of wine tasting settings with the glasses all lined up on the table. Most certainly, this is how most tastings are conducted. The problem is that most wine is consumed with food, and adding food will significantly change taste profile of most wines. And yet at most wine tastings, crackers or some other “neutral” food might be served so you can “cleanse your palate.”
Stark, Austere, Sparse Wine Tasting Setup – No Food!
Source: Setup for tasting at American Association of Wine Economists’ annual meeting, Stellenbosch, June 2013.
This makes no sense. If most wine is consumed with food and food changes the taste profile of wines, why are most tastings done “in isolation”? Sipping wines loaded with tannin (like Barolos) is not fun. Such wines taste much better with food.
In tastings of the Lenox Wine Club, we tried to get tasters to consume wine with food before scoring the wines. It did not work. The tasters viewed the tastings as “work” and wanted to complete them before food and socializing at dinner.
Despite the fact that few food-wine blind tastings have actually been conducted, numerous people have come up with “rules” on what wines to serve with different foods. For example, Ray Isle, writing for, not surprisingly, Food and Wine Magazine has come up with 15 “rules” for wine/food parings:
- Champagne is perfect with anything salty.
- Sauvignon Blanc goes with tart dressings and sauces.
- Choose Grüner Veltliner when a dish has lots of fresh herbs.
- Pinot Grigio pairs well with light fish dishes.
- Choose Chardonnay for fatty fish or fish in a rich sauce.
- Off-Dry Riesling pairs with sweet & spicy dishes.
- Moscato d’Asti loves fruit desserts.
- Rosé Champagne is great with dinner, not just hors d’oeuvres.
- Pair a dry Rosé with rich, cheesy dishes.
- Pinot Noir is great for dishes with earthy flavors.
- Old World wines and Old World dishes are intrinsically good together.
- Malbec won’t be overshadowed by sweet-spicy barbecue sauces.
- Choose Zinfandel for pâtés, mousses and terrines.
- Cabernet Sauvignon is fabulous with juicy red meat.
- Syrah matches with highly spiced dishes.
Where did these rules come from? Were there any blind tasting tests to back these up? Apparently not. My reaction – they are nonsense!
And then there are the ever-popular wine food paring dinners: light whites with hors d’oeuvres, with wines getting heavier as you get to main course and then a sweet wine with dessert. Again, nonsense. And as I have written, nothing is learned.
b. No Correlation among Tasters’ Ratings
Most tastings involve tasters proving their ranks and/or scores of the wine tasted. The ranks/scores are then totaled, and winners are declared. Far less attention is usually given to whether tasters agree on their ratings. However, when attention is turned to this issue, the finding in nearly all cases is there is little agreement to the point where statisticians conclude the results have no significance.
There is no perfect measure of taster agreement, but the Kendall’s W is probably as good a summary statistic as there is. The Kendall W’s for a number of tastings is given in Table 2.
Table 2. – Kendall W’s for Selected Tastings
The Kendall W’s for most of these tastings is low enough to suggest considerable disagreement among tasters on the wines. But for the Whites tasted at Paris, there does appear to be at least some agreement (.5348 where complete agreement would be 1.0000). The devil is in the details. Table 3 provides the actual rankings of the 8 judges for the tasting. There was unanimous agreement that the David Bruce was the worst and near unanimous agreement that the Chateau Montelena was the best. If these two are eliminated, the Kendall W falls to 0.2953.
But look again at the rankings of these “expert” judges. MD gives the Chateau Montelena a 9 ranking. And look at the difference on the Chalone and Beaune Clos des Mouches. And for lower ranked wines, CDM gives the Veedercrest a 2 ranking.
Table 3. – Ranking of White Wines Tasted at Paris
Source: Taber, op. cit.
Richard Quandt and Orley Ashenfelter have been analyzing wine tastings for more than 30 years. Quandt did the statistical analysis for the Princeton judgment. He concluded:
“…the rank order of the wines was mostly insignificant. That is, if the wine judges repeated the tasting, the results would most likely be different. From a statistically viewpoint, most wines were undistinguishable. Only the best white and the lowest ranked red were significantly different from the other wines.”
Quandt’s statement on the Princeton tastings would probably apply to most tastings. Is there more agreement amongst expert tasters? Maybe. But there were expert tasters at Stellenbosch and there was little agreement there.
- When eyes, tongue, and nose are used to taste wines, price makes no difference: people like inexpensive wines just as much as expensive wines.
- Limited evidence suggests further that when tasters are blindfolded (using only tongue and nose), 25% of tasters cannot tell the difference between red and white wines.
- Like women’s perfume, pocket books and sun glasses, wines are a “fashion good.” That is, external signals, such as price (higher better), brand name, shape and color of bottle, ratings, wine salesmen, and country of origin all add value. Rough calculations suggest that of the roughly $250 billion of wine sales annually, $100 billion is attributable external “fashion” signals. People are starting to realize they can find good wines for $10, but as long as people are ignorant, vain, and insecure, there will be a “fashion premium” for wine.
- The tasters of the big wine rating houses probably don’t “like” the wines they rate. Instead, they are probably rating them against a standard they have developed for the particular wine genre (varietal or region) that they are rating.
- Despite the fact that most wine is consumed with food and food changes the taste profile of wines (nobody likes to “sip” Barolos without food), wine tastings are usually done without food. Then, without any empirical backing, rules have been developed for what wines will complement what foods. This is all nonsense! Blind tastings are needed to decide whether any legitimate rules can be developed for wine/food parings. The tastings should start with ratings of the wines by themselves. Then taste them with different foods and rate them again.
- It is extremely troubling that in almost all tastings, the scores of individual tasters are so different that there is no real significance between the wines when the scores are totaled.
- So what do we make of this conclusion, what does it mean? Several possibilities:
- Hodgson concludes that only 10% of people applying to be judges at a major tasting event can really distinguish between wines. That might mean that much of the tasters’ differences can be explained by their inability to make accurate distinctions between wines.
- A factor contributing to this hypothesis is that over the last 20 years, there has been a marked quality increase across the board in all wines.
- But this quality improvement also adds credence to another possibility – the disagreement among judges is primarily the result of judges having different taste preferences.
I think all three factors are at play. My ability to distinguish between wines would probably not satisfy Hodgson, but when using my eyes, tongue, and nose, I do have wine preferences.
 Robin Goldstein, Johan Almenberg, Anna Dreber, John W. Emerson, Alexis Herschkowitsch, and Jacob Katz, “Do More Expensive Wines Taste Better? Evidence from a Large Sample of Blind Tastings”, Journal of Wine Economics, vol. 3, no. 1.
 Robert Ashton has done considerable work on the agreement/disagreement issue – http://www.wine-economics.org/journal/details-content/volume-8-2013-no-2/.