The Red Wine Challenge – But It’s Bota Box Again!

by , Morss Global Finance


The Lenox Wine Club just had its 9th wine tasting dinner. It was special – a challenge dinner! As I detailed in an earlier piece, a leading restaurateur/caterer in my home town was quite skeptical of how well box wines have been doing in our tastings (5 first place, 2 second place and 1 third). So I threw down the gauntlet! I said you choose the grape/varietal and a favorite wine of your in that category. So he chose Syrah and a Californian favorite of his. I chose the other 4 wines and one of them was a box – a Bota Box Shiraz. The Bota won. His selection while coming in 2nd, effectively tied with 2 other wines. The score and price were not correlated: the most expensive wine came in last. And one other interesting note: the rakings and scores differed significantly.

Tasting Details

There are three basic goals for our tasting dinners:

  1. Taste wines similar enough to make comparisons meaningful;
  2. See if price matters, and
  3. Have a good time.

Scores and Rankings

We taste 5 wines. In past tastings, we have “rated” wines 1 to 5 with ties allowed. But ratings do not allow one to register differences in the intensities of likes and dislikes. Which should be used and does it matter? As was explained to me by Neal Hulkower and Dom Cicchetti at the American Association of Wine Economists’ annual meeting in Stellenbosch last June, it can matter. So for our recent tastings, we used scores with tasters given the following scoring instructions: 60-70 = Poor/Unacceptable, 71-79 = Fair/Mediocre, 80-89 = Good/Above Average, 90-100=Excellent. From scores, we can get rankings.

How Good Are The Tasters?

While there were 5 wines, we actually tasted one wine in two separate glasses.  What is this all about? As I have described in an earlier posting, Robert Hodgson has his own winery and has been troubled by erratic ratings his wines had been receiving from judges at tastings. So he came up with a way to rate potential judges. The key to his method? Have the candidates do blind tastings that include more than one glass of the same wine. If the candidates do not score glasses of the same wine nearly the same, they are not competent to judge wines. Hodgson’s suggested overall scheme is quite rigorous: candidates must do four blind tastings of ten glasses each. We used his methodology in a less rigorous way: two glasses poured from the same bottle: we can’t be as sure this will single out incompetent tasters but the results are “indicative”.

The Syrah/Shirazes

1. Heartland Shiraz Langhorne Creek Directors’ Cut, 2010

With a 93 Wine Spectator (WS) rating, should be a great Australian challenger to the Ojai.

2.  Lapostolle Syrah Colchagua Valley Cuvée Alexandre Apalta Vineyard, 2009

This Chilean selection got 91 from WS.

3. Lindemans Shiraz/Cabernet, 2013

I included a Shiraz-Cabernet blend because I have found this blend normally does quite well against wines where Shiraz is the dominant grape. This Australian selection was not rated by WS.

4. Ojai Shiraz, 2010

This Californian wine was not rated by WS for this year. However, it received ratings of 88 and 94 in 2009 and 2008, respectively.

5. Bota Box Shiraz, 2011

Like the Ojai, this is an American Shiraz.

The Results

The results of our blind tasting are presented in Table 1 with the wine with the highest score on the left. One of the two Botas won, followed at some distance by the Ojai and Lindemans. The most expensive wine got the lowest score.

Table 1. – Syrah/Shiraz – Blind Tasting Scores* The Botas come in 3 liter boxes. The price is for a 750ML bottle equivalent
Click to enlarge

Scores Versus Ratings

As mentioned above, there is a real issue on whether you score or rate wines. So we have started to do both to see if it matters. To date, it has not. But in this tasting, it really did. The Heartland that came in last when the wines were scored came in 3rd when the wines were ranked. And the Bota (2) that tied for last when scored was a distant last when wines were ranked (Table 2).

Table 2. – Total Scores and Rankings (Reversed)** To make scores and ranks easy to compare, the ranks are reversed with the highest ranked glass getting a 6 and the lowest getting a 1.

What explains these differences? Two primary possibilities:

  • Differing intensities of likes and dislikes: the Heartland was really disliked or
  • Tasters used random “starting points”. Table 3 gives the average scores (starting points) of our tasters.

Table 3. – Average Scores of Tasters

Note AV’s high average score. Is this because relative to the other tasters, AV really enjoys the wines more than the other tasters? Or is this somewhat random? Might he have just as well scored around an 80 mean? And JR: a 72.5 score is supposed to mean a wine is “fair/mediocre”. AV gave Heartland his lowest score, so it is not surprising that the Heartland position improves when we switch to ranks.

Testing Tasters’ Competency

The Bota Box “spreads” are given in the right hand column of Table 1. They are not impressive. 7 of the 11 tasters gave the two Bota glasses different scores of 5 or more points. TB actually scored one glass 85 and the other poured from the same box a 65.


What do we make of these findings and how definitive are they? Findings are more definitive if all tasters agree. Perhaps the best statistic for measuring taster’s agreement is the Kendall’s Tau: a higher number indicates greater uniformity among tasters. The Tau for our tasting was only 0.064 suggesting virtually no agreement among tasters.

So are we to conclude that our scores and ratings are just hogwash? A quote from Gilda Radner is in order: “Never Mind”.

Share this Econintersect Article:
  • Print
  • Digg
  • Facebook
  • Yahoo! Buzz
  • Twitter
  • Google Bookmarks
  • LinkedIn
  • Wikio
  • email
  • RSS
This entry was posted in wine and tagged , , , , , , , , , , . Bookmark the permalink.

Make a Comment

Econintersect wants your comments, data and opinion on the articles posted.  As the internet is a "war zone" of trolls, hackers and spammers - Econintersect must balance its defences against ease of commenting.  We have joined with Livefyre to manage our comment streams.

To comment, just click the "Sign In" button at the top-left corner of the comment box below. You can create a commenting account using your favorite social network such as Twitter, Facebook, Google+, LinkedIn or Open ID - or open a Livefyre account using your email address.


Let me respond to a number of comments:


Millman says: Most tasters, if they are honest, do not much like the world's greatest wines.


Elliott: What makes them the greatest? The analogy to whether we prefer Renoir to Cezanne is just as problematic. Barolos are very tannic - not immediately tasty - does that make them great wines? I am a great believer it is the eye of the beholder. And I would never claim "greatness" for any wine.


Now - I want to be absolutely clear here: there are people for whatever reason really do enjoy Millman's "great" wines. And many of them can distinguish what others of us would say are indistinguishable differences. And I will say to them - great! Enjoy yourselves. It just becomes a bit much when they start looking down their nose at everyone else.


When I recently reported our findings on the Wine Spectator tasting notes, these people came out of the woodwork and attacked me with real anger. My sense is they really want to believe wines they spend $90+ per bottle for is in some way intrinsically superior to less expensive wines.....


On high and low scores. I recently wrote a piece that in essence says all wines I drink  are good - they are in the boring middle. Far more interesting to me are wines with very high and low scores.


Question: do the same tasters have low spreads on the test wine most of the time or does it vary from tasting to tasting?


Elliott: They are almost always the same.


On Kendall Taus - They are all very low, even at the famous tastings. Because of differing taste preferences, even "experienced" tasters do not agree.


At our tastings, everyone can taste in whatever order they like. I always encourage people to change the order around.



Professor Morss--Robert Millman again, the guy who complained about mixing sweet with dry Rieslings. Several comments pertaining to this rather inconclusive, perhaps will should call it Ratner-Results tasting. I am surprised that no one brought an affordable Syrah from its birth place--the Northern Rhone. There are moderately priced Crozes Hermitages readily available. I will now surmise that whoever is responsible for producing the Bota boxed wines has a good understanding of the average palate--an almost instinctive attraction to red wines with good fruit, soft tannins, low acidity and just a touch of residual sugar. This is the wines-as-food approach which has tremendous merit for everyday wine consuming purposes. I never tasted the Bota wines. I am sure I would find them most agreeable. Wine that is intended to be tasted for itself should be understood, sought out and experienced not for immediate soothing hedonic satisfaction. Most tasters, if they are honest, do not much like the world's greatest wines. On a hedonic scale, a well made boxed Cabernet will taste "better" than a demanding, tannic Cabernet based wine like the Dunn Howell Mountain or Chateau Latour. So are buyers of such wines fools or mere trophy collectors? Permit me to make a comparison: take most non expert people to a museum and show a Cezanne side by side with a Renoir. Which painting will most enjoy the more? Does this even require an answer? And isn a way they are quite correct: for sheer, immediate sensory pleasure the Renoir wines hands down. So is it snobbihsness to argue that Cezanne's painting are more profound and reach areas of mind and imagination that Renoirs rarely do?   I note that the Ojai Syrah received 4 low scores ("I just do not like this wine") and 4 relatively high scores. I am not surprised. The grower/wine maker Adam Tolmach has always had the northern Rhone as a model. He has never made a sensual style of Syrah. 


And another question:

do the same tasters have low spreads on the test wine most of the time or does it vary from tasting to tasting?


another question if I may.


Have you had a group of tasters who got a high Kendall's Tau?

Do you know of any tasting where the group got one?

Do tastings by professional/qualified wine judges get high Taus?


If even professional/qualified tasters generally don't get high Taus isn't it all just persnal taste at a given time in given circumstances and moods?


Perhaps scores ought be adjusted to the same mean for comparison purposes.


A few questions, :

1. does everyone taste in exactly the same order

2. is everyone eating the same thing immediately prior to tasting the same wine

3. is everyone about the same age (relates to desirability of sweetness at a given age).


Thanks for sharing these experiences and for featuring some Australian wines.(I'm from Sydney)


 @rdmill PS but thanks for your comments and in particular those regarding the structure of tastings and tasting a reasonable wine from the country of orgin of a leading style based on a varietal in any tasting of that varietal (if I have correctly interpreted your comment above.)


 @rdmill Blind tastings among those who claim to prefer the "demanding, tannic based wines" with multiple tastings of the same wine  would best determine the answer to the question. But my understanding from reading the various articles of Prof Morss is that such tastings don't show real uniformity of opinion on which is the "best" wine.


And if "most tasters, if they are honest do not much like the world's greatest wines" is such greatness based more on mystique or some archaic tasting guide which awards points for characteristics now judged as undesirable even by well recognised judges/tasters in a blind, double tasting process based on their scores after eliminating say those 50% with the largest spreads between identical wines?


So, for the great majority of us, why pay more, other than for an experience that is based on bragging rights rather than taste on our own palate?