CoffeeWars 4: Results

Judges were asked to rate the brews in a variety of categories on a scale from 1 to 10. For the purposes of scoring, all non- integer results were rounded up to the next integer. Also, people who submitted out-of-bounds scores (like 0, or 10+) were corrected to the nearest legitimate rating. All derived statistics (such as average or standard deviation) are based on the number of valid entries: if a judge left a category blank, this did not count either for or against the entry; it merely increased the weight of each score that was received.

Bang-for-the-buck (highest overall score for an entry, divided by the price per pound) was only calculated for those entries that included cost information, obviously.

Two entries were disqualified:

• Baby's Jet Black (entered by Dean): The considered opinion of the judges was that this coffee was flavored. Flavored coffee is not allowed. The entrant protested, and produced real live marketing literature from the roaster indicating that this was an unflavored brew. This was, therefore, a controversial decision, and we had misgivings about it: nonetheless the decision stood.
• Jacobs Kronung (entered by Wolfgang aka seele): The intent of the rules was to permit only entries in bean form, and this was pre-ground. It would have been disqualified on that basis, except that another entry, which was clearly within the spirit of the rules (because it was home-grown!) arrived in that form also. So we were going to try it, until someone spotted the Kraft Foods logo on the bag (a nearby German speaker confirmed that we were not misinterpreting the label). This pushed things over the edge: CoffeeWars is about small-batch artisan roasts, not mass-produced corporate offerings. Sorry, Wolfgang. Maybe Kraft makes excellent coffee in Europe, but we wound up disqualifying your entry.

Truthfully, we would rather not have had this happen. It's not our objective to bar anyone from entering the competition, but we had way more coffee than we could accept, even without bending the rules to allow more in. Next year's announcement will be more explicit about what constitutes a valid entry, and we apologize to those who went to the trouble of bringing their coffee, only to have it go unsampled.

Entry #13 (Prince of Darkness) deserves some special mention. The entrant, JP (not that JP), was visiting his favorite roaster, Oso Negro, in Nelson, British Columbia. That's in Canada, for those of you following along at home. He described CoffeeWars to them, and they got so excited that they donated his entry. This, truly, was amazing: our first entry backed by corporate sponsorship and our first indication that CoffeeWars is known abroad. Amazing, and exciting. Thanks, Oso Negro, for being so cool! Also: it was good coffee!

This table presents the various entries in order of average overall
score.  In the event where overall scores tied, "Taste" was used as
a tiebreaker category.  There is one mysterious entry, to which the
ballot form seems to have disappeared.  It didn't place too highly,
so I don't think there is cause for concern, but nevertheless, I 
wish I knew what it was.  The ballot submitted in Jay Dyson's name
may have been for two entries; if that's the case, this is most 
likely the first of the two.

For better or worse, the individual comments made by the judges have been omitted. Sometimes they were inscrutable (such as "meh." or "shiny!"). Sometimes they were insightful ("nutty bean flavor."). Sometimes they were harsh ("is this water?"). Likewise, they sometimes contradicted the scores on the ballots (one judge made a comment praising the excellence of the entry's aroma, and then rated both bean and brew aroma low; another wrote that the coffee was very good, but then gave it low marks overall). I don't know what to make of this information, except that we will endeavor to make the scoring process a little more rigorous next year. One metric of accuracy, perhaps, is the standard deviation, and we should strive to supply guidelines where it grows so large that it encompasses, say, 50% of the entries.

This table presents the various entries in order of our bang-for-
the-buck metric.  Bang-for-the-buck is a derived statistic, calculated
by dividing the overall score by the cost per pound (in US dollars).
Entries for which no price data was submitted are excluded, of course.
In the event of a tie, the raw overall score was used as a tiebreaker.

Truly, this is interesting data. First, the overall winner was also the lowest score, in terms of bang-for-the-buck. This just goes to show you that you get what you pay for, but there may be a point of diminishing returns. Similarly, the second place entry overall was also the highest bang-for-the-buck. This just goes to show you that excellent coffee does not have to cost an arm and a leg. Sometimes just an arm is sufficient. Or a leg. Whichever you'd miss less.

In other categories, the winners are as follows.

There's a moral here, too: the winning entry was the winner overall, in all categories. There wasn't just one score that tipped the scales.

Some have hypothesized that the reason that entry received such off-the-scale marks was just that it was the last entry to be sampled, and the judges were simply going crazy with the emotional release of finally finishing their hard work. Perhaps, to some small degree, this is true, but I think not greatly. This was some outstanding coffee, and it richly deserved to win.

OK. That's it for the results summary. All the data is available for download from this site, in case you want to do your own calculations.