CoffeeWars 4: Lessons Learned

• Guidelines: We should supply judges with some better information about how to do their jobs. This will enhance consistency. For example:
  • How should one rate "Acidity?" (is 10 more acid? Is 10 good?)
  • How should one rate "Overall?" (or, should this category be eliminated, and derived from others)
  • What will be done with non-integer ratings? (or zero, for that matter, on a scale from 1 to 10)
  Just like a wine tasting, we should publish the criteria that we're trying to judge.
• More Time: Golly, if we're going to sample this much (or more) coffee, we'd better plan to span this event across a couple of days. That way we can have a better chance at accepting all the entries people bring, and time won't be such a desperate issue.
• Sterner and More Explicit Rules: We wound up disqualifying two entries; this should not have to happen. The announcement should be more definite about what constitutes an acceptable entry. Specifically:
  • Beans only. No pre-ground.
  • Enough must be submitted to brew four pots (both for the sake of aroma, and also in case there is an accident or run-off)
  • Entries will not be returned, unless you both let us know, and also make arrangements to get them back.
• Better Entry Management: The blind tasting was good, but could have been handled better; each entry should be placed into a numbered container when it is handed around, instead of just passing the bag it came in, because this allows judges to identify their own entries, and introduces potential for bias.
• Better Record Keeping: It would have been good to capture more data (i.e., what order were the coffees brewed and tasted), and also to be more organized with its collection (had I known I'd have 120 ballots to process, I'd have had a spreadsheet running on a laptop at the event, so I could do all my tabulation live).

Obviously, this isn't the last round of thought that will go into preparation for CoffeeWars 5. We look forward to hearing anyone else's suggestions.