The Process
CRITERIA
Judges score each entry according to the criteria as stated on the website (for specific criteria please see the webpage for the relevant category). These criteria need to be addressed in the full description section (“Part B”) of an entry.
ANALYSIS OF SCORES
For each category, scores from individual judges are collated and a spreadsheet created. In addition, to help control for inter-rater variability, those entries receiving first and second places for each judge are taken into account, as are written comments accompanying the score sheets.
TELECONFERENCES
A teleconference attended by the judging panel, the awards coordinator and the event coordinator is held for each category of award. Judges’ scores for each of the criteria and related comments (written and verbal) are compared and considered during the teleconference. It is during this part of the process that provisional decisions are made regarding those to be awarded or sent letters of commendation. If judges have specific questions to put to the referees before or after the teleconference, these questions are followed up by the Awards Manager.
EMAIL CONFIRMATION PROCESS
For each category, the provisional decisions are circulated by email to the judging panel for confirmation or further discussion. This is particularly important where any judges were not available for the teleconference.
REFEREES
Entries are asked to nominate 2 referees. Referees are not routinely contacted regarding award decisions made by the panel of
judges. If judges specifically request clarification about the content of an entry, referees are contacted by the Awards Manager.