Referee Improvement Survey Analysis

By Kym Couch

At the beginning of every season, there is a struggle to recruit and train enough referees for the season’s tournaments to run smoothly. In the past, USQ has made steps to try to circumvent this issue by requiring each team to have a certain number of certified referees. However, this certification only requires these players pass a written test; it does not require them to pass a field test, have any experience refereeing, and does not even require them to ever referee. This results in a shortage of referees and an even greater shortage of qualified referees.

The pain of not having enough referees is felt by every team that attends an official tournament. Teams that offer a larger number of referees are actually penalized for doing so because these refs may be forced to referee their own games (I’m not even getting into the ethical issues of that) or may be exhausted after pulling double duty.

This season, the International Referee Development Program (IRDP) was formed in order to fill gaps left by USQ’s Referee Development Program. USQ’s decision to not partner with the IRDP has led to a refereeing crisis as USQ’s human capital was split in half. As a result, multiple tournaments this season have been forced to become unofficial (either before or after the fact), and there is a real concern that the lack of referees could severely damage our sport.

In a discussion on the Facebook group #IQAForums, started by USQ Rules Team Manager Clay Dockery, community members came together to propose possible solutions or, at the very least, small steps that can be made by USQ to make serious change in our current referee program.

In order to have a more organized discussion, Ra Hopkins, USQ certified head referee and Mission Blues player, and I chose 30 of these ideas to be reviewed and voted on by the community on a scale of 1 (This is not a priority to me) to 5 (This is a high priority to me). The survey had a total of 83 respondents.

Part 1: Who Took the Survey?

In the introductory section, respondents were asked for their name, referee tests passed for USQ, referee tests passed for IRDP, and their region.

The majority of respondents (65%) have passed some form of referee testing for USQ, while 24% have passed some referee testing for IRDP. Twenty-four percent is impressive for IRDP considering the fact that only two of those respondents are located outside of the United States, despite IRDP certification being of no use in USQ official games.

Each region had at least seven people respond to the survey with the most surveys hailing from the Northeast, and the South and Northwest providing the fewest (understandable considering that they are the least-populous regions). There was one respondent from Europe and two from Oceania.

Part 2: Funding Changes

The first section on the survey reviewed funding changes. The only item in this section to make the top 10 suggestions by average score was “A travel stipend for non-player referees to visit tournaments that are over x miles from their home.” In fact, every other item in this section were the four lowest on the entire survey with averages ranging from 2.37 - 3.2 on the five-point scale.

The travel stipend for referees attending tournaments as non-players was highly mentioned in comments as something that people feel would be a good use of the money USQ currently has through membership fees. Only seven respondents (9%) voted it as a 1 or 2.

There was a split response to all three questions regarding payment of assistant referees. Multiple comments said things like “payment for assistant and snitch refs should be higher than 2 or 3 dollars” and “Sure, it's nice to have extra cash, but that 2 dollars (or whatever small amount) isn't going to tip the scales when someone is thinking about being an assistant referee.” Although it was split, comments suggested that more people felt that it was important to raise the standards of assistant and snitch referees before we begin compensating them.

Part 3: Training Changes

Training at fantasy tournaments is something that people are highly interested in, as there was an average rating of 4.46. The only negative response was people saying that fantasy tournaments are not competitive enough to practice reffing at (but perhaps these people have not attended Snow Cup).

Training materials on what is considered disrespectful to an official (section 8.8.1. in USQ Rulebook 8) also received a high rating and comments often mentioned how poorly referees are treated in this sport. Standardization of what is and is not a cardable amount of disrespect received an average rating of 3.9 with weekly or monthly webinars for referee growth and development following close behind with an average rating of 3.8.

Part 4: Testing Changes

The highest rated suggestions for testing changes involved the field testing process, specifically how it is tested and how the tester communicates with the testee. An amazing 92% of respondents rated “require field testers to email a detailed report of field test results within two weeks of someone’s test” with a 4 or 5 on the five-point scale.

Another notable suggestion was regarding additional training, which includes video instruction. Although it would take some time and planning, it is something that is definitely within reason for a referee training program that wishes to be independent.

Part 5: Team Requirements Changes

USQ’s current system of referee requirements requires that each team have one head referee, one snitch referee, and two assistant referees pass the written testing. However, there are no regulations on the number of games that must be refereed per tournament or season.

A popular alternative to the current system is something known as the “Tiddly Method.” Developed and named after Tad Walters, the Tiddly Method requires that each team attending a tournament provide a certain number of referees, represented by various points, in order to ensure that the tournament may run smoothly and with enough referees for every game. Although two comments in this section negatively referenced the Tiddly Method, this is not reflected in the polling. Both questions that suggested requiring teams to provide referees for tournaments received over 68% positive response (a rating of a 4 or 5).

Part 6: Organization Changes

Organization changes mostly dealt with lists of referees, but this also included the highest rated item on the survey. With an average rating of 4.61 on the five-point scale and 91% of respondents rating it positively (4 or 5), a list of tournaments that may provide field testing if enough interest is received is, on average, the most important item to the community. This is another simple solution that USQ can enforce that can strongly increase users’ confidence and increase the number of persons attempting field tests.

Although not quite as important, there was also a positive response for public lists of referees. This is something that has been announced as forthcoming, but we have not yet seen the result of USQ’s efforts on that front. It will be interesting to see how they do providing it as we move forward.

Part 7: Review Changes

Referee reviews are a benefit to our sport that I often feel are not utilized properly. Currently, referees have to request a summary of their reviews from an RDT member in order to receive a review. This is not very efficient for RDT and seems to result in many referees never knowing if they have received any reviews, let alone what the content of these reviews are.

The majority of respondents feel that reviews should be available to referees in their full extent. This is an alternate system that is clearly supported by the community.

In keeping with the current system, however, respondents felt that RDT should be required to compile a report of review content any time reviews are received.

Finally, there was a proposal to require referee reviews for all official games. This would increase the number of reviews and help balance out the outliers to ensure more statistically accurate reviews. Although this was not clearly supported across the board, disagreements came with what the penalty would actually be. If RDT was able to figure out a penalty for not filling one out that was more agreeable (such as not being able to register for a tournament), people seem to be on the side of requiring referee evaluations.

Part 8: Other Changes

The lowest average score in the Other section was a 3.63 on the request that tournament directors be required to submit all referees and snitches (excluding goal judges and scorekeepers) to the USQ website. This was a controversial question because although many feel it is important to know which referees and snitches were involved in officiating a game, others felt that it was asking too much since tournament schedules tend to change at a moment’s notice.

I personally feel that knowing which officials are assigned to a game is extremely important. It would help with referee accountability, pride (we could easily keep stats on how many games each referee has reffed in a season, even as a snitch or assistant referee), and promote more accurate reviews. If this information was publicly available (or at least available to tournament directors), then finding the name of a referee you would like to review would be easier far easier, which would lead to an increase in the number of reviews. It would definitely involve some work on the website (or, god forbid, a spreadsheet to be used as an official source of information), but it would immensely help out the referee crisis we are currently seeing.

An earlier rulebook release, although admittedly not easy, is another change with overwhelming support. The proposed date on the survey was May 15 as this would allow for just under a month after World Cup for it to be completed. Although Rules Team has a lot to do in preparation for World Cup, they are working on the rulebook throughout the entire year. If they were to complete all changes by World Cup and then allow for nearly five weeks for editing, a release date of May 15 is doable. A release date of May 15 would mean that summer fantasy tournaments can serve as training ground for both players and referees to familiarize themselves with the new rulebook. This means that USQ official tournaments could start much earlier and with much more success, and the season could truly start on July 1.

USQ does not currently endorse any one specific scoresheet, which results in many different scoresheets being used when tracking a game’s score. Referees such as Nicolas Kubicki support a standardized scoresheet that includes spaces to write down the entire officials team because “A standardized scoresheet would improve every game since people wouldn't have to learn a new sheet every time.”

Although a very small thing for USQ to do, strongly recommending that every official game uses the same scorecard could significantly improve every tournament’s consistency. It could also make player stats something that can reasonably be tracked for tournaments rather than every other tournament having them. A standardized scoresheet would also make it possible for a requirement to submit the entire referee team. A standardized scoresheet received an average score of 4.11 on the five-point scale. It is notable that Martin Pyne’s scoresheet, which provides spaces to write down the officials team, was mentioned twice.

Part 9: Final Comments

Training was by far the highest rated section on the entire survey. Of the 44 respondents who did not classify it as first priority, 24 of them classified it as second or third priority.

Testing changes was the second highest with 16 selections for first priority and 20 selections for second or third priority. Funding, although falling after training, testing, and team requirements for first priority, was the highest voted second or third choice with 33% of the vote.

Paragraph by Ra Hopkins

The data shows that training changes are by far the most important to respondents. Referees want to improve and want chances to improve. They would also highly benefit from increased feedback, and they are motivated to use said feedback to improve as shown in the comments and results. Also frequently mentioned in the comments was a desire for increased opportunities for field testing, as well as frustration at the lack of opportunity for certification and (by referees failing the field test) opportunity for improvement. Increased training materials in any form would be universally welcomed. Additionally, there was a fair amount of support for the current head referee pay ($20 for USQ referees), a travel stipend for non-playing head referees, and continued training for all referees (including already certified referees) to improve standardization and consistency in all areas of officiating.

Comment Review:

At the end of each section was an open comments space where respondents could input their reasoning for how they rated items, additional ideas, and recommendations. Although you can read all comments in the summary, I thought that an interesting way to review what was regularly mentioned would be to submit the comments to Wordle, a website which increases the size of words based on the frequency that they are used. In order to represent the comments properly, they have been split into two separate Wordle photos. The first illustrates the “positive” statements, and the second shows the “negative” statements. A few words were removed so as to not be misrepresented. These words were Think, Referee, Ref, and Refs.

Positive Comments Word Cloud:

Negative Comments Word Cloud:

Interesting facts about comments:

• IRDP was mentioned 19 times.
• 17/19 mentions of IRDP were “IRDP does this already/IRDP did this/USQ should work with IRDP/Do it like IRDP” or something similar (the other two were comparing approachability and ref distribution)
• Respondent said “I want to help with this!” two times

Conclusion:

The data implies that the changes the community wants are relatively small. In the initial discussion, very few big changes were recommended. Steps such as a list of tournaments that may provide field tests and providing referee training events at fantasy tournaments were the items with the strongest support. Larger changes such as requiring referee reviews or paying assistant referees were the suggestions with highly split voting.

Every change suggested is feasible for USQ to implement, and nearly all have majority support from respondents. I cannot wait to see what the USQ Officials Manager, Brandon Kreines, takes from this survey, and which suggestions are implemented.

If you would like to see the full results of the survey you can see a summary here and you can see the unabridged responses here. If you would like to review the questions asked on the survey and help text (which cannot be seen on the summary), you can see the full survey here (please do not submit new responses). Additional facts and statistics can be found here.