Open Reining Futurity 2024
Metrics of Officiating:
What is the best way to analyze competition and identify how the competition is placed?
Was the winner scored properly and did the prize purse get distributed appropriately to all the hard-working owners, riders and horses on this journey?
Every year these questions circle around the industry. There is no doubt that Kaci and The Fireman outperformed the rest of them. This year the conversation is not about who won, but how 2nd through 29 was placed. Rightly so, 2nd through 10th pays out over $748,000 of purse money.
HOW ACCURATE IS REINING SCORING
How do we analyze the outcome of such a dynamic, quality of control, differentiating sport? Most of it comes from our individual viewpoint and some of it comes from what others have to say. The answer is we do not analyze the true performance of our sport. We only have the score the athlete receives from the officials.
The real question is: ‘How accurate is the score?’ For instance, if a rider scores a 72.5 and a 219, the industry knows what the score means, but how accurate are those scores? Should the score have been a 73 or 72? Or is the 72.5 most accurate?
HOW TO ANALYZE THE TRUE ACCURACY OF THE SPORT OF REINING
We don’t measure accuracy, but we should. In full respect to all the owners, breeders and professional riders and now the incoming audience, we need to be able to develop, manage and predict the performance of the officiating for the sport. Officiating should be done at an excellent level. The thrill of victory and the agony of defeat should remain in the saddle of the competitor and not the chair of the official.
Measuring officiating accuracy is possible, but it requires knowing where to look, which metrics to analyze, and ensuring a large enough sample size to make the analysis valid. Does any regulatory, organization or licensing group know how to do this? No.
This illustration is one example of how to measure accuracy. These are three key gauges that measure the same metric.
1st the combined averages of the individual officials (gauge to the right)
2nd the system on the round (in this case it was 2nd section open semi-finals ) (middle gauge) and
3rd the system when combined scores are tabulated (gauge on the left).
THE THREE TAKEAWAYS
This demonstrates that a system involving five expert officials result in better performance than any individual.
Additionally, the system is adequate as a stand-alone, such as in a given class or round.
The system is the strongest when, as in this case, a 2-round combined score is applied.
This series of gauges here uses a percentage basis, supported by a simple, readily available metric to measure performance. It can be expressed as a single number and one decimal point with reaching “1” as excellent performance. We are not aiming to measure perfect, but excellent. The measuring system works very well and is objectively measuring the officiating expert’s and the system performance in real time. As an example, we see numbers such as: 1.6, 1.8, 2.2, 2.4, 3.6, 4.2, 5.5 etc, where the lower the number, the better the performance.
Officiating: MANEUVER EVALUATION: USING THE SCALE
( cowboyoffice.com 2024 analysis source:nrha.com/results)
The maneuver evaluation (M.E.) analysis often sparks great discussion. We want experts and we want expert’s distinct and fine decisions on quality. It does take time, experience, confidence and commitment to become an advanced official for the sport of reining and be able to use the scale. We thank all of those that have done this hard work since 1985 and those that continue today.
The graph above shows the use of the M.E. scale across the entire Open Reining Futurity in 2024. As the rules of this game outline, scoring is from a -1.5 to +1.5 on .5 pt increments with 0 denoting an “average” or “correct” M.E. A correct M.E. of zero is basically neutral, no fault and not gaining credit, just correct as the sport dictates it to be done (NRHA 2024 handbook).
This graph displays
how much the scale was used
by 10 different officials
across 8 rounds of competition
that produced 635 runs scored
representing 24,540 M.E. decisions
that were made across 7 days
It highlights two key elements;
1- how much the quality has elevated in the sport and
2- how much the scale is not being used to its fullest potential.
WHAT IT TAKES TO PLAY
It is clear that +1.5 is not used to the level that it should be and can be used. With 635 runs scored, only 0.1% of all maneuvers were applied +1.5 as being excellent maneuvers. For anyone who watched the event, it was clear that there were many more “excellent maneuvers” than 0.1% which is 24 maneuver decisions out of 24,540. As a comparative, +1 was applied 4.7% of the time which is 1,143 maneuver decisions in the same event out of 24,540, and +.5 was applied 44.8% which is 10,984 maneuver decisions out of the 24,540.
To compete at the highest level, it now takes a horse with a score of 74 or higher, representing a 1.5 to 2-point increase in skill and talent compared to historical futurity levels. It takes a 75+ level horse to achieve major success, as this type of score allows a buffer for error in aged events for the exhibitor. In episode #33 of the Cowboy Office podcast, The Show, with the #2 all time professional rider, Shawn Flarida, he talks about the fact that reining horses come in three batches: The 71- 72 horse, the 72- 74 horse and the above 74 horse. The numerical analysis supports the accuracy of Shawn’s grouping in today’s sport.
IS THE TOP END OF THE SCALE BEING USED
The level 4 Finals in this year’s futurity exhibits interesting data. Using the scale graph below illustrates the lack of use on the top end of the scale. The L4 finals included 33 entries running pattern 10, which is an 8 M.E. pattern that demands 1,320 M.E. decisions, including 2 sets of circles, 2 sets of spins, and 4 stops, each ending with unique finishing factors; backup, rollbacks and hesitation.
In the L4 finals slightly over 75% of all M.E. were marked between +.5 and +1.
OPEN FUTURITY FINALS; SECTION 2 LEVEL 4: USING THE SCALE:
(Open Reining Futurity, section 2 L4 Finals, OKC, Using the scale: source: nrha.com/results)
When the competition gets deep, tough & condensed, using +1.5 becomes necessary to separate great execution of maneuvers.
+.5 M.E. was applied 12% more in the L4 Finals than the average of the entire event.
+1 is applied 16.6% of all M.E. in L4 Finals.
+1.5 is only applied 0.8%.
Do officials struggle to find and apply +1.5? Using Kaci's final run we can look at her cards and analyze this (graph below)
The entire sport has elevated in quality, What used to be a 0 is now a +.5 and a +.5 is now a +1 M.E. Penalty application or rule infractions have decreased dramatically over time as the quality of horsemanship and showmanship continues to rise. The entire M.E. scale is there as a key tool for every expert official to use to report what every exhibitor displays.
In the first M.E. the back up made this M.E. an easy +1.5 and should have been unanimous - great example of excellent back up. The reason all officials didn’t use a +1.5 isn’t clear, but this should have nothing to do with style or personal preference. This is an example of an officiating mistake in M.E. execution. You can use the same analysis on the last M.E. with a stop and hesitate, again an obvious +1.5 and should have been unanimously applied.
The spins and the rollbacks display a varied set of decisions from the officials. Kaci’s display of the maneuver quality and control should have made it easy for officials of this caliber to easily apply +1 M.E. The only wild card in this run was the RRB in Kaci’s final run, as the roll was poor, making this entire M.E. a +.5 coming down from very good or +1 (up to the point of roll). The LRB was very good and a +1 all the way through the rollback.
PURSE DISTRIBUTION ACCURACY OF THE TOP 29 IN THE OPEN FUTURITY
If we apply this same analysis across the entire futurity, the real difficulty was not in finding and scoring Kaci, as she brought it to them
and won, hands down -Congrats and great job- The real issue in accuracy is down card.
Reserve champion through place 29th requires similar analysis. An analysis on draw 7 back #278, King Arthur II ridden by Casey Deary and draw 11 #439, Hoss ridden by Andrea Fappani there are similar errors in accuracy as the use of the scale is being limited in application. We will also find officials have overmarked riders as the use of “benefit of the doubt” is applied too often.
The accuracy of M.E. obviously affects the final score and outcome of purse distribution. The importance for accuracy is also necessary for making the cut for the finals. Those competitors below a 439.5 combined score for L4 finals (which averages to 219.8) is the set of horses that will care the most about accuracy.
This concept runs true across the entire event:
L3 cut for finals was 434.5 average 217.3
L2 cut was 432 which is 216 average
L1 cut 423.5 with average 211.8
Since our system uses only .5 pt increments, the averages for cutlines would look like this:
L4- 220, L3- 217.5, L2- 216, L1- 212
Cowboy Office analyzes objective data across all open aged events in the USA that have a 5-official system in place throughout the year. In the L4, section 2 Open Reining Finals: 12% of runs incurred a penalty. Over 7% of the runs have non-unanimously applied penalties.
This equates to 58% of runs that incurred a penalty were non-unanimous penalty applications.
This makes a difference in the results of today’s competition. This futurity had a $1.44 million dollar L4 purse alone. A .5pt & 1pt penalty will make a difference on the outcome of the score in the system, especially when two or more officials apply it but not all five, which is very common. Not only does this affect placings, but also the payout to the riders, horses, owners and breeders.
The trend in the industry for the past several years at the open aged event level shows that 55-70% of all penalties applied are not applied unanimously. The question becomes: did it or didn’t it occur” as penalties should be absolute. A rule infraction either happens or does not happen! A horse either stepped out of lead, or did not step out of lead.
Good news: the national trend for the past few years is that the rate of runs that have penalties is estimated at 12%-13% of all runs in open aged events. Penalties applied in the 2024 L4 Reining finals is on par with the national trend: 7% of these runs had non-unanimous penalties applied.
THE OFFICIATING SYSTEM PERFORMANCE ACROSS THE ENTIRE EVENT.
(CowboyOffice.com 2024 score placing analysis source:nrha.com/results)
Around a table of reiners, it is often said that the “system works” and thus, all is good. It is true that the system works and is far superior to any individual expert’s decision. On a national trend, the system for the past few years has been on a 63 -65% Performance Rate.
The gauges above show the “system” performance across the 2024 open futurity. The overall performance rate across 4 sections of go round was 63%, same for the semi-finals section 1, then decreased to 56% in the semi-finals section 2.
The performance rate of the system increases both at the composite score which determines cuts for finals, as well as in section 2 of the L4 Finals.
IS IT GOOD ENOUGH
This leaves us with the most important question: is 63-65% performance good enough in today’s industry? Is this good enough as the tool being used to score the best of the best in a great sport?
The accuracy rate for 2024 is running at 57%. The national trend annually is headed downward as the horses and riders are headed upward. The 2023 accuracy rate is about 10% higher than the accuracy as 2024 concludes.
The performance of the system lies with the assumption that dropping the high and low is as good as we can perform. The industry assumes the three remaining scores are generally correct. This does not consider the error rate in accuracy that human experts make, or nonunanimous penalty application.
Remember Kaci’s winning run referenced above, as two maneuvers had obvious accuracy errors in it and more than two officials made similar errors. Take this across 635 runs scored in 8 sections across 7 days. Finding the acceptable individual official error rate and then inserting this into the system will lead us to the best performance rate achievable. Through research, the need to advance the system in specific detailed areas such as eliminating the rule of dropping the high & low score can aid in improving accuracy.
The system can perform in the 80th percentile, however it continually performs in the 60th percentile. This alone should demand attention and adjustment. The national industry trend line shows the system and officials’ performance is going downward or backward while horses, breeders, trainers and showmen continue to ride upward and forward.
Congratulations to Kaci O’Rourke
Horse: The Fireman
Owner: Peter and Courtney Morgan
Breeder: Tamarack Ranch.
Keep Your Questions Coming. Until the next time, enjoy the ride.
Email questions to exec@cowboyoffice.com.
Website: CowboyOffice.com
Facebook: @cowboyofficeshow
Instagram: @cowboyofficeshow
LinkedIn: cowboy-office>
TikTok:@cowboyofficeshow
Email: Exec@CowboyOffice.com
– Media Resource:
Kaci O’Rourke & Jen Kasper Media
nrha.com/results
2024 nrha rulebook
https://www.nrha.com
Featured Image: Kaci O’Rourke & The Fireman
Image copyright: Jen Kasper Media
2024 Open Reining Champion. Kaci O”Rourke and The Fireman make history as the 1st woman to win this prestigious event.
We review the event and provide data and analytics for the officiating performance relative to the professional talent displayed.
After the submission of an anonymous question about improving the reining judging system, past judges and NRHA monitors outline necessary steps and 3 practical ways advancement to officiating could happen.
A compelling year of reining concluded in Oklahoma City at the 2023 NRHA Futurity with a tie for Champion. In an epic scoring analysis, Brian and Jody break down whether there should have been a tie to begin with.
Reining judges analyze competition, covering NRHA Rules, judging systems, variable scores, and the acceptable range of score spreads and variances, emphasizing the importance of managing mistakes for a fair judging assessment.
The 2022 Open NRHA Reining Futurity episode unfolds as Andiamoe, ridden by Shawn Flarida, claims victory with a 0.5-point edge over Customize My Dream, analyzed through a comprehensive officiating breakdown by the Cowboy Office.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.
