Journal of Science and Medicine in Sport 23 (2020) 35–40

Contents lists available at ScienceDirect

Journal of Science and Medicine in Sport

journal homepage: www.elsevier.com/locate/jsams

Original research

Key performance indicators in Australian sub-elite rugby union

Tim J. Mosey ∗ , Lachlan J.G. Mitchell
Queensland Academy of Sport, Australia

a r t i c l e i n f o a b s t r a c t

Article history: Objectives: The primary aim of this study was to determine which key performance indicators (PIs) were
Received 12 April 2019 most important to success in sub-elite rugby union, and whether the analysis of absolute or relative data
Received in revised form 6 August 2019 sets as a method for determining match outcome was stronger than the other.
Accepted 16 August 2019
Methods: Data was taken from 17 PIs from 76 matches across the 2018 Queensland Premier Rugby
Available online 22 August 2019
Union season. A random forest classification model was created using these data sets based on win/loss
outcomes.
Keywords:
Results: The randomForest model classified 53 from 73 losses (72.6%) and 53 from 73 wins for an overall
Statistics
randomForest
percentage accuracy of 72.6%. The randomForest model based on the relative data set classified 57 from 73
Sports losses (78.1%) and 57 from 73 wins for an overall percentage accuracy of 78.1%. McNemar’s value of p = 0.84
Science confirmed that the relative data model did not outperform the absolute data set. There were positive
associations between match outcome and relative number of kicks in play, meters carried, turnovers
conceded and initial clean breaks.
Conclusions: Outcomes in Queensland Premier Rugby can be predicted using relative and absolute data
sets, though the difference between absolute and relative set usage was not as substantial as in profes-
sional rugby. Absolute and relative data sets can be used to create match strategies and assess match
performance. A game plan based around an out of hand kicking game and accumulating more metres
than the opposition, whilst minimising turnovers when in possession were key to success.
© 2019 Sports Medicine Australia. Published by Elsevier Ltd. All rights reserved.

Practical Implications 1. Introduction

• Develop match strategy and options which maximise kicking in
general play and accruing 200 m in total-carries more than the
opposition.
• Developing training drills and playing styles which encourage
players to maximise their ability to carry the ball through the
tackle and turn the ball over at the breakdown.
• Develop game sense in players which enables them to use multi-
ple strategies at the ruck to minimise turning the ball over when
attacking.
• Use partial dependence plots to set objective team performance
markers in important performance indicators.
• Develop strategies to limit the amount and effectiveness of the
opposition’s kicking game.
∗ Corresponding author.
E-mail address: tim.mosey@npsr.qld.gov.au (T.J. Mosey).
Performance Indicators (PIs) are often used to assess and
quantify success in a variety of sports,1 including rugby union
(henceforth rugby).2 Understanding what is critical to perfor-
mance is important for coaches and performance staff alike, if they
wish to improve skills and physical attributes.3 The most directly
attributable PIs should distinguish between winning and losing in
team sports such as rugby. In contemporary literature, researchers
have looked to analyse multiple PIs and their relationship to match
outcome4 and how PIs effect match performance5 . Furthermore,
recent studies on rugby at the elite level, have suggested there are
PIs which are more important than others in having an effect on
match outcome.2,6
It appears that the frequency of kicking in general play differen-
tiates between success and failure in domestic and international
rugby matches4 ; winning teams tend to kick away greater por-
tions of their possession across a match. Winners in rugby also
have lower error4,7 and turnover6,7 rates than their opposition.
Additionally, winners have a more effective defensive game, with a
better success rate at the ruck6 and complete more successful tack-
les overall.4 From an attacking perspective teams who carried the

https://doi.org/10.1016/j.jsams.2019.08.014
1440-2440/© 2019 Sports Medicine Australia. Published by Elsevier Ltd. All rights reserved.
36 T.J. Mosey, L.J.G. Mitchell / Journal of Science and Medicine in Sport 23 (2020) 35–40
ball further with each carry6 and effected a higher number of clean
line breaks4,6,8 were more likely to win. This general play effective-
ness was coupled with a greater ability to win set piece ball at the
scrum8 and lineouts won against the throw.7
It is challenging to suggest that winning and losing in com-
plex team sports can be attributed to analysing absolute PI data
in isolation.9 Previous literature on elite rugby populations has
shown which PIs contribute most substantially to match outcomes
at Super Rugby and International levels.4,6 Kicks in general play and
tackles made were the top two discriminators between winners
and losers in Super Rugby,4 whereas in international competition
a wider range of PIs discriminated between winning and losing.4,6
and no significant differences were shown between winning and
losing teams across all the PIs assessed.4 It was suggested that dif-
ferences between winners and losers at international level were
masked by variations in playing style across teams.4
There are two sides in a contest and research focusing on abso-
lute data across key PIs limits the potential to assess winning
performance against the performance of the losing team within a
match. Comparing sports data or PIs relative to the opposition could
be a more accurate descriptor of match outcome.10 Using relative
PIs, that is, how many more or less of a particular action a team per-
formed when compared to their opposition, may be a more effective
method of analysis.2 The use of relative numbers rather than abso-
lute numbers can differentiate a PI’s strength in determining match
outcome. Using data that is relative to the opposition gives differ-
ent results. Relative data analysis in a professional domestic rugby
competition was reported to be a more accurate method of assess-
ment than analysing absolute PIs.2 This type of analysis gives elite
level rugby coaches and support staff the opportunity to be clear
and evidence-based in their match evaluation strategies.
For coaches and practitioners working with sub-elite popula-
tions, it may not be appropriate to apply data from elite competition
to their circumstances. Indeed using the results of studies done
on elite populations, and applying them to sub-elite groups can
be problematic. With this in mind the primary aim of this study
was to replicate the methodology of a study which assessed the
impact of PIs in elite level rugby competition2 using sub-elite com-
petition data to ascertain whether the same PIs were of similar
importance in sub-elite competition. Further to this we wanted
to know whether or not using absolute or relative data sets may
be more accurate in predicting match outcome. Finally, we sug-
gest how this data may be utilised from a practical perspective for
sub-elite level coaches in rugby.
2. Methods
PIs for the 2018 Queensland Premier Rugby (QPR) regular and
post season competition were downloaded from the OPTA website
(optarugby.com). Intra and inter-rater reliability of PIs derived from
this method have been reported to be <5%11 . The 2018 season con-
sisted of 9 teams playing 18 rounds of 4 matches plus 4 post season
fixtures (76 matches). Due to the assessment of PIs on a binomial
outcome (win/loss), matches ending in draw (n = 6) were excluded
from the analysis. The study design used in a recent paper2 was
replicated and the full set of PIs for each match was analysed. The
PI’s selected were the same as a previous study2 for comparative
purposes between elite and sub-elite competitions. These PIs were:
turnovers conceded, turnovers won, missed tackles, average carry
by an individual player (metres), penalties conceded whilst defend-
ing, total metres carried in the match, initial clean line breaks, own
lineouts won, defenders beaten, kicks in general play, penalties con-
ceded when attacking, tackles made to tackles missed ratio, total
tackles in the match, breaks to defenders beaten ratio, offloads in
the tackle, own lineouts lost, and total carries in the match. Briefly,
turnovers conceded means the team in possession gives possession
to the other team at the ruck only. This didn’t include ball lost in
the tackle. Turnovers won means the opposite: the team defend-
ing dispossesses the attacking team at the ruck to win the ball. An
initial clean break refers to an attacking player going through the
defensive line without being touched by a defending player. Total
carries in the match refers to the total number of times the team
with the ball takes the ball ahead either being tackled by the oppo-
sition or breaking through the defensive line. Breaks to defenders
beaten ratio refers to the number of times the attacking team breaks
through the line compared to the totalled number of defenders they
evade a) at the defensive line and b) subsequently after they have
broken through the defending line. Data was readily available in
the public domain therefore ethics approval was not necessary to
obtain.
The PIs determined by numbers in the data collected for a single
team in a match were considered the absolute data. For example
if team A kicked 20 times during the match and team B kicked 15
times, the assigned absolute values would be 20 for team A and
15 for team B. Secondly, for each game, descriptive conversion was
undertaken by calculating the difference between teams’ absolute
data, with this data set being termed the relative data set. Using the
previous example, the relative data values for team A would be +5
kicks and team B would be -5 kicks.
Multi-collinearity between predictors in both relative and abso-
lute data sets was investigated using the rfUtilities package.12 No
multi-collinearity was found between predictors in either data set.
Methods of analysis and the justification of these methods have
been described previously,2 though briefly, PIs from both absolute
and relative data sets were used as predictors for match out-
comes (win/loss) in two separate analyses using a randomForest
classification model using the randomForest package13 in R.14 A
decision tree uses a binary separation in a variable to classify data
in to categories, or produce an estimate for a continuous outcome
variable.15 The randomForest approach utilised a large number of
these decision trees and aggregates their results to improve pre-
diction accuracy.16 The mean decrease of accuracy (MDA)16 and
mean of the distribution of minimal depth17 of each PI were cal-
culated to establish the importance of each predictor in predicting
game outcome. The use of relative PIs rather than absolute val-
ues gives an entirely different perspective. While the impact of
absolute PIs on the likelihood of winning a match is implicitly
compared to mean values, there may be cases where the oppo-
sition performed many more of a particular action than the team
whose outcome is being predicted. In this case, if we assume that
the likelihood of a team winning increases for each of a certain
PI, this must also be true for the other team. A simple example
may actually be points. If a team scores 30 points then statisti-
cally they are more likely to win than they are to lose. Obviously
though, if their relative score is -5 then they will lose the game
as the other team will have outscored the mean value by more.
Therefore, the relative frequency (how many more or less than
the opposition), could be a useful metric that tells a unique story.
This has been supported previously.2 The relationship between PI
MDA and the mean minimum depth in each model was quantified
using a Pearson’s correlation coefficient.18 When interpreting MDA
values a negative value denotes a less important variable.19 When
interpreting mean minimal depth a lower number denotes a more
important variable. The rfPermute package20 was used to calculate
a p value for each PI for both the absolute and relative datasets.
This p value describes whether each PI significantly improves the
model’s ability to predict whether a team won or lost by being
included in the analysis. Partial dependency plots21 were gener-
ated for example PIs which the research team thought were most
helpful in demonstrating the value of this type of analysis. These
plots show the probability of winning (100% likelihood = −1.0) or
 T.J. Mosey, L.J.G. Mitchell / Journal of Science and Medicine in Sport 23 (2020) 35–40 37

Table 1
Mean accuracy decrease, p values and the mean value of minimum depth distribution for the Random Forest model on the absolute set of data.

Variable Mean decrease in accuracy p value Mean minimum depth

Turnovers conceded 13.5 0.0099 2.49

Turnovers won 10.5 0.0198 2.86
Missed tackles 9.1 0.0297 3.33
Average carry (m) 7.4 0.0297 3.97
Pens conceded in defence 6.7 0.0297 3.93
Metres carried (total) 6.5 0.0396 3.12
Initial clean break 5.6 0.0297 4.21
Own LO won 5.3 0.0396 4.2
Defenders beaten 5.1 0.0693 3.56
Kicks in play 4.8 0.0495 3.51
Pens con in attack 4.2 0.0594 4.8
Tackle made/missed ratio 2.8 0.1782 4.05
Tackles 0.8 0.3663 4.03
Breaks to defenders beaten Ratio −0.4 0.4851 4.02
Offloads −0.9 0.5842 4.29
Own LO lost −0.9 0.5743 5.49
Total carries −1.0 0.5050 4.02

Table 2
Mean accuracy decrease, p values and the mean value of minimum depth distribution for the Random Forest model on the relative set of data.

Variable Mean decrease in accuracy p value Mean minimum depth

Kicks in play 25.1 0.0099 1.89

Metres carried (total) 23.1 0.0099 2.03
Turnovers conceded 18.5 0.0099 2.4
Initial clean break 10.6 0.0198 3.2
Defenders beaten 7.1 0.0495 4.29
Own LO won 6.6 0.0099 3.34
Average carry (m) 6.1 0.0495 3.74
Pens conceded in defence 6.1 0.0990 3.02
Offloads 6.1 0.0198 3.73
Missed tackles 4.7 0.0792 4.28
Tackles 3.4 0.1287 4.22
Total carries 2.4 0.1485 3.87
Own LO lost 1.8 0.1782 3.34
Tackle made/missed ratio 1.5 0.2475 3.98
Breaks to defenders beaten ratio 1.4 0.2772 4.27
Pens conceded in attack 0.6 0.3366 4.08

losing (100% likelihood = 1.0) across the range of values for each PI
presented.
McNemar’s Test was used to calculate the statistical significance
of each model.22
3. Results
The randomForest model based on the absolute data set from
the 2018 QPR season classified 53 from 73 losses (72.6%; 95% CI
65.4–79.8%) and 53 from 73 wins for an overall percentage accuracy
of 72.6% (CI 65.4–79.8%). The randomForest model based on the
relative data set classified 57 from 73 losses (78.1%; CI 65.9–79.3%)
and 57 from 73 wins for an overall percentage accuracy of 78.1%
(CI 65.9–79.3%). The McNemar’s value of p = 0.84 confirmed that
the relative data model did not outperform the absolute data set.
Data relating to each variable’s MDA is shown in Table 1 and 2 —
the absolute and relative data sets respectively. The absolute data
set model contained 9 predictors whose distribution varied signif-
icantly from the null, while the relative data set model included 8
predictors who were varied significantly from the null. The mag-
nitude of significant MDA values ranged from 13.5 to −1.0 in the
absolute data set and 25.1 to 0.6 in the relative data set. Mean val-
ues of mean minimum depth for predictors in the absolute data
set varied from 2.49 for the strongest predictor to 5.49 for the
weakest. In the relative data set the strongest predictor was 1.89
and weakest 4.29. A strong, negative correlation existed between
MDA values of predictor importance and mean minimum depth
distribution within the relative dataset, while a moderate correla-
tion existed between these variables in the absolute dataset. Partial
dependence plots for the top four predictors (based on MDA) were
produced for the absolute (Figs. 1A–D) and relative (Figs. 2A–D)
data sets. These demonstrate positive associations between match
outcome and absolute number of turnovers conceded, turnovers
won, missed tackles and average carry (metres). Positive associa-
tions between match outcome and relative number of kicks in play,
meters carried, turnovers conceded and initial clean breaks for the
relative data set are shown.
4. Discussion
The primary aim of this study was to ascertain which PIs were
the most important predictors of success in sub-elite rugby. Our
data indicate that relative PIs and absolute PIs predict performance
with a similar level of accuracy, in contrast to previous research.2
The top four predictors of match outcome from the relative data
set (Fig. 2) suggest increases in kicks in general play, total metres
carried per match, turnovers conceded and initial line breaks are
the strongest PIs related to match outcome. In contrast, the data
collected in an elite population2 found penalties conceded whilst
defending as the fourth most important PI. This PI ranked eighth in
our data using a sub-elite population. There could be a number of
reasons for this. Perhaps the penalty options taken by elite teams
differ to that in sub-elite competitions. Perhaps elite teams accrue
more points from penalty kicks and hence accumulate more points
throughout matches. To our knowledge no research has looked at
this point. Perhaps sub-elite teams may not convert penalty oppor-
38 T.J. Mosey, L.J.G. Mitchell / Journal of Science and Medicine in Sport 23 (2020) 35–40

Fig. 1. Partial dependency plots depicting the likely performance outcomes calculated for four key PIs using absolute values. Negative values indicate an increase in the
probability of a win, which positive values indicate an increase in the probability of a loss.

tunities into points as well as elite teams, though further analysis
into whether this is the case is warranted. Similarly, to the elite
population,2 penalties conceded when attacking was ranked last.
This may suggest a consistent refereeing style across the south-
ern and northern hemisphere in which attacking teams are given
appropriate advantage. Further research into refereeing practices
and rule interpretation is warranted. Our findings highlight that dif-
ferent strategies are employed by successful teams within different
competition levels in rugby.
We observed that kicking in general play from a relative per-
spective was the top PI. This was comparable with the elite data
set2 though the impact on the model was not as large (MDA 25.1 vs
51.6). Strategic kicking may be more important in elite competition.
Elite level players required to kick are likely more accurate than
their sub-elite counterparts and, consequently, may impact game
outcomes in a more meaningful manner. Type of kick (e.g. grubber,
high ball, chip kick, long kick) was not delineated in either this study
or previous2 and therefore the outcome of general play kicking was
not considered. The context to this is important. Effective kicking
can potentially result in the attacking team (the team that has just
kicked) gaining an advantage by either regaining possession before
the opposition picks up a loose ball, by the opposition making a
mistake upon trying to gain possession (by catching or picking up
the ball) or by a territorial gain, where the attacking team may kick
deep and a) tackle the opposition in their own territory or b) find
touch and have a lineout in the oppositions territory. Further anal-
ysis into determining how to quantify the outcome of general play
kicking is warranted.
How consistently predictors are chosen in the early splits of
each tree within the randomForest analysis is an indication of
how important these predictors are and will result in a low
mean minimum depth value.23 Pearson’s correlation coefficients
between minimum depths of each of the predictors did not confirm
greater levels of agreement for the relative model as is observed in
elite competition.2 These smaller differences would suggest that
using either absolute or relative data may accurately differentiate
between winners and losers in sub-elite rugby.
The partial dependence plots created using the current dataset
suggest there are upper limits for some performance indicators
above which no further advantage exists in relation to match
outcome. These upper (and lower) limits can offer a quantitative
threshold measure for coaches and support staff who wish to set
objective performance markers within game plans. For example,
aiming to carry 200 more total metres in a match than the oppo-
sition improves the likelihood of match success, though staying
more than 200 m ahead does not increase the likelihood of success
any further (Fig. 2b). Similarly, accruing 4 less turnovers across the
match than the opposition increases the potential for match suc-
cess, but the likelihood of match success increases no more when
staying 15 less turnovers ahead of the opposition (Fig. 2c). These
 T.J. Mosey, L.J.G. Mitchell / Journal of Science and Medicine in Sport 23 (2020) 35–40 39

Fig. 2. Partial dependency plots depicting the likely performance outcomes calculated for four key PIs using relative values. Negative values indicate an increase in the
probability of a win, which positive values indicate an increase in the probability of a loss.

thresholds could be seen used to prioritise a team’s strategic objec-
tives, for example, instead of attempting to accrue 15 less turnovers
than the opposition, it may be more beneficial for players to focus
their efforts on minimising the oppositions run metres or their
ability to execute kicks in play instead.
There may also be a discrepancy between sub-elite and elite
players in match awareness and strategy. Fitness, strength and
anthropometric differences also exist between elite and sub-elite
populations24,25 and between international and provincial level
players.25,26 Sub-elite players may lack the necessary fitness qual-
ities to maximise the benefits of an in-game kicking plan. General
play kicking requires the attacking team to chase the ball down-
field. As multiple long kick-chases may take longer to recover from.
Players at the sub-elite level may lack the necessary fitness to
repeatedly chase kicks effectively and maximise territorial gained.
Further research as to why the difference in strength within this PI
exists is warranted.
The MDA for total metres carried surpassed that of both average
carry and clean line breaks. Previous research,2 conversely, demon-
strated that the average carry PI was more important for winning
matches. Further research also indicated that clean line breaks dif-
ferentiated between match success in domestic27 and international
competition.8 Sub-elite defensive patterns and players’ fitness lev-
els may combine to explain this result as players exhibit lower rates
of ‘working off the ball’. Sub-elite players may exhibit a reduced
ability to cover-tackle, arising from a lower fitness level which
may inhibit them reaching the attacking player early enough, if at
all. Another explanation may be the contrasting playing styles and
structures between northern and southern hemisphere teams. The
wetter winter in Europe compared to the drier winter of Australia
may result in playing styles more suited to faster game speeds in
the southern hemisphere. However, this remains to be confirmed
in sub-elite populations and further investigation is warranted in
order to determine whether this is the case.
The use of both absolute and relative data sets can be impor-
tant and results should be interpreted within the appropriate
context to be effectively utilised in preparing players for com-
petition. For example, strategies to maximise a team’s ability to
accumulate 200 m or more than the opposition (relative analysis)
can be developed by coaching staff whilst setting a bench mark
of 550 m (absolute analysis) carried as a minimum, to exploit the
winning potential of their team. Absolute values set the minimum
requirement which teams need to be able to attain during a match
to maximise the chance of winning. Thorough pre-match plan-
ning may suggest the use of absolute values if an understanding
of the opposition’s match statistics or typical match strategy is
unavailable. The coaches’ use of relative data when creating match
strategies may be more appropriate when they have a thorough
understanding of opposition tactics and they understand the min-
imum absolute values needed to hit during a match to maximize
40 T.J. Mosey, L.J.G. Mitchell / Journal of Science and Medicine in Sport 23 (2020) 35–40
the chance of winning. Absolute datasets are important and can
provide coaches and support staff objective measures upon which
to build match strategies. Relative data sets can give further depth
and context, as well as flexibility, to these objective targets.
The ranking and relative effects of individual PIs on match
outcome can be useful tools for coaches when planning match
strategies, interpreting opposition statistics and identifying physi-
cal or skill based weaknesses within their own playing group which
may not allow them to execute these PIs in a match. The comparison
of PI rankings between the elite and sub-elite level may also help
with both preparing players for the physical demands of the elite
level, while also providing an insight into which players may excel
when moving up to the elite level and which may not. The chal-
lenge in comparing data across the elite and sub-elite populations
is that the elite data is taken from the English Rugby Championship
whereas the sub-elite data is from Queensland Club Rugby compe-
tition. The two styles and approaches to the game can be different
due to weather differences dictating game approach and style. The
more turbulent winter weather of England compared to the drier,
sunnier winter in Australia can be significant contributors to match
style and hence player type. Players looking to move from the sub-
elite level to the elite level should display the necessary physical
qualities or skill sets deemed important at the elite level. For exam-
ple the PI of average carry (m) was ranked in the top 3 at the
elite level2 compared to seventh in this study. As a priority then,
coaches at the sub-elite level may not value this quality as highly
as a player who can make a clean break. At the elite level, however,
clean breaks may not occur as easily due to higher quality defense
and territory on the field may become more of a strategic prior-
ity. Sub-elite coaches may have conflicting priorities depending on
whether they are developing players to win at the sub-elite level,
or whether they are looking to prepare players for elite competi-
tion. It is important to note that players who excel at the sub-elite
level, may not thrive at the elite level where different skills are
required to win matches. Talent identification targeted at select-
ing athletes for elite level competition should focus on selecting
players who demonstrate skill sets or physical qualities which help
them successfully fulfill the PIs deemed important at elite level. This
strategy may be more beneficial than focusing on players who may
be instrumental in assisting their team win at the sub-elite level in a
manner which would likely be less successful against elite opposi-
tion. A deeper understanding may come from comparing Australian
elite to Australian sub-elite PIs and is suggested for further study.
5. Conclusion
In sub-elite rugby the absolute and relative PIs which are most
important for match success have been determined from an abso-
lute and relative perspective. Match success at this level is related to
winning turnover ball and minimising ball turned over, minimis-
ing missed tackles and maximising metres gained per ball carry.
Compared to the opposition a well-developed in-match kicking
strategy, turnovers both for and against at the breakdown and
teams who can limit their missed tackles and accrue more metres
both per carry and as a team across a match have more chance of
winning. Comparing PIs against elite level match outcome proba-
bility can guide strategy and coaching focus.
Acknowledgements
No financial assistance and grant support has been provided for
this project.
References
1. Hughes MD, Bartlett RM. The use of performance indicators in performance
analysis. J Sports Sci 2002; 20(10):739–754.
2. Bennett M, Bezodis N, Shearer DA et al. Descriptive conversion of performance
indicators in rugby union. J Sci Med Sport 2019; 22(3):330–334.
3. McGarry T. Applied and theoretical perspectives of performance analysis
in sport: scientific issues and challenges. Int J Perform Anal Sport 2009;
9(1):128–140.
4. Vaz L, Van Rooyen M, Sampaio J. Rugby game-related statistics that discriminate
between winning and losing teams in IRB and super twelve close games. J Sports
Sci Med 2010; 9(1):51–55.
5. Prim S, van Rooyen M, Lambert M. A comparison of performance indicators
between the four South African teams and the winners of the 2005 super 12
rugby competition. What separates top from bottom? Int J Perform Anal Sport
2006; 6(2):126–133. http://dx.doi.org/10.1080/24748668.2006.11868378.
6. Watson N, Durbach I, Hendricks S et al. On the validity of team performance
indicators in rugby union. Int J Perform Anal Sport 2017; 17(4):609–621.
7. Jones NMP, Mellalieu SD, James N. Team performance indicators as a function
of winning and losing in rugby union. Int J Perform Anal Sport 2004; 4(1):61–71.
8. Ortega E, Villarejo D, Palao JM. Differences in game statistics between winning
and losing rugby teams in the six nations tournament. J Sports Sci Med 2009;
8(4):523–527.
9. Hughes MT, Hughes MD, Williams J et al. Performance indicators in rugby union.
J Hum Sport Exerc 2012; 7(2):383–401.
10. Ofoghi B, Zeleznikow J, MacMahon C et al. Data mining in elite sports: a review
and a framework. Meas Phys Educ Exerc Sci 2013; 17(3):171–186.
11. Bishop L, Barnes A. Performance indicators that discriminate winning and losing
in the knockout stages of the 2011 Rugby World Cup. Int J Perf Anal Sport 2013;
13(1):149–159.
12. Evans JS, Murphy MA, R package version 1.0-02014 ‘rfUtilities, 2014.
13. Liaw A, Wiener M. Classification and Regression by randomForest. R News 2002;
2(3):18–22.
14. R Core Team, URL R: A language and environment for statistical computing, Vienna,
Austria, R Foundation for Statistical Computing, 2017 https://www.R-project.
org/.
15. Jones Z, Linder F. Exploratory data analysis using random forests, Prepared for
the 73rd Annual MPSA Conference, 2015.
16. Breiman L. Random forests. Mach Learn 2001; 45(1):5–32.
17. Ishwaran H, Kogalur UB, Gorodeski EZ et al. High-dimensional variable selection
for survival data. J Am Stat Assoc 2010; 105(489):205–217.
18. Diedenhofen B, Musch J. Cocor: a comprehensive solution for the statistical
comparison of correlations. PLoS One 2015; 10(4):e0121945.
19. Ng VW, Breiman L. Bivariate variable selection for classification problem, technical
report, Department of Statistics, University of California-Berkeley, 2005.
20. Archer E. rfPermute: estimate permutation p-values for random forest impor-
tance metrics. R Package (Zenodo) 2016; 2(1). Version.
21. Friedman J, Hastie T, Tibshirani R. The elements of statistical learning, vol. 1.
Springer series in statistics New York, 2001.
22. Bostanci B, Bostanci E. An evaluation of classification algorithms using McNe-
mar’s test. Proceedings of Seventh International Conference on Bio-Inspired
Computing: Theories and Applications (BIC-TA 2012) 2013:15–26.
23. Huazhen W, Fan Y, Zhiyuan L. An experimental study of the intrinsic stability of
random forest variable importance measures. BMC Bioinf 2016; 17(60).
24. Baker D. Differences in strength and power among junior-high, senior-high,
college-aged, and elite professional rugby league players december. J Strength
Cond Res 2002; 16(4):581–585.
25. Fontana FY, Colosio A, De Roia GF et al. Anthropometrics of Italian senior male
rugby union players: from elite to second division. Int J Sports Physiol Perform
2015; 10(6):674–680.
26. Ross A, Gill ND, Cronin JB. Comparison of the anthropometric and physical
characteristics of international and provincial rugby sevens players. Int J Sports
Physiol Perform 2015; 10(6):780–785.
27. den Hollander S, Brown J, Lambert M et al. Skills associated with line breaks in
elite rugby union. J Sports Sci Med 2016; 15(3):501–508.