Professional Documents
Culture Documents
Kemethofer
Kemethofer
To cite this article: David Kemethofer, Christoph Helm & Julia Warwas (2022): Does
educational leadership enhance instructional quality and student achievement? The case
of Austrian primary school leaders, International Journal of Leadership in Education, DOI:
10.1080/13603124.2021.2021294
RESEARCH
ABSTRACT
School leadership ranks as an essential factor for successful schools.
The purpose of this study is therefore to investigate the effects of two
central leadership practices – setting directions and managing instruc
tional processes – on both instructional quality (mediator) and stu
dent achievement (target) in Austrian primary schools. The data for
this study originate from the 2018 national educational standards
test in mathematics, allowing the use of information from 3,785
teachers and 73,780 students from 2,961 schools. In line with theore
tical leadership effectiveness models, the results of multi-level struc
tural equation analysis reveal systematic relations of instructional
quality with student achievement. However, no direct or indirect
effects of the two leadership dimensions on student achievement
are apparent when simultaneous effects of student GPA and student
background variables, such as highest parental occupational status,
books at home, migration background, and language spoken at
home, are controlled for. We discuss these results as well as previous
findings with respect to methodological issues and contextual con
ditions that deserve more attention in future studies.
Introduction
During the last decades, many school systems around the globe have undergone far-
reaching transformations in their governance structure. In the quest for effective
ness, equity, and high quality in education, policymakers launched concepts of New
Public Management, emphasizing patterns of decentralization, market orientation,
comparisons, and accountability (Ball, 1998; Skedsmo & Huber, 2019). Schools
received more competencies for decisions at the operational level, but at the same
time, they were made more responsible for their success or failure in the perfor
mance of their students. Due to the shift toward more autonomy, the range of tasks
for school leaders has increased as well. Their sphere of responsibilities now
includes not only administrative tasks but also organizational and personnel
CONTACT Christoph Helm christoph.helm@jku.at Abteilung Für Bildungsforschung, Johannes Kepler Universitat
Institut Fur Padagogik Und Psychologie, Linz, Austria
© 2022 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License
(http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any med
ium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way.
2 D. KEMETHOFER ET AL.
composition that may affect both leadership behavior and achievement measures and
therefore present important control variables in leadership effectiveness studies. We
complement these considerations with fundamental methodological challenges of pro
viding statistical evidence of leadership effects. In Section 4, we conflate the presented
ideas in a conceptual model and basic assumptions that guide subsequent statistical
analyses of representative data from Austrian primary schools.
new role are coping with an established culture in the school and introducing their own
ideas about the school and teaching (Walker & Qian, 2006). In this scenario, Eder et al.
(2011) describe an existing tradition of teacher autonomy with respect to instruction. In
a sample of 537 teachers, 36% represented the autonomy–parity pattern and another 27%
described themselves as ‘lone fighters’, whereas only 36% belonged to the group of
teachers with a clear team orientation.
In addition to the above-mentioned tasks, principals in Austria also have a partial
obligation to teach in small schools with fewer than eight classes. Law regulates the exact
extent of teaching duties, which may be up to 18 hours per week. Besides the school size,
the existence of afternoon care groups, affiliated special schools and the number of
children with special educational needs have an impact on the total amount of teaching
hours (Heißenberger, 2019; Schratz, 2016). According to Heißenberger’s survey and
analysis of policy documents (Heißenberger, 2019), the main arguments in favor of
a teaching obligation are that regular teaching practice supports the evaluation of
teachers and that principals do not lose touch with the ‘core operations’ of an educational
institution. Some interviewed persons, on the other hand, see no relevance of mandatory
teaching for leadership tasks. In primary schools, which form the basis of our study,
about two out of three principals fall into this category of leading a school whilst having
teaching duties. In fact, the average number of classes in an Austrian primary school is
marginally above six (Statistik Austria, 2020).
To sum up, several context specificities affect the ways in which ideal conceptions of
effective educational leadership can be realized in the focal school system. Designing the
instructional program is often reduced to implementing and supervising binding
national curricula rather than a prolific area of activity for Austrian school leaders.
Own restraints, a late introduction of preparation programs, as well as long-standing
school cultures impede the necessary role change from an administrator and equal
colleague to an inspiring and motivating leader. Giving mandatory lessons does not
guarantee an individual ability to stimulate school-wide instructional improvements but
certainly restricts time for managerial tasks such as planning, coordination and
evaluation.
small total effects on achievement measures. Thus, a prevailing view among educational
researchers today is that principals unfold most of their effects on learning outcomes
indirectly, through a (context-specific) set of four core leadership practices: 1) building
a vision and setting directions, 2) taking care and developing the personnel, 3) (re)
designing the organization, and 4) managing instructional processes (Day et al., 2016;
Hallinger, 2011; Grissom et al., 2021; Leithwood et al., 2008; Leithwood, Harris et al.,
2020; Leithwood, Sun, et al., 2020).
Day et al. (2016) used a longitudinal mixed-methods approach to investigate the
relationship between leadership and student outcomes. They conclude that successful
leaders respond to the needs and possibilities of the school by focusing on improving the
quality of teaching and learning. This is achieved through their influence on structures and
culture. Similarly, Bruggencate et al. (2012) report school leadership effects on student
engagement and average exam scores that were mediated by development orientation and
classroom practices. Their findings indicate a strong relationship between leader behavior
and a school’s culture. Drawing on quantitative data from Germany, Pietsch and
Tulowitzki (2017) found effects of leadership on essential elements of a good school,
such as collaboration, commitment, and job satisfaction. Additionally, their analysis indi
cates effects of instructional leadership on teachers' classroom management and student
orientation.
To sum up: School leadership practices aim to enhance essential pedagogical activities
of the teaching staff. More precisely, they aim to boost the quality of instructional
processes, such as those postulated by Praetorius et al. (2018): classroom management,
cognitive activation, and needs-based support. These instructional processes are most
proximally related to the students’ learning success (e.g. Seidel & Shavelson, 2007) and
shaped not only through a principal’s direct involvement in the school’s instructional
processes but also through the conditions and visions he/she creates for these processes.
Instructional quality
Practical orientation
School leadership Support of cognitive activation – elaboration
behaviour Differentiation
Perceived clarity
Student achievement in
Setting directions Support of cognitive activation – problem
mathematics
Managing solving
instructional Support of meta-cognition
processes Support of motivation
Consideration of students‘ voice
Perceived discipline
Perceived pace of teaching
Student background
In this study, we use a conceptual framework that includes essential elements of estab
lished leadership models. Figure 1 depicts the structure of this model, while the following
sections of the paper elaborate the content and function of its constitutive elements.
practices, job satisfaction and commitment to the school, and learning opportunities for
the staff. But they also include the consistent implementation of specific, high-quality
teaching strategies.
The present study concentrates on instructional quality as one particularly important
path through which leadership is linked to outcomes (e.g. Bryk et al., 2010; Hallinger,
2011). This mediator plays a prominent role, since it contains variables that are demon
strably most proximally related to students’ academic success (Seidel & Shavelson, 2007).
As Creemers and Kyriakides (2012) point out, effective learning and strong learning
outcomes are the ultimate aims of any school. Accordingly, school leadership is expected,
first and foremost, to improve instructional practice. Generic conceptual frameworks of
instructional quality often demarcate three basic quality dimensions (Künsting et al.,
2016; Praetorius et al., 2018; Schlesinger & Jentsch, 2016) with several inherent facets that
also appear in Figure 1 above.
Firstly, classroom management comprises teacher actions that maximize students’ time
on task and thus ensure an orderly learning environment free of disruption. High levels
of discipline and attention arise, for instance, from teachers’ communication of clear
rules and their monitoring of student activities.
Secondly, the multifaceted construct of student learning support entails various
approaches to meet students’ basic psychological needs and thus approaches to foster
self-regulated learning. To support students’ experiences of competence, teachers pro
vide differentiated and adaptive instruction, align the pacing of instruction to the present
learner group, and give constructive feedback. To enhance students’ experience of
autonomy, teachers allow individual choices among (differentiated) tasks and create
learning material that is of practical relevance and interest to students’ lives. To foster
experiences of social relatedness, teachers demonstrate openness toward students’ opi
nions and contributions whilst encouraging the students to treat each other in a friendly,
considerate, and helpful way. All these means aim to establish a warm and trustful
learning climate.
Thirdly, cognitive activation results from cognitively challenging tasks, ques
tions, or even demanding problems. Furthermore, teachers who take up and put
students’ prior knowledge to the test, elicit and continually explore students’ lines
of thinking, or stimulate discursive and co-constructive learning activities also take
effective measures to foster students’ deep elaboration and understanding of the
instructional content.
Methodology
Some studies (e.g. Grissom et al., 2015; Maeyer et al., 2007) demonstrated that the effect
of school leadership on student achievement depends on the choice of the specified
model. We therefore concentrate on theoretically sound and (according to the above-
mentioned meta-analyses) empirically robust constituents of successful leadership and
mediating variables of instructional quality. We further comply with Scheerens’s call
(Scheerens, 2012, p. 137) to use multi-level structural equation modeling (see also,
Grissom et al., 2021, p. 86). This is done to account for the nested structure of the data
(students within teachers, teachers within schools) and the measurement error. It also
serves to model the theoretically assumed relations at the correct level of analysis, i.e.
specifying effects of instructional quality at the group level. Furthermore, we draw on
external assessments for all relevant constructs to prevent self-serving biases in reports of
INTERNATIONAL JOURNAL OF LEADERSHIP IN EDUCATION 9
School leadership
Eight items relate to leadership behavior. Four of them represent setting directions and
inform about a principal’s effort to define, discuss, and ensure the achievement of
educational goals (e.g. ‘The school leader defines goals that must be achieved by the
teachers of this school’.). Another four items assess managing instructional processes (e.g.
‘The school leader gives the teachers suggestions to improve the teaching’.) and focus on
classroom observations, providing feedback, and discussing teaching objectives. Teachers
rated all statements on a four-point Likert-type scale ranging from (1) never to (4) very
often.
Research offers heterogeneous approaches to model school leadership statistically. As
described above, scholars of the field claim several conceptually distinct dimensions of
school leadership. However, few studies have investigated whether this assumption holds
in the light of empirical data structures. For instance, Pietsch and Tulowitzki (2017)
found that a bifactor model (assuming a general factor and sub-dimensions) was the only
model that adequately represented school leadership practices. The bifactor model was
superior to a range of ‘correlated-factors’ models that are commonly applied in the light
of high empirical intercorrelations between the theoretically differentiated facets of
school leadership in previous studies (e.g. Bruggencate et al., 2012; Pietsch &
Tulowitzki, 2017). These high correlations cause problems of multicollinearity when
predicting outcomes of different leadership dimensions. Since bifactor models assume
a general factor (g-factor) that is uncorrelated with (orthogonal to) the specific factors
(s-factors) and that partials out the common variance of the s-factors (leading to low
10 D. KEMETHOFER ET AL.
Instructional quality
The student questionnaire included 39 items related to instruction. Based on research about
instructional quality and teacher effectiveness (e.g. Baumert et al., 2010; Kunter et al., 2013),
we selected 32 items referring to 10 distinguishable aspects of desirable instructional
practices which themselves are well compatible with the three overarching ‘basic dimensions’
INTERNATIONAL JOURNAL OF LEADERSHIP IN EDUCATION 11
of instructional quality we outlined in chapter 2 (see, Table 1): practical orientation (e.g. ‘The
teacher explains mathematics by examples from everyday life’.), cognitive activation –
elaboration focus (e.g. ‘The teacher gives tasks that make us think’.), differentiation (e.g.
‘We work in groups, and each group has different tasks’.), cognitive activation – problem
solving focus (e.g. ‘The teacher is interested in the fact that we can explain how we solve
tasks’.), perceived clarity (e.g. ‘In class, we are clear about what we are supposed to do’.),
support of meta-cognition (e.g. ‘The teacher tells us how we can improve’.), support of
motivation (e.g. ‘The teacher makes the lessons really exciting’.), consideration of students’
voice (e.g. ‘I get to choose whether I work alone, in pairs, or in a small group’.), perceived
discipline (e.g. ‘We can always work in a concentrated way’.), and perceived pace of teaching
(e.g. ‘The lessons are designed so that we can follow along well’.). Students rated all items on
a four-point Likert-type scale, with the response categories ranging from (1) in every lesson to
(4) never or almost never. For easier interpretation of the results, we inverted these items. To
account for the hierarchical structure of the data (students nested in classes), the measure
ment, and sample error, we specified one separate shared construct for each of the 10
instructional quality dimensions (Stapleton & Johnson, 2019). See, Table 1 for the psycho
metrics properties of the scales (model fit, descriptive statistics, and reliability indices).
Student achievement
The outcome variable is student achievement, which is represented by the performance
in mathematics. As part of the educational standards test in mathematics, every student
had to solve about 70 items divided into two test booklets of 40 minutes each. Each test
12 D. KEMETHOFER ET AL.
booklet roughly had the same difficulty including multiple-choice and open-ended items.
For comparison purposes, the test scores were transformed to a scale similar to PISA. On
average, students scored 551 points with a standard deviation of 99 points (BIFIE, 2019).
Further information on the data (e.g. composition of students) and the instrument used
to assess students’ mathematics achievement is provided in the national results report
(BIFIE, 2019) and by Breit and Schreiner (2016).
Control variables: composition of the student body of a class and student grades
To account for the social composition of the student body of a class, we included
measures that represent established indicators of students’ socio-economic status in
our statistical model. These indicators are the highest parental occupational status
(HISEI), books at home, migration background, and language spoken at home.
Moreover, we included students’ school grades to control for a class average academic
achievement level.
Analytic procedure
As argued above, given the nature of the available data (low school level variability of the
indicators) and our theoretical considerations (assessing instructional quality and its
effects as constructs that vary primarily between classes), it is more appropriate to
concentrate on the teacher and student levels in the present study. To investigate direct
and indirect effects of school leadership, we perform multilevel structural equation
models. More concretely, we perform a separate multilevel mediation analysis for each
of the 10 instructional dimensions that presumedly play a mediating role between school
leadership and student achievement.
All analyses were conducted using the R package ‘MplusAutomation’ (Hallquist &
Wiley, 2018) in combination with Mplus (Muthén & Muthén, 1998–2017). Since the
BIFIE provided full census data, no weighting was needed.
Use of plausible values: Students’ mathematics performance is generated by multiple
imputations based on the assumption that these values are conditionally distributed
depending on student test data (item responses) and other student data (covariates;
Foy et al., 2012; Robitzsch et al., 2016). The plausible values provided by the BIFIE are
then evaluated using the pooling method (Rubin, 1987). This was performed by the
Mplus command TYPE = IMPUTATION.
Missing data: All variables used except mathematics performance have missing values.
The item with the largest share of missing values (11%) is ‘We work in groups, and each
group has different tasks’. Subsequent analyses apply the Full Information Maximum
Likelihood (FIML) method. ‘In this method, all students are included for an analysis and
the missing data is “integrated out”’ (Robitzsch et al., 2016, pp. 290 f.). A central
assumption of the FIML procedure is that, considering all variables in the analysis,
missing values are randomly distributed (i.e. independent of the levels of other variables
in the analysis). Since our analyses include student achievement and student socio-
economic background, we believe that we do consider central variables that are predictive
for missing student data.
INTERNATIONAL JOURNAL OF LEADERSHIP IN EDUCATION 13
To determine the model fit, we used common cutoff criteria (Hu & Bentler, 1999;
Little, 2013) – Bentler’s comparative fit index (CFI ≥ .90), the Tucker–Lewis index
(TLI ≥ .90), the root mean square error of approximation (RMSEA ≤ .08), and the
standardized root mean square residual – at both the student and class levels (SRMR
L2 ≤ .10).
Results
All measurement models (Table 1) and all structural models (Table 2) have an
adequate model fit. Most of them even have a very good model fit, as indicated by
meeting or surpassing the criteria mentioned above (CFI, TLI > .95, RMSEA, SRMR
< .05). Only the model ‘perceived clarity’ conveys a TLI value below the cutoff.
However, all other fit indicators point again at a well-fitting model for this
mediator.
Direct effects
In Table 3, we report the direct effects of school leadership and the instructional
quality on students’ mathematics achievement in grade 4. The findings (see the left-
hand side of Table 3) suggest that school leadership only very weakly predicts
students’ math achievement beyond the effects of instructional quality. While the
global factor of leadership shows small but significant positive effects – slightly
varying between β = .049 and β = .066, depending on which instructional quality
dimension is included in the model – the specific school leadership factor ‘managing
instructional processes’ shows a small but significant negative effect, slightly varying
between β = −.073 and β = −.106, depending on which instructional quality
dimension is included in the model. However, these effects vanish if students’ socio-
economic background is controlled for (see the right-hand side of Table 3).
In contrast to the many non-significant direct effects of school leadership, all
dimensions of instructional quality are weakly to moderately related to students’
mathematics achievement (β = .128 to .430, p < .000). The strongest association is
between ‘perceived pace of teaching’ and math achievement (β = .430, p < .000), and
the smallest between ‘cognitive activation – elaboration focus’ and math achieve
ment (β = .128, p < .000). These findings are stable, even if students’ background
(e.g. highest parental occupational status, language spoken at home, books at home)
and grades are controlled for, though they slightly decrease in their magnitude.
In Table 4, we report the standardized effects of the three school leadership
factors on instructional quality in mathematics in grade 4. Neither the global
factor of school leadership nor its specific factors are significantly related to
instructional quality, with few exceptions: The specific factor ‘instructional leader
ship’ is significantly and negatively related to ‘support of motivation’ (β = −.104,
p = .006), ‘perceived discipline’ (β = −.086, p < .000), and ‘perceived pace of
teaching’ (β = −.069, p = .010). However, the effect size is negligible for all three
associations.
14
D. KEMETHOFER ET AL.
Table 4. Standardized effects (β) of the three school leadership factors on instructional quality in
mathematics.
Models without control variables Models controlling for the social composition of the student body
Table 5. Unstandardized indirect effects (b) of the three school leadership factors via instructional
quality on students’ mathematics achievement.
Models without control variables Models controlling for the social composition of the student body
Indirect effects
In Table 5, we report the unstandardized indirect effects of the school leadership factors
on students’ mathematics achievement before and after controlling for the social com
position of the student body of a class. As can be seen, there are only three statistically
significant and negative indirect effects of the specific factor ‘managing instructional
processes’ via ’support of motivation’ (b = −.020, p = .006; controlling for the social
composition of the student body: b = −.007, p = .045), ‘perceived discipline’ (b = −.020,
p < .001; b = −.008, p = .002), and ‘perceived pace of teaching’ (b = −.010, p = .011;
b = −.006, p = .032). Again, these effect sizes are weak and negligible. This finding
contradicts our assumption that school leadership (as perceived by teachers) is related to
student math achievement via instructional quality (as perceived by students).
INTERNATIONAL JOURNAL OF LEADERSHIP IN EDUCATION 17
Discussion
Gained insights
In the present study, we explored the effects of school leadership on instructional quality
and student (mathematical) achievement in Austrian primary schools. In doing so, we
drew on two central leadership strategies: setting directions and managing instructional
processes. Based on a framework of indirect effects, we hypothesized that school leader
ship affects student achievement via instructional quality. However, against the systemic
background that the Austrian school system represents a low-stakes context and that its
primary schools in particular have a long-standing tradition of principals acting as
teachers with additional administrative functions, we also assumed that leadership effects
might be small.
For our analysis, we used data obtained in national tests in 2018 allowing access to
information from over 3,500 teachers and 75,000 students. The sample covers a whole
cohort of Austrian primary students and their teachers. In accordance with methodolo
gical guidelines for leadership effectiveness studies, our analytical procedures entail
multilevel structural equation modeling and draw on external assessments for all relevant
constructs.
The results of our study support the assumed positive relations between the mediator
variables (dimensions of instructional quality) and the outcome variables (student
achievement). In particular, ‘perceived discipline’, ‘perceived pace of teaching’, and
‘cognitive activation – problem solving’ are most strongly related to students’ test scores
in mathematics. However, we find negligibly small and non-significant direct effects of
leadership on student achievement before and after controlling for the socio-economic
composition of the student body. No statistically significant indirect effects of leadership
(via instructional quality) appeared either.
assumptions about predictors and outcomes and if crucial control variables are specified.
In the present study, we claim both. Given that we could demonstrate theoretically
plausible relations between all dimensions of instructional quality and student achieve
ment but not between dimensions of leadership practices and either of these constructs, it
is questionable if a longitudinal approach would reveal these missing associations over
time. In other words: Are changes in the principals’ behavior expected to evoke changes
in teachers’ instructional work if differences in leadership practices are not associated
with differences in instructional quality? In addition, the full census data also have merits.
Advantages of the present large-scale cross-sectional study are the representativeness and
the size of the sample (high power). In contrast, longitudinal panel studies are often not
representative and small (too little power) due to costs of repeated data collection and
study dropout. In this sense, we argue that both study designs have their justification. Of
course, large panels would be best suited for our research question, but with the data at
hand, we argue to provide an initial informative picture of systematic relations among
educational leadership, instructional quality, and student achievement in the specific
setting of Austrian primary schools.
Further restrictions concern the operationalization of the school leader dimensions.
Although plausible in terms of content, the two scales used in our study do not represent
validated scales. Moreover, they reflect arguably important yet not the full range of
possible leadership behaviors that was mentioned in the first chapter.
Moreover, secondary data analyses set limits with regard to modeling theoretical
assumptions. Leithwood and Jantzi (2006, p. 205), for example, mention motivation,
capacity, and work settings as ‘key variables in a general model of employee perfor
mance’. This set of variables has an impact on classroom practices and, subsequently, on
student achievement. For school leaders, it is much more likely to influence these factors
than the actual teaching process. However, these intermediate processes could not be
modeled with the available data.
Finally, our output measure is limited to mathematics achievement. Although studies
on school leadership effects traditionally focus on student achievement, principals might
also have an impact on other valuable educational outcomes, such as student well-being
or school engagement.
Relevance of the study to the current situation: principals’ role during the COVID-
19 pandemic
Many studies, particularly in German-speaking countries, highlight the increased
importance of school leadership in meeting the challenges imposed on the school
system by the COVID-19 pandemic (Huber et al., 2020; Jesacher-Rößler & Klein,
2020; Lavonen & Salmela-Aro, 2022; Reyes-Guerra et al., 2021; S-CLEVER
Konsortium, 2021; Weiner et al., 2021). For instance, Weiner et al. (2021) argue that
the school leaders’ task to create conditions so organizational members can reflect their
current practice and to facilitate problem solving and innovation is particularly neces
sary during times of crisis. Moreover, the pandemic calls upon school leaders to best
utilize ‘the efforts and skills of their workforce to adapt to changing conditions and
perform under pressure’ (ibid., p. 2). To put it more concretely: Were school leaders
able to create learning environments in which students, teachers, and parents felt safe
to take risks, make mistakes, and learn? This addresses the role of school leadership in
ensuring instructional quality during COVID-19-related school closures. Our study
provides empirical evidence suggesting that a direct effect of school leaders’ behavior
on instructional quality is rather unlikely. This may also be the case for teaching during
COVID-19 as school leaders, teachers, and students were locally separated from each
other. Rather, we assume that a school leader’s behavior affects teachers’ working
environment during COVID-19, which in turn is a prerequisite for instructional
quality. Early studies show that during COVID-19, school leaders’ behavior was
20 D. KEMETHOFER ET AL.
associated with teachers’ well-being (Lavonen & Salmela-Aro, 2022), teacher coopera
tion (Huber et al., 2020; Jesacher-Rößler & Klein, 2020), teacher-student/parent coop
eration (S-CLEVER Konsortium, 2021), implementation of online teaching and
maintaining educational standards (Jesacher-Rößler & Klein, 2020), coordination of
teachers’ actions (Huber et al., 2020), etc. Moreover, school leaders were responsible for
ensuring a safe learning environment when schools were allowed to reopen. Hence,
given the multiple tasks of school leaders during the pandemic, we argue – in line with
Reyes-Guerra et al. (2021), pp. – that the crisis has corroborated that setting directions
and managing instructional processes represent only two of many leadership approaches
that are needed to tackle the challenges implied by the pandemic. Particularly under
drastic contextual changes, additional leadership knowledge, skills, and dispositions
come into play. Thus, future studies are necessary that integrate a broader set of
leadership practices when investigating school leadership effects on teachers’ work
(e.g. instructional quality) in particularly challenging conditions such as the recent
pandemic.
Notes
1. Students with special educational needs who were taught mathematics according to the
curriculum of special schools or a lower school level were excluded. The same counts for
students with physical or sensory disabilities if they were not able to take part in the test
under standardized conditions.
2. There was also a questionnaire for principals and parents, which is not included in the
present analysis.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Notes on contributors
Julia Warwas is a full professor in the field of Education for Business and Economics. Her research
interests cover Educational Leadership and Governance, Teaching Quality and Teacher
Cooperation, Stress and Coping in Working and Learning Contexts, and Competence
Assessment for Vocational Education and Training.
David Kemethofer is professor at the University College of Teacher Education, Upper Austria. His
research focus lies in the areas of school management and leadership, quality management in the
school system, school inspections, and educational standards and standard-based performance
tests.
Christoph Helm is a full professor in the field of Educational Research. His research is on School
and Teacher Effectiveness Research, Instructional Quality Research, COVID-19 and Education,
and Competence Assessment.
ORCID
David Kemethofer http://orcid.org/0000-0001-5153-5863
Christoph Helm http://orcid.org/0000-0001-5854-4500
INTERNATIONAL JOURNAL OF LEADERSHIP IN EDUCATION 21
References
Altrichter, H. (2017). The short flourishing of an inspection system. In J. Baxter (Ed.), School
inspectors: Policy implementers, policy shapers in national policy contexts (pp. 207–230).
Springer.
Ball, S. J. (1998). Big policies/small world: An introduction to international perspectives in
education policy. Comparative Education, 34(2), 119–130. https://doi.org/10.1080/
03050069828225
Baumert, J., Kunter, M., Blum, W., Brunner, M., Voss, T., Jordan, A., Klusmann, U., Krauss, S.,
Neubrand, M., & Tsai, Y.-M. (2010). Teachers’ mathematical knowledge, cognitive activation in
the classroom, and student progress. American Educational Research Journal, 47(1), 133–180.
https://doi.org/10.3102/0002831209345157
BIFIE. (2019). Bundesergebnisbericht: Standardüberprüfung 2018 Mathematik, 4. Schulstufe
[Federal Results Report: Standardized Performance Tests 2018 Mathematics, 4th Grade].
Salzburg.
Bossert, S. T., Dwyer, D. C., Rowan, B., & Lee, G. V. (1982). The instructional management role of
the principal. Educational Administration Quarterly, 18(3), 34–64. https://doi.org/10.1177/
0013161X82018003004
Brauckmann-Sajkiewicz, S., Kemethofer, D., & Warwas, J. (2020). Effekte und Bedingungen
wirksamen Schulleitungshandelns - theoretische Modellierungen, empirische Befunde, offene
Fragen [Effects and conditions of effective schol leadership – Theoretical modeling, empirical
findings, open questions]. In M. Warmt, M. Pietsch, S. Graw-
Krausholz, and S. Tosana (Eds.), Schulinspektion in Hamburg. Der zweite Zyklus 2012-2020.
Perspektiven aus Theorie, Empirie und Praxis (pp. 23–44). wvb .
Brauckmann, S., & Pashiardis, P. (2011). A validation study of the leadership styles of a holistic
leadership theoretical framework. International Journal of Educational Management, 25(1),
11–32. https://doi.org/10.1108/09513541111100099
Brauckmann, S., & Schwarz, A. (2015). No time to manage? The trade-off between relevant tasks
and actual priorities of school leaders in Germany. International Journal of Educational
Management, 29(6), 749–765. https://doi.org/10.1108/IJEM-10-2014-0138
Breit, S., & Schreiner, C. (Eds.). (2016). Large-scale assessment mit R: Methodische
Grundlagen der österreichischen Bildungsstandardüberprüfung [large-scale assessment with
R: Methodological pricipals of the Austrian Educational Standards Performance Tests].
Facultas.
Bruggencate, G., Luyten, H., Scheerens, J., & Sleegers, P. (2012). Modeling the influence of school
leaders on student achievement. Educational Administration Quarterly, 48(4), 699–732. https://
doi.org/10.1177/0013161X11436272
Bryk, A. S., Bender Sebring, P., Allensworth, E., Luppescu, S., & Easton, J. Q. (2010). Organizing
schools for improvement: Lessons from Chicago. University of Chicago Press.
Creemers, B. P. M., & Kyriakides, L. (2012). Improving quality in education: Dynamic approaches
to school improvement. Routledge.
Day, C., Gu, Q., & Sammons, P. (2016). The Impact of Leadership on Student Outcomes.
Educational Administration Quarterly, 52(2), 221–258. https://doi.org/10.1177/
0013161X15616863
De Maeyer, S., Rymenans, R., Van Petegem, P., van den Bergh, H., & Rijlaarsdam, G. (2007).
Educational leadership and pupil achievement: The choice of a valid conceptual model to test
effectsin school effectiveness research. School Effectiveness and School Improvement, 18(2),
125–145. https://doi.org/10.1080/09243450600853415
Eder, F., Dämon, K., & Hörl, G. (2011). Das „Autonomie-Paritäts-Muster“: Vorberuflich erlerntes
Stereoty, Bewältigungsstrategie oder Ergebnis der beruflichen sozialisation [Autonomy-parity-
pattern: A preprofessionally learnt stereotype, a coping strategy or a result of work-related
socialization]? Zeitschrift Für Bildungsforschung, 1(3), 199–217. https://doi.org/10.1007/s35834-
011-0021-1
22 D. KEMETHOFER ET AL.
Foy, P., Brossman, B., & Galia, J. (2012). TIMSS and PIRLS achievement scaling methodology. In
M. O. Martin & I. V. S. Mullis (Eds.), Methods and procedures in TIMSS and PIRLS 2011 (pp.
1–28). TIMSS & PIRLS International Study Center, Boston College.
Grissom, J. A., Egalite, A. J., & Lindsay, C. A. (2021). How principals affect students and schools.
A systematic synthesis of two decades of research. The Wallace Foundation. http://www.wallace
foundation.org/principalsynthesis
Grissom, J. A., Kalogrides, D., & Loeb, S. (2015). Using student test scores to measure principal
performance. Educational Evaluation and Policy Analysis, 37(1), 3–28. https://doi.org/10.3102/
0162373714523831
Gumus, S., Bellibas, M. S., Esen, M., & Gumus, E. (2018). A systematic review of studies on
leadership models in educational research from 1980 to 2014. Educational Management
Administration & Leadership, 46(1), 25–48. https://doi.org/10.1177/1741143216659296
Hallinger, P., & Heck, R. H. (1996). Reassessing the principal’s role in school effectiveness:
A review of empirical research, 1980-1995. Educational Administration Quarterly, 32(1),
5–44. https://doi.org/10.1177/0013161X96032001002
Hallinger, P. (2005). Instructional leadership and the school principal: A passing fancy that refuses
to fade away. Leadership and Policy in Schools, 4(3), 221–239. https://doi.org/10.1080/
15700760500244793
Hallinger, P. (2011). Leadership for learning: Lessons from 40 years of empirical research.
Journal of Educational Administration, 49(2), 125–142. https://doi.org/10.1108/
09578231111116699
Hallinger, P. (2018). Bringing context out of the shadows of leadership. Educational Management
Administration & Leadership, 46(1), 5–24. https://doi.org/10.1177/1741143216670652
Hallquist, M. N., & Wiley, J. F. (2018). MplusAutomation: An R Package for Facilitating Large-Scale
Latent Variable Analyses in Mplus. Structural Equation Modeling, 1-18 . doi:10.1080/
10705511.2017.1402334
Heißenberger, P. (2019). Berufsbild Schulleiter/in. Europäische Qualifizierungsimpulse [Job
description of a school principal. European qualification impulses]. https://docplayer.org/
184000324-Berufsbild-schulleiter-in.html
Hu, L. -., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis:
Conventional criteria versus new alternatives. Structural Equation Modeling:
A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
Huber, S. G., Günther, P. S., Schneider, N., Helm, C., Schwander, M., Schneider, J., & Pruitt, J.
(2020). COVID-19 und aktuelle Herausforderungen in Schule und Bildung [COVID-19 and
current challenges in school and education]. Waxmann.
Huber, S. G., Wolfgramm, C., & Kilic, S. (2013). Vorlieben und Belastungen im Schulleitungshandeln:
Ausgewählte Ergebnisse aus der Schulleitungsstudie 2011/2012 in Deutschland, Österreich,
Liechtenstein und der Schweiz [Preferences and pressures in school leadership: Selected results
from the 2011/2012 school leadership study in Germany, Austria, Liechtenstein, and Switzerland].
In S. G. Huber (Ed.), Jahrbuch Schulleitung 2013 (pp. 259–271). Carl Link.
Jesacher-Rößler, L., & Klein, E. D. (2020). COVID-19: Strategien der Schulentwicklung in der Krise:
Ergebnisse einer Schulleitungsbefragung in Österreich [COVID-19: School development strate
gies in crisis: results of a school leadership survey in Austria]. https://diglib.uibk.ac.at/ulbtir
oloa/content/pageview/5350042
Kemethofer, D., & Weber, C. (2020). Schulleitungshandeln im Kontext von Autonomie und
Accountability [School leadership in the context of autonomy and accountability]. Erziehung
Und Unterricht, 170(1–2), 98–107.
Klein, E. D. (2017). Schulleitungshandeln an staatlichen Schulen in Deutschland und den USA
[The role of principals in public schools in Germany and the USA]. Zeitschrift Für
Erziehungswissenschaft, 20(1), 61–87. https://doi.org/10.1007/s11618-016-0695-1
Künsting, J., Neuber, V., & Lipowsky, F. (2016). Teacher self-efficacy as a long-term predictor of
instructional quality in the classroom. European Journal of Psychology of Education, 31(3),
299–322. https://doi.org/10.1007/s10212-015-0272-7
INTERNATIONAL JOURNAL OF LEADERSHIP IN EDUCATION 23
Kunter, M., Baumert, J., Blum, W., Klusmann, U., Krauss, S., & Neubrand, M. (Eds.). (2013).
Mathematics teacher education: Vol. 8. Cognitive activation in the mathematics classroom and
professional competence of teachers: Results from the COACTIV project. Springer.
Kunter, M., & Baumert, J. (2007). Who is the expert? Construct and criteria validity of student and
teacher ratings of instruction. Learning Environments Research, 9(3), 231–251. https://doi.org/
10.1007/s10984-006-9015-7
Lavonen, J., & Salmela-Aro, K. (2022). Experiences of moving quickly to distance teaching and
learning at all levels of education in Finland. In F. M. Reimers (Ed.), Primary and secondary
education during Covid-19. Springer (pp. 105–123). https://doi.org/10.1007/978-3-030-81500-
4_4
Leithwood, K., Harris, A., & Hopkins, D. (2008). Seven strong claims about successful school
leadership. School Leadership & Management, 28(1), 27–42. https://doi.org/10.1080/
13632430701800060
Leithwood, K., Harris, A., & Hopkins, D. (2020). Seven strong claims about successful school
leadership revisited. School Leadership & Management, 40(1), 5–22. https://doi.org/10.1080/
13632434.2019.1596077
Leithwood, K., & Jantzi, D. (2006). Transformational school leadership for large-scale reform:
Effects on students, teachers, and their classroom practices. School Effectiveness and School
Improvement, 17(2), 201–227. https://doi.org/10.1080/09243450600565829
Leithwood, K., Louis, K. S., Anderson, S., & Wahlstrom, K. (2004). How leadership influences
student learning. The Wallace Foundation.
Leithwood, K., Sun, J., & Schumacker, R. (2020). How school leadership influences student
learning: A test of “the four paths model”. Educational Administration Quarterly, 56(4),
570–599. https://doi.org/10.1177/0013161X19878772
Leithwood, K., & Sun, J. (2012). The nature and effects of transformational school leadership.
Educational Administration Quarterly, 48(3), 387–423. https://doi.org/10.1177/
0013161X11436268
Liebowitz, D. D., & Porter, L. (2019). The effect of principal behaviors on student, teacher, and
school outcomes: A systematic review and meta-analysis of the empirical literature. Review of
Educational Research, 89(5), 785–827. https://doi.org/10.3102/0034654319866133
Little, T. D. (2013). Longitudinal structural equation modeling. Guilford Press. http://site.ebrary.
com/lib/subhamburg/Doc?id=10664545
Lüdtke, O., Marsh, H. W., Robitzsch, A., & Trautwein, U. (2011). A 2 × 2 taxonomy of multilevel
latent contextual models: Accuracy-bias trade-offs in full and partial error correction models.
Psychological Methods, 16(4), 444–467. https://doi.org/10.1037/a0024376
Luyten, H., & Bazo, M. (2019). Transformational leadership, professional learning communities,
teacher learning and learner centred teaching practices; Evidence on their interrelations in
Mozambican primary education. Studies in Educational Evaluation, 60, 14–31. https://doi.org/
10.1016/j.stueduc.2018.11.002
Marsh, H. W., Lüdtke, O., Robitzsch, A., Trautwein, U., Asparouhov, T., Muthén, B., &
Nagengast, B. (2009). Doubly-latent models of school contextual effects: Integrating multilevel
and structural equation approaches to control measurement and sampling error. Multivariate
Behavioral Research, 44(6), 764–802. https://doi.org/10.1080/00273170903333665
Marzano, R. J., Waters, T., & McNulty, B. A. (2005). School leadership that works: From research to
results. ASCD.
McDonald, R. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum.
Muthén, L. K., & Muthén, B. O. (1998-2017). Mplus user’s guide (Eighth ed.). Muthén & Muthén.
OECD. (2016). PISA 2015 results (volume 2): Policies and practices for successful schools.
OECD. (2019). PISA 2018 results (volume 2): Where all students can succeed.
Pietsch, M., & Tulowitzki, P. (2017). Disentangling school leadership and its ties to instructional
practices – An empirical comparison of various leadership styles. School Effectiveness and School
Improvement, 28(4), 629–649. https://doi.org/10.1080/09243453.2017.1363787
Pont, B., Nusche, D., & Moorman, H. (2008). Improving school leadership improving school
leadership. Volume 1: Policies and practices. OECD.
24 D. KEMETHOFER ET AL.
Praetorius, A.-K., Klieme, E., Herbert, B., & Pinger, P. (2018). Generic dimensions of teaching
quality: The German framework of three basic dimensions. ZDM, 50(3), 407–426. https://doi.
org/10.1007/s11858-018-0918-4
Reyes-Guerra, D., Maslin-Ostrowski, P., Barakat, M. Y., & Stefanovic, M. A. (2021).
Confronting a compound crisis: The school principal’s role during initial phase of the
COVID-19 pandemic. Frontiers in Education, 6, Article 617875. https://doi.org/10.3389/
feduc.2021.617875
Reynolds, D., Teddlie, C., Chapman, C., & Stringfield, S. (2016). Effective school processes. In
C. Chapman (Ed.), Routledge international handbooks of education. Routledge international
handbook of educational effectiveness (pp. 77–99). Routledge.
Robinson, V. M. J., Lloyd, C. A., & Rowe, K. J. (2008). The impact of leadership on student
outcomes: An analysis of the differential effects of leadership types. Educational Administration
Quarterly, 44(5), 635–674. https://doi.org/10.1177/0013161X08321509
Robinson, V. M. J. (2010). From instructional leadership to leadership capabilities: Empirical
findings and methodological challenges. Leadership and Policy in Schools, 9(1), 1–26. https://doi.
org/10.1080/15700760903026748
Robinson, V., & Gray, E. (2019). What difference does school leadership make to student
outcomes? Journal of the Royal Society of New Zealand, 49(2), 171–187. https://doi.org/10.
1080/03036758.2019.1582075
Robitzsch, A., Pham, G., & Yanagida, T. (2016). Fehlende daten und plausible values [missing data
and plausible values]. In S. Breit & C. Schreiner (Eds.), Large-Scale Assessment mit R:
Methodische Grundlagen der österreichischen Bildungsstandardüberprüfung (pp. 259–293).
Facultas.
Rößler, L., & Schratz, M. (2018). Leadership for learning. Teacher leaders as mediators for school-
wide innovation and change. In J. Madalińska-Michalak (Ed.), Teacher leadership (pp.
282–303). FRSE Publications.
Rubin, D. B. (1987). Multiple Imputation for nonresponse in surveys. Wiley series in probability and
mathematical statistics Applied probability and statistics. Wiley.
S-CLEVER Konsortium. (2021). S-CLEVER. Schulentwicklung vor neuen Herausforderungen:
Eine trinationale Längsschnittstudie in Deutschland, der Schweiz und in Österreich
[S-CLEVER. School development facing new challenges: a trinational longitudinal study in
Germany, Switzerland, and Austria]. https://s-clever.org/
Scheerens, J. (2012). School leadership effects revisited: Review and meta-analysis of empirical
studies. Springer.
Scheerens, J. (2017). The perspective of “limited malleability” in educational effectiveness:
Treatment effects in schooling. Educational Research and Evaluation, 23(5–6), 247–266.
https://doi.org/10.1080/13803611.2017.1455286
Schlesinger, L., & Jentsch, A. (2016). Theoretical and methodological challenges in measuring
instructional quality in mathematics education using classroom observations. ZDM, 48(1–2),
29–40. https://doi.org/10.1007/s11858-016-0765-0
Schratz, M. (2003). From administering to leading a school: Challenges in German-speaking
countries. Cambridge Journal of Education, 33(3), 395–416. https://doi.org/10.1080/
0305764032000122032
Schratz, M., Wiesner, C., Kemethofer, D., George, A. C., Rauscher, E., Krenn, S., &
Huber, S. G. (2016). Schulleitung im Wandel: Anforderungen an eine ergebnisorientierte
Führungskultur [School leadership in transition: requirements for a results-oriented
leadership culture]. In M. Bruneforth, F. Eder, K. Krainer, C. Schreiner, A. Seel, and
C. Spiel (Eds.), Nationaler Bildungsbericht Österreich 2015, Band 2: Fokussierte Analysen
bildungspolitischer Schwerpunktthemen (pp. 221–262). Leykam. https://doi.org/10.17888/
nbb2015-2-6
Schratz, M. (2016). Austria: Overcoming a bureaucratic heritage as a trigger for research on
leadership in Austria. In H. Ärlestig, C. Day, & O. Johansson (Eds.), A decade of research on
school principals: Cases from 24 countries (pp. 307–329). Springer.
INTERNATIONAL JOURNAL OF LEADERSHIP IN EDUCATION 25
Sebastian, J., Huang, H., & Allensworth, E. (2017). Examining integrated leadership systems in
high schools: Connecting principal and teacher leadership to organizational processes and
student outcomes. School Effectiveness and School Improvement, 28(3), 463–488. https://doi.
org/10.1080/09243453.2017.1319392
Seidel, T., & Shavelson, R. J. (2007). Teaching effectiveness research in the past decade: The role of
theory and research design in disentangling meta-analysis results. Review of Educational
Research, 77(4), 454–499. https://doi.org/10.3102/0034654307310317
Skedsmo, G., & Huber, S. G. (2019). Forms and practices of accountability in education.
Educational Assessment, Evaluation and Accountability, 31(3), 251–255. https://doi.org/10.
1007/s11092-019-09305-8
Specht, W., & Sobanski, F. (2012). OECD Review on Evaluation and Assessment Frameworks for
Improving School Outcomes: Country Background Report for Austria. Austrian Federal Ministry
for Education (BMUKK). https://www.oecd.org/education/school/49578470.pdf
Stapleton, L. M., & Johnson, T. L. (2019). Models to examine the validity of cluster-level factor
structure using individual-level data. Advances in Methods and Practices in Psychological
Science, 2(3), 312–329. https://doi.org/10.1177/2515245919855039
Statistik Austria. (2020). Schülerinnen und Schüler an öffentlichen und privaten Schulen 1923/24 bis
2018/19 [Students at public and private schools 1923/24 to 2018/19]. http://www.statistik.at/
web_de/statistiken/menschen_und_gesellschaft/bildung/schulen/schulbesuch/index.html
Stoll, L., Moorman, H., & Rahm, S. (2008). Building leadership capacity for system improvement
in Austria. In B. Pont, D. Nusche, & D. Hopkins (Eds.), Improving school leadership. Volume 2:
Case studies on system leadership (pp. 215–252). OECD.
Walker, A., & Qian, H. (2006). Beginning principals: Balancing at the top of the greasy pole.
Journal of Educational Administration, 44(4), 297–309. https://doi.org/10.1108/
09578230610674921
Warwas, J. (2012). Berufliches Selbstverständnis, Beanspruchung und Bewältigung in der
Schulleitung [Professional identity, work stress, and coping of school principals]. Springer VS.
Weiner, J., Francois, C., Stone-Johnson, C., & Childs, J. (2021). Keep safe, keep learning:
Principals’ role in creating psychological safety and organizational learning during the
COVID-19 pandemic. Frontiers in Education, 5, Article 618483. https://doi.org/10.3389/feduc.
2020.618483
Witziers, B., Bosker, R. J., & Krüger, M. L. (2003). Educational leadership and student achieve
ment: The elusive search for an association. Educational Administration Quarterly, 39(3),
398–425. https://doi.org/10.1177/0013161X03253411