The Australian Educational Researcher (2019) 46:213–229

https://doi.org/10.1007/s13384-019-00307-5

‘Aboriginal Voices’: An overview of the methodology

applied in the systematic review of recent research
across ten key areas of Australian Indigenous education

Kevin Lowe1 · Christine Tennent2 · John Guenther3 · Neil Harrison1 ·

Cathie Burgess6 · Nikki Moodie4 · Greg Vass5

Received: 15 August 2018 / Accepted: 11 February 2019 / Published online: 4 March 2019
© The Australian Association for Research in Education, Inc. 2019, corrected publication 2019

Abstract
The ‘Aboriginal Voices’ project is a consortium of researchers invested in seeking
long-term solutions to the many and varied issues that have been identified as factors
contributing to the underachievement of Indigenous students in education across
Australia. The Aboriginal Voices systematic review project established an agreed
methodology and protocols in order to map ten reviews to an overarching question:
‘What are the issues affecting the underachievement of Indigenous students in Aus-
tralia and how can research inform solutions to the complex and inter-related issues
needing to be addressed?’ The Project focuses on bringing clarity to the overarching
question by applying a rigorous process with an Indigenous methodology to under-
take these unique reviews. This introductory paper outlines the development of a
critical Indigenous methodology used in these discrete but inter-linked systematic
investigations. Using a consistent systematic methodology facilitated the re-aggre-
gation of separate review findings to achieve new whole-of-systems insights into
the complex issues affecting the educational opportunities of Aboriginal and Torres
Strait Islander students.

Keywords Systematic review · Methodology · Educational research · Aboriginal

Voices · Indigenous education

* Kevin Lowe
kevinlowe01@mac.com
1
Macquarie University, Macquarie Park, Australia
2
The University of Sydney, Camperdown, Australia
3
Batchelor Institute of Indigenous Tertiary Education, Batchelor, Australia
4
Melbourne University, Parkville, Australia
5
Griffith University, Brisbane, Australia
6
University of Sydney, Sydney, Australia

13
Vol.:(0123456789)
214 K. Lowe et al.

‘Aboriginal Voices’: an introduction

This paper describes the process of undertaking a simultaneous systematic review

of educational and social sciences research literature as part of the larger ‘Aborigi-
nal Voices’ project, a cross-institutional collaboration between 13 Indigenous and
non-Indigenous scholars1 with significant systemwide experience in educational
research. The purpose of this collaboration was to gain greater clarity on the effec-
tive engagement of Indigenous students in Australian schools. This paper elaborates
on the unique methodology applied by the authors to the ten reviews developed
within the broader project, six of which are contained within this Special Issue, to
answer the overarching question: ‘What are the issues affecting the underachieve-
ment of Indigenous students in Australia and how can research inform solutions to
the complex and inter-related issues needing to be addressed?’.
The methodology used to undertake these ten discrete systematic reviews was
established with the support of staff from Fisher Library at the University of Sydney
and the endorsement of the Metropolitan North Aboriginal Education Consultative
Group (AECG), which has received three monthly reports on the progress of the
project. Although the methodology draws heavily on the work of Petticrew and Rob-
erts (2006) and more broadly on the advice of the Cochrane Collaboration (2011)
Campbell Collaborations (n/d), and the Joanna Briggs Institute (2014), it also uses
a critical Indigenous methodology (Lowe 2017) that provided a specific view of the
development of the review questions, the development of the review processes and
the analysis and synthesis of the findings found within the studies included in these
reviews.
Each review aimed to identify research that provides an understanding of the
issues and solutions to the multiple and complex problems affecting Indigenous
school education in Australia. In wanting to understand the many concerns that
are seen to affect Indigenous students, the authors agreed to undertake simultane-
ous systematic reviews of recent2 Australian research across ten key areas known
to impact Indigenous students’ engagement and achievement at school. The ‘Abo-
riginal Voices’ project emanated from the need to know whether a review of current
school-based research would deepen our understanding and highlight possible solu-
tions to the issues affecting the schooling of Indigenous students (Steering Commit-
tee for the Review of Government Service Provision 2014, 2016). The issues sitting
within it are complex, and emerge from the ongoing experiences of the colonisa-
tion of Indigenous Australia, where the schooling (or lack) of Indigenous students
has been shown to have occurred in openly hostile and assimilatory environments
(Marsden 2018). Consequently, the authors chose to disaggregate this complex set-
ting, into ten discrete issues so as to allow greater attention to be paid to questions of

1
Australian Aboriginal and Torres Strait Islander peoples are two specific Indigenous groups located
within Australia. When the term ‘Indigenous’ is used in this article, it refers to traditional peoples from
these two groups.
2
2006–2017 with the exception of the review on racism which extended its date to range from 1989 to
2016.

13
‘Aboriginal Voices’: An overview of the methodology applied… 215

Fig. 1  Fields of research in the Aboriginal Voices systematic review project

how they independently and dependently affect Indigenous student engagement and
educational outcomes.
To make the inquiry relevant to the immediate needs of stakeholders (inclusive
of Indigenous students and their communities, teachers and schools), the research-
ers agreed that the project would investigate ‘current’ research into the issues shown
to impact on students’ success at school, to identify gaps in systemic and teacher
understanding, and to identify (where possible) solutions to, the complex issues
residing in current educational policy and practice. This paper discusses the chal-
lenges faced by the researchers in developing a robust process to guide these sys-
tematic reviews, including mastering the complexity and logistics of this multifac-
eted project and working with a cross-institutional team of scholars, many of whom
were new to this particular analytical approach of bringing an understanding to the
breadth of the literature.
The central problem that has plagued nearly all stakeholders within Indigenous
education, is that it has been difficult, if not impossible, to provide a research-
informed answer to the disarmingly simple question: “What are the issues affect-
ing the underachievement of Indigenous students in Australia, and how can research
inform solution/s to the array of long-term issues that need to be simultaneously
addressed?” Answers to this question are often drawn from a limited understand-
ing of the research and/or ‘knowledge’ found in the ‘common sense’ discourses of
deficit research (Vass 2013). Although this question is seemingly straightforward, in
reality it needed to be disaggregated into the salient fields of education to uncover

13
216 K. Lowe et al.

school practices, policies and interactions seen to impact on the educational success
of Indigenous students (see Fig. 1).
It was only at the completion of all ten individual reviews in the Aboriginal
Voices project, that the researchers were then able to view these findings to gain a
deeper understanding of the overarching inquiry question. In this regard, an analysis
of the research sought to situate findings that resonated with the voices of Indig-
enous people and that spoke of their hopes for an inclusive and responsive education
that would support their identities and underpin their educational aspirations.

Defining a methodology for the Aboriginal Voices project

The purpose of a systematic review is to critically analyse a wide range of research

studies according to predetermined criteria within the parameters of an agreed pro-
tocol and a methodology that systematises the overarching approach to be under-
taken. The first steps in this approach were to determine a question informed by
Indigenous perspectives that was to be addressed through this inquiry, and the sys-
temising of the literature search based on the key variables identified in the review to
ensure that all appropriate studies would be identified (Whittemore and Knafl 2005).
This required the development and adoption of an effective process that provided the
project with transparency and reliability for each of the reviews, while also facili-
tating the later re-aggregation of these findings to the central, overarching inquiry
question. Davies (2003) noted that the overall veracity of the systematic review was
dependent on the clarity of the question, the development of the protocols, and rig-
our of the methodology. Gough et al. (2017) summed this up when they wrote:
Reviewing research systematically involves four key activities: clarifying the
question being asked; identifying and describing the relevant research (‘map-
ping’ the research); critically appraising research reports in a systematic man-
ner, bringing together the findings into a coherent statement, known as synthe-
sis; and establishing what evidence claims can be made from the research. (p.
4)
A well-established systematic review needs to engage a rigorous and transparent
methodology that underpins a complete and exhaustive search of the research lit-
erature relating to the specific research inquiry being investigated. The efficacy of
a well described systematic review is determined by the development of a robust
research tool to structure the review, locate the diverse studies, and facilitate an anal-
ysis that enables the identification of those findings specific to each inquiry. At this
stage in the investigation, the researchers need to identify the information databases
most likely to have indexed relevant studies of Australian research on Indigenous
education. It was also deemed important to locate and assess government funded
research reports and ‘quality’ grey literature (Papaioannou et al. 2010). These proto-
cols were designed to scan widely across the field for peer-reviewed and quality grey
literature to ensure that all relevant studies and reports would be available to report
on, and as a way of eliminating bias in the review (Korhonen et al. 2013).

13
‘Aboriginal Voices’: An overview of the methodology applied… 217

The methodology of systematic reviews in academic research has been formal-

ised using structures and methods set out in the manuals, publications and research
protocols from various collaborative professional organisations such as the Cochrane
Collaboration (2011), Campbell Collaborations (n/d), and the Joanna Briggs Insti-
tute (2014). However, the model used to establish the systematic review protocols
for use in this project, drew on the methodological work of Petticrew and Roberts’
(2006) text on supporting practitioners undertaking systematic reviews within the
social sciences. This methodology gives significant emphasis to establishing the
review’s protocols and structures needed to develop the three interlocking phases of
investigation, evaluation and synthesis of the widely diverse body of evidence found
within social science research.
The Aboriginal Voices project concluded that undertaking these systematic
reviews was the most effective way of identifying, locating, evaluating and analys-
ing the research literature across the ten topics described in Fig. 1. However, it also
agreed that the findings from this project needed to be analysed such that we could
provide an analytical approach to these separate reviews and facilitate their incorpo-
ration into a cohesive model that would enhance our understanding of Indigenous
education in Australia (Lowe 2017). The researchers suggest that the power of this
research is not in the editorialisation of the findings, but instead lies in the frame-
work that brought the research findings to the common points for analysis. This pro-
vides an essential precursor to developing an informed, overarching and synthesised
appreciation of complex issues affecting the educational possibilities for Indigenous
students. It was on the basis of needing to bring this understanding to the research,
that the research team adopted an Indigenous frame to critically engage in the task
of constructing and making meaning for these reviews.

Key elements of a critical Indigenous methodology and a systematic

review

The following issues are central to the conceptualisation of the critical Indigenous
methodology that informed the development of the research question and the later
analysis and synthesis of the research data. Smith (2012) argued that the research
undertaken in, and/or about Indigenous people must be grounded in the notion of
being socially responsible, being centred on community healing, spirituality and
recovery, while also being responsible to local families and communities (Rigney
1997). It is argued that the researcher must aim to understand the experiences, aspi-
rations and needs of Indigenous people, while looking to provide Indigenous people
with both the tools and insights to facilitate their interrogation of the macro and
micro environments of colonisation within the discursive spaces that their communi-
ties have been forced to co- inhabit with the coloniser (Denzin 2005, 2010; Grande
2004; Meyer 2008).
This task of ‘meaning making’ in this context is underpinned by the application
of the key, critical principles of relationality with the voices within the research and
the audience of communities; the construct of knowledge and our understanding of
it; and our critical stance on supporting substantive change to Indigenous students’

13
218 K. Lowe et al.

experiences of schooling (Wilson 2001). These were the principles to which the
researchers in this project ascribed when constructing their inquiry questions, and
undertaking their analysis of the research data. Drawson et al. (2017, p. 13) under-
stood the utility of this outcome for non-indigenous researchers in looking to adopt
these principles, when noting that “by utilising an Indigenous method or framework,
(non-Indigenous) researchers (could affect) the incorporation of Indigenous knowl-
edge, history, and experiences into their research process”.
The adoption of this critical methodology meant that the research studies cited in
each of the reviews were interrogated to determine if their methodologies and analy-
ses were cognisant of the unique contextual identities of their Indigenous research
informants. Further, it meant that the research studies were examined to highlight
their understanding of the importance of local community histories, their acts of
agency and resistance to government (Foley 2003) and an awareness of the need to
transform both policies and practices used to subjugate Indigenous peoples (Smith
2000).

Methods

Defining the research protocol

From the beginning, the research team agreed to adopt a common systematic review
protocol to structure the project and ensure its internal consistency. The cross-insti-
tutional team worked closely with a project librarian from The University of Sydney,
and other collaborating university librarians, to develop database-specific search
strategies to inform the searches required for each review.
The protocols adopted for this project provided the overarching structure guiding
these investigations and as such formed a conceptual systematised ‘road map’ ensur-
ing that collaborating researchers could replicate their search results (Victor 2008).
The project protocols were applied to ensure that the ‘right’ question/s were posed
(Cochrane Collaboration 2011, pp. 85–93) using the PICo3 mnemonic to ensure
that the central elements of this qualitative research methodology were identified;
namely its research population, the phenomenon of interest being investigated and
the specific context to which the research was relevant (Stern et al. 2014); and that
the systematic review question/s were relevant and meaningful to Indigenous com-
munities. The inquiry questions used in each review were critiqued by the research
team to ensure ‘common sense’ deficit discourses of Indigenous students and com-
munity disability were problematised and challenged, and re-written so they cap-
tured stakeholder expectations (Head 2008). In this regard, the Aboriginal Voices
project drew on the work of Smith (2000), Schroeder (2014) and Lowe (2017) who
identified the agentic work required by Indigenous people to challenge the conceptu-
alisation of Indigenous peoples as ‘problems requiring a solution’. The overall effec-
tiveness of each systematic review stands on the implementation of this rigorously

3
The PICo acronym stands for. P—population. I—Interest. Co—Context.

13
‘Aboriginal Voices’: An overview of the methodology applied… 219

formed methodology that supported the intentions of each inquiry and also helped to
bring a sense of these investigations to the research audience.
In summary, the review protocol delineated the methods, with particular refer-
ence to ‘decisions about the overarching review question, the inclusion and exclu-
sion criteria, the search strategy, data extraction, quality assessment of the studies,
data synthesis and the dissemination strategy as laid out by Petticrew and Roberts
(2008). Additionally, specific elements of the Indigenous methodology were applied
to the key phases of the project, including the development of the review question,
the analysis of the findings and the overall synthesis of the research findings. These
elements were:

1. Each review would be inclusive of qualitative, quantitative and/or mixed methods

research.
2. The key research question/s illuminate what the research has found about the
experiences and aspirations of schools, teachers, Aboriginal students or their
families.
3. The common criteria for each review in the Aboriginal Voices project was as
follows:
a. That population reported on in the study findings clearly identified as Austral-
ian Indigenous students
b. That the research and its findings must be on issues that are germane to the
inquiry question
c. That the publication language was in English.
d. The time-period for the publication was 2006 to 2017. However, it was agreed
that this could be modified if the research team advised that their review
would be hampered if they could not incorporate earlier studies. This needed
to be argued on the prevalence of studies and/or critical contextual events that
occurred outside this timeframe but were shown to impact on the inquiry’s
veracity.
e. That the research context was Australian schools (from pre-school to senior
secondary) and/or their communities.
f. That the studies were peer-reviewed and research-based academic journal
articles, government reports, grey literature, research theses and research
focused books and book chapters.
g. The studies needed to meet an agreed quality criterion as described by Long
and Godfrey (2004), Ryan et al. (2007) and Coughlan et al. (2007).

Search criteria, methods used to search the literature and recording

results

Access to the most appropriate subscription databases for the review was essential,
despite their use being hampered due to them employing a non-standardised termi-
nology. The inconsistent indexing and/or poorly developed database thesauri meant
that the researchers were required to develop an extensive keyword/phrase search

13
220 K. Lowe et al.

strategy to retrieve results. This was an essential stage of the research process. The
particular expertise provided by library staff enhanced the interrogation of these sys-
tems, and informed search strategies, controlled vocabulary (thesaurus) navigation
and search syntax construction rules for each system.
A series of scoping searches of the target databases was undertaken at the com-
mencement of the searching stage of each systematic review. This was to ascertain
which databases indexed content that was relevant to the question, assess the func-
tionality of each database interface, and explore the thesaurus of indexed terms.
Based on the question and the inclusion and exclusion criteria, the concepts relating
to the population (who), the phenomenon (what) and the context (where) inherent
in each of the review questions, were then analysed for synonyms, and broader and
narrower terms, to ensure that the review captured the full range of possible terms
found in record titles, keywords, indexing and within abstracts. As each systematic
review search was conducted, these terms were mapped to the database thesaurus
terms (where available) and keywords/phrases were also searched, building up a
series of search result sets from each database system.
The search syntax rules of each system were followed, for example, truncation
symbols (e.g. cultur* for cultural, cultures and culture) were used to replace the
word ending in keyword searches in order to broaden search results by finding words
with variant endings. The search strategy involved combining the array of searches
using Boolean operators resulting in a large data set for each concept. The resulting
sets for the population, phenomenon and context were then cross matched using the
Boolean AND operator to focus the search to ensure that the studies met the search
requirements. Multidisciplinary databases such as Proquest Central, Web of Science
and Scopus were searched last using an extensive search string (a combination of
key words and phrases representing the concepts combined with Boolean operators)
which was replicated in the command search function of each system. All searches
were saved in each database, the records downloaded to a clearly labelled and dated
EndNote folder and search alerts (via email) for new material matching the search
criteria were set up in each database system for the period of the review. Addition-
ally, to ensure transparency, search terms and search strategies were carefully docu-
mented in a shared document drive along with a record of search results from each
database.
Databases were searched in order of their relevance to the Australian education
context, with those searched first being chosen as they primarily indexed Australian
research on topics specific to the Australian Indigenous education context. In most
cases, A + Education via Informit Online (which also includes the AEI-ATSIS subset)
database was chosen as the initial database to search. This system had an inbuilt the-
saurus of index terms which usefully suggested a variety of synonyms for the primary
concepts being investigated. This database thesaurus provided a model that was then
replicated with modifications for each of the other databases. Each additional database
required search modifications incorporating new thesaurus terms or keywords that were
both topic specific, and sensitive to the spelling variants according to the locations

13
‘Aboriginal Voices’: An overview of the methodology applied… 221

Fig. 2  Aboriginal Voices’ systematic review process flowchart

from which the databases were generated (Petticrew and Roberts 2006). The following
databases were interrogated using an iterative process starting from specific regional
education database systems, to broader and large multidisciplinary databases including:
A + Education; Australian Education Index (via Informit); AEI -ATSIS Australian Edu-
cation Index Aboriginal and Torres Strait Islander Subset; Family: Australian Family &
Society Abstracts Database (via Informit); ERIC: Educational Resources Information
Center (via OVIDSP); Indigenous Collection (via Informit); PsycINFO (via OVIDSP);
Proquest Central; Web of Science; Scopus; Dissertations & Theses Global (via Pro-
quest); and the Libraries Australia union catalogue, which was searched for Govern-
ment reports and theses.

13
222 K. Lowe et al.

The process flowchart

The following process flowchart (see Fig. 2) guided the authors in developing and
implementing their individual systematic review using the agreed protocols. The
PRISMA4 2009 flow diagram (see Fig. 3) (Moher et al. 2009; PRISMA, 2009) out-
lined a structured guide that the researchers agreed to follow to manage identifica-
tion, screening, eligibility and inclusion of studies that formed the body of research
studies included in each investigation. The PRISMA flowchart informed the devel-
opment of the Aboriginal Voices process flowchart, stepping the researchers through
the phases of their investigation. The process flowchart laid out the key stages of the
review, from establishing the inquiry question, the steps to include/exclude studies,
the analysis and synthesis of research findings that were consistent with both the
systematic review (Petticrew and Roberts 2006; Victor 2008) and critical Indigenous
methodologies (Smith 2000, Byrd 2011).
The results of each stage of the identification (Step 4) and screening of the stud-
ies (Steps 5–10) were managed (in most cases) using an Endnote library that was
set up for each database search. These database sets were initially sorted to identify
duplicate records and those that met the study design data range criteria (Step 6).
The criteria for inclusion/exclusion was applied to this final set and then both kept
and discarded citations were double checked against the question and protocol by at
least one other member of the team, in an effort to eliminate researcher bias in delet-
ing or including studies that did not meet the protocol criteria (Step 9). These stud-
ies formed the body of research to be interrogated in line with their unique inquiry
question/s. The actual application of the phases in Fig. 2 can be seen in Fig. 3, dem-
onstrating how these steps were applied and how their application progressively
excluded studies.

Screening for inclusion based on validity criteria for research quality

A key element of the systematic review process is assessment of the quality of evi-
dence. This needed to occur prior to the final selection of studies. Figure 4 draws on
the appraisal of qualitative studies (Dixon-Woods et al. (2007); Ryan et al. (2007);
Long and Godfrey (2004)), and quantitative studies (Coughlan et al. 2007) to con-
struct a series of questions that enabled a qualitative assessment of the research stud-
ies and their appropriateness. Figure 4 sets out the key assessment criteria of this
quality appraisal across studies using qualitative, quantitative and mixed methods,
with studies being scored against each criterion, as either (1) if met, (0) if not met
and (0.5) if partially met/described. Each study was read specifically to ascertain
how they addressed the six quality criteria in Fig. 4. Studies were graded on whether
these criteria were found and the degree to which they were described or met, with
researchers using their professional judgement and the assessment criteria. The

4
PRISMA—‘Preferred Reporting Items for Systematic Reviews and Meta-Analyses’.

13
‘Aboriginal Voices’: An overview of the methodology applied… 223

Fig. 3  Example of the application of the systematic review flowchart (Guenther 2018)

13
224 K. Lowe et al.

Fig. 4  Quality of evidence framework—adapted from (Coughlan et al. 2007; Dixon-Woods et al. 2007;
Ryan et al. 2007; Long and Godfrey 2004)

scores for each of the studies were then aggregated to a total of 6, with those not
scoring at least 3, being removed from the analysis as having not met the quality
research criteria.

Synthesis processes

The synthesis of the findings from these studies occurred across two levels. The
first looked at how the findings were reported, while the second sought to ensure
that the thematic analysis and synthesis was consistent with the primary purpose
and focus of these reviews. Several approaches have been suggested as having
equal validity in reporting on the research findings, with Sandelowski et al. (2006,
p. 32) suggesting either a ‘segregated’ method, where the findings from the quan-
titative, qualitative and/or mixed methods approaches are reported separately, or
an ‘integrated’ approach where the findings were synthesised using a holistic the-
matic analysis of the studies. This latter model was recommended in the Joanna
Briggs Reviewers’ Manual (2014, p. 9) as it facilitated the assimilation of data
into a single point of synthesis of all the studies, with quantitative and mixed
methods data being converted into a common set of themes, codified and then
brought together within one cross-methods approach. Both methods were used in

13
‘Aboriginal Voices’: An overview of the methodology applied… 225

the reviews, but in both cases, the researchers identified a range of themes to pro-
vide a unique understanding of the key issues within the inquiry (Bazeley 2009).
The second element of the synthesis applied to the critical Indigenous meth-
odology used to critique the findings from within the ‘known’ perspectives of
Indigenous people with a consequence of enabling the community to interrogate
these discursive findings and add to them their own voices, developing a deeper
and richer understanding of the key findings of these ten reviews, This critique
empowers Indigenous voices through underpinning their ability to establish a
unique, transformative, counter dialogue between families, communities and
children, and schools. It is this critically positioned understanding that underpins
Indigenous peoples’ ability to challenge their social, cultural and epistemic mar-
ginalisation and those institutionally sponsored discourses of deficit that have set-
tled into the ‘business’ of doing schooling for Indigenous students. In this regard,
these reviews were unique in that they set out to establish a critically informed
road map to re-evaluate the huge body of existing research to achieve an Indig-
enous understanding of how the practices of schooling impact on the education of
Indigenous children.

Methodological challenges: hearing ‘Aboriginal Voices’ in research

From the outset, one of the enduring challenges for the project revolved around
developing a common understanding of the overarching inquiry question and how
this was then to be re-articulated to inform the development of the ten systematic
review questions. Each of these review questions needed to be sensitive to the lived
experiences of Indigenous students and their families and the professional and social
experiences of teachers.
A second challenge was to establish consistent procedures across the ten reviews,
with the 13 scholars agreeing that the overarching question would guide the estab-
lishment of the project protocols. Booth (2016) noted that where questions are set
too broadly, the task becomes too unwieldy with the outcome of the review being
lost. Conversely, he argued, that a too narrowly defined inquiry had the effect of
limiting the usefulness of the review by limiting the body of literature that it could
draw on. Petticrew and Roberts (2006) identified that getting this phase right is criti-
cal to a project’s overall success, not only in terms of determining scope and focus,
but more importantly, whether the ‘right’ articulation of the question is tagged to the
‘real’ issue/s being investigated.
In part, this challenge was met through the use of critical Indigenous method-
ology with its critique of the unique problematisation of the discrete issues under
review. This methodology was crucial as it provided a focus from the outset, the
development of the ten discrete inquiry questions and the subsequent phases of
analysis and synthesis of the findings across the reviews. Further, the Indigenous
research principles enabled the studies to be read from the perspective of those
Indigenous families who have been pushed to the educational margins, by providing
an informed insight into the schooling experiences of their children.

13
226 K. Lowe et al.

While these reviews have proven to be an extremely useful tool to rigorously

investigate the research, it should be acknowledged that there were several methodo-
logically imposed limitations within this research. The first of these centred on the
application of the ‘rigorous’ protocols. The protocols, which were written deliber-
ately to ensure that the reviews reported on the findings of evaluative research that
met the rules written into the review protocols, had the effect of excluding some
studies as they failed to meet the ‘evaluative’ criteria (see protocols), or were seen
as biographical and/or personal or experiential narratives. A second concern raised
in the writing of the reviews, was that many of the research studies were seen as
being inadequately described in their titles, abstracts and/or keywords. The concern
here, was that the inbuilt search tools used to locate these studies relied on these
very elements to ‘discover’ them from within the database files. The failure of some
authors to use commonly accepted keywords meant they became almost undiscover-
able except when other search strategies such as hand searching and recommenda-
tions from experts were utilised. A third concern came to light when the ‘evidence
of quality framework’ (Fig. 4) was applied to determine each study’s research qual-
ity. In a number of cases, studies were excluded as they failed to identify important
methodological information including how participants were selected, how many
were involved in the research or from what theoretical position the data were ana-
lysed. This failure had the effect of studies being excluded.
While it is acknowledged that there are potential limitations to these system-
atic reviews, it is argued that their usefulness as a way of synthesising a large body
of evidence is largely dependent on the ‘conservative’ and rigorous methodology
applied to the key discovery and analysis phases of the project. The findings from
these reviews are seen to represent the overarching conclusions from the research
that met both the exacting review protocols and the key principles of the critical
Indigenous methodology.

Conclusions

Systematic reviews are a specific, carefully defined approach that can be applied to
the task of undertaking an intensive review of the research literature in a particular
area of study. Given the increased interest from governments, and the influence sys-
tematic reviews have on policy makers (Petticrew and Roberts 2008), it is surprising
that a greater effort from those seeking to affect change, is not made to undertake
this form of review as a way of testing the claims of programs said to improve the
educational outcomes of Indigenous students. The Aboriginal Voices: Systematic
Review project sought to simultaneously research ten key issues in the education of
Indigenous students and for the first time provide deeper insights into the complex
issues seen to affect the educational opportunities of students.
However, while being cognisant of the limitations of this particular literature
review methodology, the undertaking of these ten systematic reviews proved to
have significant utility for the team of researchers, as it has demonstrated that these
reviews provide a powerful tool to bring individual and collective insight and Indige-
nous meaning to one of the great conundrums of Australian education—its failure to

13
‘Aboriginal Voices’: An overview of the methodology applied… 227

address the needs of Indigenous students. The rigorous application of the systematic
review methodology and the application of its protocols, along with the overview
provided by the Indigenous methodology, enabled the identification of over 10,000
Australian studies that made claims to have researched in the space of Indigenous
education, and their findings analysed to assist our understanding of each inquiry
question. The findings of these reviews will now provide schools, policy writers
and Indigenous communities with insights into Indigenous people’s experiences of
schooling across Australia. At its completion, it provided a basis from which the
investigators were able to look across the body of research and begin to answer the
overarching inquiry question: ‘What are the issues affecting the underachievement
of Indigenous students in Australia and how can research inform solutions to the
array of long-term issues that need to be addressed?’

References
Bazeley, P. (2009). Analysing qualitative data: More than ‘identifying themes’. Malaysian Journal of
Qualitative Research, 2(2), 6–22.
Booth, A. (2016). Searching for qualitative research for inclusion in systematic reviews: A structured
methodological review. Systematic Reviews, 5(1), 74.
Byrd, J. (2011). Transit of empire: Indigenous critiques of colonisation. Minneapolis, MN: University of
Minnesota Press.
Campbell Collaboration (n/a). So, you want to write a Campbell systematic review? Retrieved from https​
://goo.gl/fAFwu​d
Cochrane Collaboration. (2011). Cochrane handbook for systematic reviews of interventions. Cochrane.
Retrieved from http://handb​ook-5-1.cochr​ane.org/
Coughlan, M., et al. (2007). Step-by-step guide to critiquing research. Part 1: Quantitative research. Brit-
ish Journal of Nursing, 16(11), 658–663.
Davies, P. (2003). Systematic reviews: How are they different from what we already do? In L. Anderson
& N. Bennett (Eds.), Developing educational leadership (pp. 25–38). London: SAGE Publications.
Denzin, N. (2005). Emancipatory discourses and the ethics and politics of interpretation. In N. K. Denzin
& Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (pp. 933–958). Thousand Oaks,
CA: Sage.
Denzin, N. (2010). Grounded and Indigenous theories and the politics of pragmatism. Sociological
Inquiry, 80(2), 296–312.
Dixon-Woods, M., Sutton, A., Shaw, R., Miller, T., Smith, J., Young, B., et al. (2007). Appraising qualita-
tive research for inclusion in systematic reviews: A quantitative and qualitative comparison of three
methods. Journal of Health Services Research and Policy, 12(1), 42–47.
Drawson, A., Toombs, E., & Mushquash, C. (2017). Indigenous research methods: A systematic review.
The International Journal of Indigenous Policy, 8(2), 5.
Foley, D. (2003). Indigenous epistemology and Indigenous standpoint theory. Social Alternatives, 22(1),
44–52.
Gough, D., Oliver, S., & Thomas, J. (Eds.). (2017). An introduction to systematic reviews (2nd ed.). Lon-
don: Sage.
Grande, S. (2004). Red pedagogy: Native American social and political thought. Lanham, MD: Rowman
& Littlefield.
Guenther, J. (2018). Factors contributing to educational outcomes for First Nations students in remote
communities: A systematic review. Manuscript in preparation
Head, B. (2008). Wicked problems in public policy. Public Policy, 3(2), 101.
Joanna Briggs Institute. (2014). The Joanna Briggs Institute reviewers manual 2014: Methodology for
JBI mixed methods systematic reviews. Adelaide, Joanna Briggs Institute University of Adelaide.
Retrieved from https​://goo.gl/nLxXq​y

13
228 K. Lowe et al.

Korhonen, A., Hakulinen-Viitanen, T., Jylhä, V., & Holopainen, A. (2013). Meta-synthesis and evidence-
based health care—A method for systematic review. Scandinavian Journal of Caring Sciences,
27(4), 1027–1034.
Long, A. F., & Godfrey, M. (2004). An evaluation tool to assess the quality of qualitative research stud-
ies. International Journal of Social Research Methodology, 7(2), 181–196.
Lowe, K. (2017). Walanbaa warramildanha: The impact of authentic Aboriginal community and school
engagement on teachers’ professional knowledge. The Australian Educational Researcher, 44(1),
35–54.
Marsden, B. (2018). The system of compulsory education is failing: Assimilation, mobility and Aborigi-
nal students in Victorian State schools, 1961–1968. History of Education Review, 47(2), 143.
Meyer, M. (2008). Indigenous and authentic—Hawaiian epistemology and triangulation of meaning. In
N. Denzin, Y. Lincoln, & L. T. Smith (Eds.), Handbook of critical and indigenous methodologies
(pp. 217–232). Thousand Oaks, CA: Sage Publications.
Moher, D., Liberati, A., Tetzlaff, J., Altman, D., & The PRISMA Group. (2009). Preferred reporting items
for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med, 6(7), e1000097.
Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010). Literature searching for social
science systematic reviews: Consideration of a range of search techniques. Health Information and
Libraries Journal, 27(2), 114–122.
Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Pad-
stow, UK: Blackwell Publishing.
Petticrew, M., & Roberts, H. (2008). Systematic reviews in the social sciences: A practical guide. Wiley.
PRISMA. (2009). PRISMA 2009 checklist. Retrieved from https​://goo.gl/NYTwQ​T
Rigney, L. (1997). Internationalisation of an Indigenous anti-colonial cultural critique of research meth-
odologies: A guide to Indigenist research methodology and its principles. Journal for Native Ameri-
can Studies, WICAZO sa Review, 14(2), 109–121.
Ryan, F., et al. (2007). Step-by-step guide to critiquing research. Part 2: Qualitative research. British
Journal of Nursing, 16(12), 738–744.
Sandelowski, M., Voils, C. I., & Barroso, J. (2006). Defining and designing mixed research synthesis
studies. Research in the Schools, 13(1), 29.
Schroeder, R. (2014). Exploring critical and Indigenous research methods with a research community:
Part I—The leap. In the Library with the Lead Pipe. Retrieved from https​://goo.gl/6P7Gm​9
Smith, L. T. (2000). Kaupapa Maori research. In M. Battiste (Ed.), Reclaiming Indigenous voice and
vision (pp. 225–247). Vancouver: UBC Press.
Smith, L. T. (2012). Decolonising methodologies: Research and Indigenous peoples. London: Zed Books
Ltd.
Steering Committee for the Review of Government Service Provision. (2014). Overcoming Indigenous
disadvantage: Key indicators 2014—report. Canberra: Productivity Commission.
Steering Committee for the Review of Government Service Provision. (2016). Overcoming Indigenous
disadvantage: Key indicators 2016. Canberra: Productivity Commission.
Stern, C., Jordan, Z., & McArthur, A. (2014). Developing the review question and inclusion criteria.
American Journal of Nursing, 114(4), 53–56.
Vass, G. (2013). ‘So. What is wrong with Indigenous education?’ Perspective, position and power beyond
a deficit discourse. The Australian Journal of Indigenous Education, 41(2), 85–96.
Victor, L. (2008). Systematic reviewing. Social Research Update, 54(1), 1–4.
Whittemore, R., & Knafl, K. (2005). The integrative review: Updated methodology. Journal of Advanced
Nursing, 52(5), 546–553.

Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published
maps and institutional affiliations.

Kevin Lowe is a Gubbi Gubbi man from southeast Queensland. He is a currently a Post-Doctoral Fellow
at Macquarie University working on an innovative, community focused research project on developing a
model of sustainable improvement in Aboriginal education. Kevin has had extensive, experience across
the education sector, including teaching, TAFE administrator, University lecturer, and Inspector, Aborigi-
nal Education in the NSW Board of Studies. Kevin has expertise in working with establishing successful

13
‘Aboriginal Voices’: An overview of the methodology applied… 229

programs with Aboriginal community organisations on establishing Aboriginal language policy and cur-
riculum development and its implementation. Over the last 20 years Kevin has led educational projects
with Aboriginal communities, schools and education systems that centre on the development of effec-
tive school-community learning partnerships. Recently Kevin has worked collaboratively with a team of
cross-institutional academics to review research across key areas of schooling and established the Abo-
riginal Voices a broad-base, holistic project which is developing a new pedagogic framework for teachers.

Christine Tennent is an Academic Liaison Librarian at the University of Sydney. She is an expert in pro-
cessing, and extracting value from big data, and was a key driver of the application of NVivo for this Sys-
tematic Literature Review. She works across multiple fields and disciplines to ensure rigorous collection
and analysis of data.

John Guenther is a researcher with 20 years’ experience working in overlapping fields of social inquiry,
typically in areas related to education and training, and its intersections with mental health and wellbe-
ing, justice, employment, child protection, and welfare. In the last 10 years, he has more intentionally
focussed on issues affecting Aboriginal and Torres Strait Islander people living in remote parts of Aus-
tralia. He led the Remote Education Systems project under the CRC for Remote Economic Participation
(2011-2016) and is currently the Research Leader for Education and Training with the Batchelor Institute
of Indigenous Tertiary Education, based in Darwin, Northern Territory. John is a leading academic in the
field of remote education and has published widely on his findings, often under the banner of ‘Red Dirt
Thinking’.

Neil Harrison is an Associate Professor in the Department of Educational Studies, Macquarie University,
Australia. His research focuses on the ways in which Indigenous knowledge of Country can be embedded
in the Australian Curriculum. His latest book (together with Juanita Sellwood), Learning and Teaching in
Aboriginal and Torres Strait Islander Education (Oxford) is used widely in teacher education programs
throughout Australia.

Cathie Burgess coordinates Aboriginal Studies, Aboriginal Community Engagement and the Master of
Education: Leadership in Aboriginal Education programs at the University of Sydney. Cathie’s research
involves community-led initiatives positioning Aboriginal cultural educators as experts through projects
such as Learning from Country in the City, Aboriginal Voices: Insights into Aboriginal Education and
Redfern Cultural Program.

Nikki Moodie (Gomeroi) is a Senior Lecturer in Indigenous Studies in the School of Social-Political Sci-
ences at the University of Melbourne. The focus of her research is Indigenous public policy and the soci-
ology of education, with a specific interest in relationality, networks, and identity.

Greg Vass is a currently a lecturer at Griffith University. Building on his experiences as a high school
teacher, his work is concerned with learner identities and schooling practices that impact on the experi-
ences and achievements of students. His research interests aim to investigate relationships between policy
enactment, pedagogy/curriculum, and educational inequities/privileges. His current research project is
focused on working with learning communities as they engage with the skills, knowledges and practices
that support Culturally Responsive Schooling.

13