Computer Assisted Language Learning

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/ncal20

Meeting the challenges of research bibliography in

MALL

Jack Burston

To cite this article: Jack Burston (2021) Meeting the challenges of research bibliography in MALL,
Computer Assisted Language Learning, 34:7, 813-819, DOI: 10.1080/09588221.2021.1987924

To link to this article: https://doi.org/10.1080/09588221.2021.1987924

Published online: 21 Oct 2021.

Submit your article to this journal

Article views: 384

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=ncal20
Computer Assisted Language Learning
2021, VOL. 34, NO. 7, 813–819
https://doi.org/10.1080/09588221.2021.1987924

EDITORIAL

Meeting the challenges of research bibliography in

MALL

The problem of unreported MALL studies

It is critical for researchers in Mobile-Assisted Language Learning (MALL)
to have a thorough grasp of the published literature. This is no less true
in the field of Computer-Assisted Language Learning (CALL), the research
domain most closely aligned with MALL. In MALL and CALL, as in
Applied Linguistics generally, it is common practice that any study sub-
mitted for publication include a literature review relating to the specific
topic under investigation. Such information is essential to demonstrate
the knowledgeability of the author(s) and, more importantly, to justify
the need for further research in the field. Any new publication is expected
to contribute new knowledge or, in the case of an intentional replication,
confirm, modify or refute previous claims. Likewise, such knowledge is
essential in the formulation of the research questions that motivate a
study. It is simply not possible to meaningfully articulate research ques-
tions without a comprehensive knowledge of what has already been pub-
lished about the topic under investigation.
Within CALL, the transition from institutional mainframe to personal
desktop computers (PCs) generated a massive increase in language acqui-
sition studies owing to the much greater accessibility of computer-based
resources, for investigators, instructors and students alike. As the number
of CALL publications increased and accumulated over the years, the need
to facilitate the compilation of bibliographic research references became
very apparent. In the mid-1980s, this resulted in the publication of two
comprehensive CALL bibliographies (Stevens, 1985; Stevens, Sussex, &
Tuman, 1986). The challenge was taken up again some twenty years later
by Kiliçkaya (2009). Notwithstanding these efforts, knowledge of published
CALL research leaves much to be desired. Beatty (2010, p. 189) attributes
this lack of knowledge to delays in conducting, reporting, publishing and
disseminating research and laments that ‘The nature of this delay has led
not just to an extensive duplication of effort, but also to a lack of rec-
ognition of leadership in CALL research …’. The bibliographical research

This article has been corrected with minor changes. These changes do not impact the academic content of
the article.
© 2021 Informa UK Limited, trading as Taylor & Francis Group
814 EDITORIAL

base of CALL studies continues to be a source of concern, as Levy,

Hubbard, Stockwell, and Colpaert (2015, p. 5) make clear:
As the use of technology in language learning has gone more mainstream, teach-
ers, developers, and even researchers increasingly present and publish with little
or no awareness that there is a decades-long research base for their endeavors.
There is a tendency to jump from digital bandwagon to digital bandwagon with
eyes solely on the newest technology “affordances” rather than on what the extant
CALL literature might do to ground their efforts.

As with the effect of PCs upon CALL, the advent of readily accessible
smartphones, tablet computers and other mobile devices during the past
decade has provided an enormous stimulus for MALL studies, and the
same need to facilitate the compilation of bibliographic research references.
The first, and up to now most extensive, step in this direction was Burston
(2013), an annotated MALL bibliography covering the previous twenty
years. However, despite the central importance of research bibliography,
its inadequacy constitutes an endemic shortcoming in MALL studies. One
very obvious proof of this is the very high number of papers written on
the same topic, most notably L2 vocabulary acquisition. According to
Burston and Arispe (2022), over the past 27 years more than 40% of all
experimental MALL studies have doggedly focused on vocabulary acqui-
sition. Moreover, that percentage has in fact increased over the past three
years. As anyone familiar with this literature can attest, it is devoid of
replication studies and involves much reinventing of the wheel. Not only
does this reflect badly upon MALL authors’ knowledge of the field, the
fact that such a high proportion of repetitive MALL studies has been
accepted for publication points to a similar lack of reference bibliography
knowledge on the part of journal editors, conference organizers and
reviewers.
The problem of bibliographical lacunae in MALL was raised in detail
recently in Burston (2021b). In that investigation, it was shown that despite
the existence of some 3000 MALL studies, MALL meta-analyses, i.e.
comprehensive comparative overviews of experimental MALL studies,
typically reference no more than a few dozen studies. On closer inspec-
tion, it was confirmed that in the four broad-based meta-analyses pub-
lished prior to 2012, over a half to more than three-quarters of possibly
relevant MALL studies were overlooked. In seven meta-analyses published
since 2012, with one exception, they accounted for only between 4% and
18% of MALL implementations during the periods they covered. The one
exception, Burston and Athanasiou (2020), was based on a MALL research
database that included 95% of the potentially relevant studies. It is to be
noted that that this database was compiled following the procedures
described in Burston (2021b). Following those procedures, Burston (2021b)
showed that one of the most comprehensive of previous MALL
 Computer Assisted Language Learning 815

meta-analyses, Burston (2015), omitted over a quarter of relevant studies

and that these omissions affected the accuracy of the results derived from
the data.
Compared to meta-analyses, which by definition constitute a compar-
ative, secondary, level of research, primary MALL studies are by their
very nature less comprehensive and their literature reviews even more
prone to significant omissions. In some cases this lack of attention to
MALL reference bibliography may be attributable to inexperience, i.e.
newcomers to the field. However, as the meta-analyses reviewed in Burston
(2021b) attest, the failure to account for MALL publications is character-
istic of even the most experienced researchers. This is not to say that
MALL practitioners take the importance of research bibliography lightly.
On the contrary, meta-analysis database searches in particular routinely
start with hundreds, if not thousands, of initial finds. Notwithstanding,
they just as routinely reject the near totality of these as irrelevant to
experimental MALL research. The fact is that there are serious obstacles
preventing researchers from properly identifying potentially relevant MALL
studies, to the point where authors do not even realize that they are
missing anything significant.

Sources of the problem

The fundamental reason why so many MALL studies go unreported in
the literature relates to the great diversity of research interests covered
by MALL. From the beginning, MALL studies have covered a wide range
of topics: mobile device ownership, software reviews, technical design,
pedagogical frameworks, learning theory, institutional infrastructure,
teacher training, student attitudes, learning outcomes, etc. So, too, despite
the preponderance of English as a lingua franca, some 270 MALL papers
have been published in nineteen other languages, including Arabic,
Bulgarian, Catalan, Chinese, Czech, French, German, Greek, Indonesian,
Italian, Japanese, Korean, Polish, Portuguese, Russian, Spanish, Thai,
Turkish and Ukrainian. Nonetheless, unless they themselves are writing
in another language, MALL researchers nearly always limit literature
reviews to studies written in English. Given the great diversity of topic
areas, not surprisingly, MALL studies appear in an enormous variety of
places. The most obvious are CALL, educational technology and mobile
technology journals, and associated conference proceedings. Yet combined,
these account for only about 10% of all MALL publications (Burston &
Giannakou, 2021). The rest occur in sources where few MALL researchers
would ever think to look, e.g. International Journal of Distributed and
Parallel Systems; Disability and Rehabilitation: Assistive Technology; Journal
of Hospitality, Leisure, Sport and Tourism Education; Journal of US-China
816 EDITORIAL

Public Administration; Executive Journal; Agro Food Industry Hi-Tech;

Journal of Engineering; Review of Public Administration and Management;
International Journal of Electronic Business; Journal of Clinical and
Counselling Psychology. In addition, MALL studies are to be found in
Masters theses and PhD dissertations, often written in languages other
than English and lodged in university libraries that are difficult to access.

Meeting the challenge

It goes without saying that no MALL research study or experimental
implementation project should ever be undertaken without first consulting
the existing research literature. This is a truism known even to neophytes.
Unfortunately, how this information is acquired leaves much to be desired.
There are two frequent starting points. Firstly, the web sites of the most
prominent CALL, educational technology and mobile technology journals
are targeted, where keyword searches are made, tables of content consulted
and abstracts scanned. The second major source is academic databases
such a Scopus, SSCI, ERIC, ProQuest. Rarely are both types of sources
employed and, even where this is the case, the bibliographical search is
bound to be incomplete. As indicated above, only about 10% of MALL
publications are to be found in the prominent journals. Likewise, the
majority of journals in which MALL studies appear is not indexed in the
academic databases. The only way that a comprehensive MALL research
database can be compiled is by complementing journal and academic
database searches with the systematic mining of references in published
papers. In essence, this involves extracting the references from all the
MALL studies initially identified then repeating the process in those
studies until no new references are found. Compiling a comprehensive
MALL reference bibliography significantly increases the time and effort
needed to undertake a research or experimental implementation project.
However, unless it is done, the resulting outcomes are destined to be
wanting. It is not possible to justify a MALL study with an inadequate
knowledge of published research literature. Well established observations
will be needlessly rehashed, avoidable mistakes will be repeated and prom-
ising lines of inquiry missed.
Practicing what is preached, two MALL bibliographies (Burston, 2021a)
have been compiled in conformity with the procedures described in
Burston (2021b), i.e. a combination of journal/conference publication
searches, academic databases and bibliographical mining. Both bibliogra-
phies are maximally inclusive with regard to the language in which studies
are written and the target language, L1 as well as L2. Both extend from
1994 through the end of 2020. For each entry, where possible, a hyperlink
is included to a download source. While the majority of these are free,
 Computer Assisted Language Learning 817

many can only be obtained through a library subscription or direct pay-

ment. Fortunately, unless entire journal volumes or books are required,
costs are generally not excessive.
The first bibliography, General MALL Bibliography 1994–2020, is com-
prehensive in nature and includes the entire range of possible MALL topic
areas. It contains over 3800 entries. To facilitate access to summaries of
MALL studies, this bibliography is accompanied by a table consisting of
73 overviews and meta-analyses that have appeared since 2006. The second
bibliography, Mobile-Assisted Language Learning: A Selected Annotated
Bibliography of Implementation Studies 1994–2020, is a subset of the first
that is restricted to just experimental studies. These are defined as those
which involve the application of mobile-based or mobile-accessible apps
and/or mobile device affordances (e.g. audio/video recording, picture/note
taking) for the teaching and learning of languages in a defined learning
environment with specified participants and learning conditions. It con-
tains over 1270 entries, each accompanied by a brief (∼90 word) synopsis.
To the extent that the publication provides such information, the summary
includes essential details relating to the country in which the experimental
implementation took place, the institutional environment, mobile device
used, native language, target language, language proficiency level of par-
ticipants, targeted skill area(s), treatment details, sample size, intervention
duration and outcomes.
As might be imagined, the compilation of these MALL bibliographies
is a long running, ongoing, process. The general MALL bibliography
derives from the database underlying Burston (2013) and the annotated
experimental MALL bibliography is an extension of the latter publication.
In the hopes of helping colleagues meet the challenges of MALL research
bibliography, a hyperlink to an online version of these bibliographies is
included in the References that follow. Colleagues who avail themselves
of these bibliographical resources are encouraged to advise the author of
any errors that might be encountered (e.g. reference details, mistranslations
of non-English titles, dead hyperlinks, etc.). While these online MALL
reference databases can certainly guide the direction of MALL research
studies and planning of experimental implementations, it is important to
remember that they should not be used to the exclusion of personal
MALL literature searches. Although every effort will be made to keep
these bibliographies up to date, new studies as well as newly discovered
references to earlier published studies appear constantly. Over the last
eight years, MALL publications have consistently exceeded three hundred
annually, so a significant number of new bibliographical entries is always
waiting to be discovered. This is especially true of publications that have
appeared within the last two years, owing to the unavoidable time delay
before references to them appear in MALL studies and academic databases.
818 EDITORIAL

As colleagues discover new references, it would be especially appreciated

if they could be sent on to the author (with download hyperlinks, and
an e-copy, if possible). With the ever growing prominence of MALL
within the domain of language acquisition studies, as a collaborative effort,
the ongoing compilation of these MALL bibliographies can only get better,
to the benefit of all researchers and practitioners working in the field.
As MALL bibliographies become more comprehensive and studies more
thoroughly researched, inevitably both will require more space in papers
submitted for publication. Keeping within prescribed word length con-
straints, which already can be problematic, is bound to become an issue
for editors no less than authors. One possible solution to this challenge
is to move supporting materials (e.g. Tables, Appendices) to a hyperlinked
online venue either provided by the publisher or a professional website
such as that made available on Academia.edu. That being said, there are
limits to what materials can be displaced this way. References, for example,
must remain with the text and the illustrative value of figures would be
greatly diminished were they not to be simultaneously viewable. Despite
the considerable pressure of publishers to limit the size of published
volumes, editors will need to accept that the more comprehensive a study
is, the more space it will require and increase word limits accordingly.
This may very well result in fewer, but better, studies being published,
which would not be such an undesirable outcome after all.

Disclosure statement
No potential conflict of interest was reported by the authors.

Notes on contributor
Jack Burston holds the position of Honorary Research Fellow in the Language Centre
of the Cyprus University of Technology. His current research is focused on Mobile-Assisted
Language Learning and advanced-level foreign language instruction. Jack is a member
of the Editorial Board of the ReCALL Journal and Language Learning & Technology.

ORCID
Jack Burston http://orcid.org/0000-0003-2905-5585

References
Beatty, K. (2010). Teaching and researching computer-assisted language learning (2nd ed.).
Oxford, UK and New York, NY: Routledge. Retrieved from https://www.routledge.com/
Teaching--Researching-Computer-Assisted-Language-Learning/Beatty/p/book/
9781408205006
 Computer Assisted Language Learning 819

Burston, J. (2013). Mobile-assisted language learning: A selected annotated bibliography

of implementation studies 1994-2012. Language Learning & Technology, Special Volume
on Mobile-Assisted Language Learning, 17(3), 157–224. Retrieved from https://www.
lltjournal.org/item/2830
Burston, J. (2015). 20 Years of MALL project implementation: A meta-analysis of learn-
ing outcomes. ReCALL, 27(1), 4–20. Retrieved from https://www.cambridge.org/core/
journals/recall/article/twenty-years-of-mall-project-implementation-a-metaanalysi
s-of-learning-outcomes/EE135E77EB10AD5DB8D1FA3F2CCC9317
Burston, J. (2021a). MALL Research Bibliographies. Retrieved from https://cut.academia.
edu/JBurston/MALL-Bibliographies#mallbibliographies
Burston, J. (2021b). Unreported MALL studies: What difference do they make to pub-
lished experimental MALL research results? In V. Morgana & A. Kukulska-Hulme
(Eds.), Mobile assisted language learning across educational contexts. Oxford, UK and
New York, NY: Routledge Focus Collection. Retrieved from https://www.routledge.com/
Mobile-Assisted-Language-Learning-Across-Educational-Contexts/Morgana-Kukulska-
Hulme/p/book/9780367521745
Burston, J., & Arispe, K. (2022). Experimental MALL research in SLA: Past, present, and
future. In N. Ziegler & M. González-Lloret (Eds.), The Routledge handbook of technol-
ogy & SLA. Oxford, UK and New York, NY: Routledge. Retrieved from https://www.
routledge.com/The-Routledge-Handbook-of-Second-Language-Acquisition-and-
Technology/Ziegler-Gonzalez-Lloret/p/book/9780815360773
Burston, J., & Athanasiou, A. (2020). Twenty-five years of MALL experimental implemen-
tation studies: What do we really know about it?. In A. Anjudar (Ed.), Recent tools for
computer- and mobile-assisted foreign language learning (pp. 35–59). Hershey, PA: IGI
Global. Retrieved from https://www.igi-global.com/book/recent-tools-computer-
mobile-assisted/230992
Burston, J., & Giannakou, K. (2021). MALL language learning outcomes: A comprehen-
sive meta-analysis 1994–2019. ReCALL Journal, 1–22. Retrieved from https://www.
cambridge.org/core/journals/recall/article/mall-language-learning-outcomes-a-comprehe
nsive-metaanalysis-19942019/4FE89DA5D55A233B4E34650E96FADB9D
Kiliçkaya, F. (2009). A topical bibliography on Computer-Assisted Language Learning
(CALL). Unpublished. Retrieved from https://files.eric.ed.gov/fulltext/ED506357.pdf
Levy, M., Hubbard, P., Stockwell, G., & Colpaert, J. (2015). Research challenges in CALL.
Computer Assisted Language Learning, 28(1), 1–6. doi:10.1080/09588221.2014.987035
Stevens, V. (1985). Annotated bibliography of publications concerning computers in
education: Emphasis on microcomputers in ESL and language learning. Unpublished,
1–232. Retrieved from https://files.eric.ed.gov/fulltext/ED266646.pdf
Stevens, V., Sussex, R., & Tuman, W. (1986). A bibliography of computer-aided language
learning (Ams Studies in Education). Brooklyn, NY: Ams Pr Inc. Retrieved from https://
www.amazon.com/Bibliography-Computer-Aided-Language-Learning-Education/
dp/0404126669

Jack Burston
Language Centre, Cyprus University of Technology,
Limassol, Cyprus
jack.burston@cut.ac.cy