10 Years after Sutton (2012):

Quo Vadis Feedback Literacy?

A Bibliometric Study

Imelda Gozali, Nunung Suryati, Abdul Syahid

Postgraduate Study
English Language Education, Faculty of Letters
Universitas Negeri Malang
 Giving and receiving feedback is often taken for
granted; we think we know how to give (and
receive) feedback. However, our teaching
experience tells us that not all students appreciate
feedback, nor know what to do with the feedback. I
personally feel that feedback literacy is urgently
needed.

My personal advocacy
TABLE OF CONTENTS
(1/2)

01 02 03
Literature
Introduction Review Methodology
Background and On feedback literacy PRISMA for the data
research questions and bibliometric search protocol and
studies data analysis
TABLE OF CONTENTS
(2/2)

04 05 06
Results Discussion Conclusion
Performance and What the data tell us Summary,
science mapping about feedback implications, future
analyses literacy research research and
limitations
Introduction
Background and research
questions
01
INTRODUCTION:
Feedback Literacy
Paul Sutton (2012)
The three dimensional aspects of
Feedback Literacy (extrapolated
from Academic Literacy)
Epistemological
(summative and
formative)
Ontological – shaping
learners’ being in
response to feedback
Practical – taking the
appropriate action on
the feedback
INTRODUCTION: Students’ Feedback Literacy

Students’ feedback literacy is “The

understandings, capacities and dispositions
needed to make sense of information and use
it to enhance work or learning strategies.”

—Carless & Boud,

2018, p.1315
INTRO: Framework for Students’ Feedback
Literacy

(Carless & Boud, 2018, p. 1319)

INTRO: Research on various aspects of
feedback literacy – students (conceptual)

3 dimensions of students’ Added 3 more variables to Integrating students’ feedback

feedback literacy Carless and Boud (2018) literacy to the curriculum

—Chong (2020) —Molloy et al. (2020) —Malecka et al. (2020).

Winstone et al. (2020)

3 Facets of students’ feedback 2 constructs of students’ The variables within cognitive

literacy profile
—Han and Xu (2019a)
feedback literacy: Cognitive and socio-affective readiness
and socio-affective
—Li and Han
readiness
(2020)
—Han and Xu (2019b)
 INTRO: Research on various aspects of
feedback literacy – students (empirical)
Fernandez-Toro & Duensing (2020)
01.
examined the relationship between the students’ feedback literacy and peer-marking

02. Wei et al. (2020)

Studied students’ expectation of teachers’ feedback

03. Ketonen et al. (2020)

studied the development of secondary school students’ feedback literacy in a science class
through a year-long peer formative assessment activity
 INTRO: Research on various aspects of
feedback literacy – teachers
Boud & Dawson (2021)
01. unveiled 19 empirically-derived category of teacher’s feedback literacy competencies,
which are grouped into macro, meso, and micro levels

02. Heron et al. (2021)

investigated the role of feedback talk in the relational dimension of teachers’ feedback
literacy

03. Deneen & Hoo (2021)

examined of the role of students’ peer-feedback and self-assessment in mediating teachers’
feedback literacy
INTRO: Background

● In view of this mounting interest on

feedback literacy, it seems propitious to
carry out a synthesis on research works
on the topic, as a way of commemorating
the 10th year anniversary of Sutton’s
(2012) seminal work
● This is best accomplished by a
bibliometric study
INTRO: Research Purposes
RP1
To carry out a performance
analysis: a descriptive
analysis of the constituents of feedback literacy research
performance of various
constituents (authors,
institutions, countries,
journals) (Donthu et al.,
2021) in feedback literacy
research
RP2
To conduct a science mapping analysis:
examines the relationship between
(Donthu et al., 2021), through co-word
analysis, co-citation analysis, and co-
authorship analysis. The analyses reveal,
respectively, the conceptual structure,
intellectual structure, and the social
structure of the field.
 Literature
Review
On bibliometric studies and
research reviews on feedback
02
LIT REVIEW: 1. Bibliometric
studies

2,000 studies in Scopus

in 2020 CiteSpace,
VOSViewer, Gephi,
Gaining Leximancer
popularity Various Tools

Advantages References
Can analyse large Donthu et al.
dataset quantitatively (2021), Mukherjee
and qualitatively et al. (2022)
LIT REVIEW: 2. Research Reviews on
Feedback (Systematic Review)
Altamini & Masood
(2021) Chong (2019)
Electronic WCF WCF

Winstone et al.
Czaholi (2021) (2017)
Oral CF Learners’ agentic
engagement with
feedback
 LIT REVIEW: 2. Research Reviews on
Feedback (Bibliometric Studies)

Chin & Chew (2021)

01. examined 412 articles from Scopus database published between 1991 and 2021 dealing
with electronic feedback. Used VOSViewer.

02. Nguoi & Habil (2021)

retrieved 276 Scopus-indexed journal articles from the year 1985 to 2021, with peer
feedback as the search criteria. Used VOSViewer.

03. Xie (2022)

made use of CiteSpace to conduct a bibliometric study investigating teachers' feedback,
drawing from Chinese Academic Journal’s related publications of the past decade.
Methodology
Data collection (PRISMA for
search protocol) and data
03
analysis
METHODOLOGY: Design

Bibliometric analysis Bibliometric analysis

to synthesize the performance of And to identify the scientific
the various research on feedback structure of the field for future
literacy in the past decade research direction (Donthu et al.,
2021)
METHODOLOGY: Data
collection

1. Database search
Scopus

2. Search string
TITLE-ABS-KEY-AUTH
("feedback literac*") AND
3. Data (EXCLUDE
(DOCTYPE,"cr") OR
cleaning EXCLUDE
Remove duplicates (DOCTYPE,"er"))
METHOLODOGY: Data
analysis
Performa Science
nce mapping
analysis analysis
Conceptual
Overview
structure

Intellectual
Sources
structure

Social
Authors
structure

Bibliometrix 4.0, an R-tool Documents

(Aria & Cuccurullo, 2017)
 Results
Performance analysis and
science mapping analysis
04
RESULTS: 1. Overview (1/4) – Main
information
Description Results
Timespan 2012:2022
Sources (Journals, Books, etc) 52
Documents 105
Average years from publication 1.3
Average citations per documents 11.76
Average citations per year per doc 3.828
References 3599
DOCUMENT TYPES
article 90
book chapter 6
conference paper 4
letter 1
note 3
review 1
RESULTS: 1. Overview (2/4) – Main
information
Description Results
DOCUMENT CONTENTS
Keywords Plus (ID) 136
Author's Keywords (DE) 272
AUTHORS
Authors 208
Author Appearances 291
Authors of single-authored documents 16
Authors of multi-authored documents 192
AUTHORS COLLABORATION
Single-authored documents 24
Documents per Author 0.505
Authors per Document 1.98
Co-Authors per Documents 2.77
Collaboration Index 2.37
RESULTS: 1. Overview (3/4) – Annual scientific
production
RESULTS: 1. Overview (4/4) – 3-field plot
RESULTS: 2. Sources (1/2)
Top 20 based on Bradford’s Law
Source Rank Freq cumFreq Zone
ASSESSMENT AND EVALUATION IN
HIGHER EDUCATION 1 39 39 Zone 1
INNOVATIONS IN EDUCATION AND
TEACHING INTERNATIONAL 2 4 43 Zone 2
TEACHING IN HIGHER EDUCATION 3 4 47 Zone 2
ASSESSING WRITING 4 3 50 Zone 2
BMC MEDICAL EDUCATION 5 3 53 Zone 2
ACM INTERNATIONAL
CONFERENCE PROCEEDING
SERIES 6 2 55 Zone 2
ASIA-PACIFIC EDUCATION
RESEARCHER 7 2 57 Zone 2
MEDICAL TEACHER 8 2 59 Zone 2
TEACHING AND LEARNING
INQUIRY 9 2 61 Zone 2
VOPROSY OBRAZOVANIYA /
EDUCATIONAL STUDIES MOSCOW 10 2 63 Zone 2
RESULTS: 2. Sources (2/2)
Top 20 based on Bradford’s Law
Source Rank Freq cumFreq Zone
ACADEMIC MEDICINE 11 1 64 Zone 2
ACTIVE LEARNING IN HIGHER EDUCATION 12 1 65 Zone 2
ADVANCES IN HEALTH SCIENCES
EDUCATION 13 1 66 Zone 2
ANATOMICAL SCIENCES EDUCATION 14 1 67 Zone 2
APPLICATIONS OF WORK INTEGRATED
LEARNING AMONG GEN Z AND Y STUDENTS 15 1 68 Zone 2
ASSESSMENT AS LEARNING: MAXIMISING
OPPORTUNITIES FOR STUDENT LEARNING
AND ACHIEVEMENT 16 1 69 Zone 2
BANGLADESH JOURNAL OF MEDICAL
SCIENCE 17 1 70 Zone 2
COGENT EDUCATION 18 1 71 Zone 2
COMPUTER ASSISTED LANGUAGE LEARNING 19 1 72 Zone 3
CRITICAL STUDIES IN EDUCATION 20 1 73 Zone 3
RESULTS: 3. Authors (1/3)
Authors’ impact by h-index
RESULTS: 3. Authors (2/3)
Top authors’ production over time
RESULTS: 3. Authors (3/3)
Top 10 most-cited countries by corresponding
authors
No. Country Articles Freq SCP MCP MCP_Ratio
1 AUSTRALIA 17 0.25 10 7 0.4118
2 UNITED KINGDOM 16 0.2353 15 1 0.0625
3 CHINA 10 0.1471 6 4 0.4
4 HONG KONG 8 0.1176 5 3 0.375
5 FINLAND 2 0.0294 1 1 0.5
6 IRELAND 2 0.0294 2 0 0
7 KOREA 2 0.0294 2 0 0
8 NETHERLANDS 2 0.0294 2 0 0
9 SINGAPORE 2 0.0294 1 1 0.5
10 SOUTH AFRICA 2 0.0294 2 0 0
RESULTS: 4. Documents (1/5)
Top 10 most-cited documents
No Authors Year Sources Titles Global
Citations
ASSESSMENT & The development of student
CARLESS D &
1 2018 EVALUATION IN HIGHER feedback literacy: enabling uptake 346
BOUD D
EDUCATION of feedback
INNOVATIONS IN
EDUCATION AND Conceptualizing feedback literacy:
2 SUTTON P 2012 102
TEACHING knowing, being, and acting
INTERNATIONAL
ASSESSMENT &
MOLLOY E, BOUD Developing a learning-centred
3 2020 EVALUATION IN HIGHER 75
D, HENDERSON M framework for feedback literacy
EDUCATION
ASSESSMENT &
Feedback loops and the longer-
4 CARLESS D 2019 EVALUATION IN HIGHER 65
term: towards feedback spirals
EDUCATION
Teacher feedback literacy and its
CARLESS D & TEACHING IN HIGHER
5 2020 interplay with student feedback 57
WINSTONE NE EDUCATION
literacy
RESULTS: 4. Documents (2/5)
Top 10 most-cited documents
No Authors Year Sources Titles Global
Citations
‘Only true friends could be cruelly
ASSESSMENT &
honest’: cognitive scaffolding and social-
6 XU Y & CARLESS D 2017 EVALUATION IN HIGHER 43
affective support in teacher feedback
EDUCATION
literacy
ASSESSMENT &
Reconsidering student feedback literacy
7 CHONG SW 2021 EVALUATION IN HIGHER 42
from an ecological perspective
EDUCATION
Medical Student Perceptions of
BOWEN L, MASHALL
Feedback and Feedback Behaviors
8 M, MURDOCH-EATON 2017 ACADEMIC MEDICINE 35
Within the Context of the "Educational
D
Alliance"
ASSESSMENT & Improving the student learning
9 HILL J & WEST H 2020 EVALUATION IN HIGHER experience through dialogic feed-forward 28
EDUCATION assessment
ASSESSMENT & The development of student feedback
10 HAN Y & XU Y 2020 EVALUATION IN HIGHER literacy: the influences of teacher 27
EDUCATION feedback on peer feedback
RESULTS: 4. Documents (3/5)
Word cloud
RESULTS: 4. Documents (4/5)
Word growth
RESULTS: 4. Documents (5/5)
Trend topics
RESULTS:
5.
Conceptual
structure
(1/4)
Co-
occurrence
network
(authors’
keywords)
 RESULTS: 5. Conceptual structure (2/4)
Co-occurrence network (authors’ keywords)
Keywords in Cluster 1 Keywords in other clusters Cluster

feedback literacy teacher feedback literacy course design school education

feedback formative assessment covid-19 adolescents

2
student feedback literacy peer review ecological perspective communicative competence
peer feedback academic writing engagement feedback at school
peer assessment assessment literacy feedback dialogue field research
assessment curriculum internal feedback questionnaire interview
higher education online feedback learner agency feedback literacies
3
self-assessment academic development sociomaterial interculturality
teacher feedback agency student agency assessment feedback
4
thematic analysis
RESULTS:
5.
Conceptual
structure
(3/4)
Thematic
map
(authors’
keywords)
RESULTS: 5. Conceptual structure (4/4)
Thematic map (authors’ keywords)
Quadrant Cluster Label
student feedback literacy, peer feedback, self-
1 (motor)
assessment, feedback literacies

feedback literacy, higher education, formative

2 (basic)
assessment, interprofessional education

3 (niche) school education

4 (emerging/declining) assessment feedback
RESULTS:
6.
Intellectual
structure -
Co-citation
network
(papers)
RESULTS:
7. Social
structure -
Collaboratio
n network
(institutions)
RESULTS:
7. Social
structure -
Collaboratio
n network
(authors)
 DISCUSSI
ON
What the data tell us about
Feedback Literacy research
05
DISCUSSION
1. Paucity of feedback research in general

This study
105 documents on
feedback literacy over
Chin & Chew
10 years
(2021)
412 articles on electronic
feedback over 30 years

Nguoi & Habil (2021)

276 articles on peer
feedback over 36 years
DISCUSSION
2. The dominance of higher
education as the setting of
feedback literacy research as
seen from the relevant
(impactful) journals
1. Assessment and Evaluation
in Higher Education
2. Studies in Higher Education
3. Teaching in Higher
Education
DISCUSSION
Language
Education
learning
3. The variety of research field
for feedback literacy
Health
Medicine
sciences

Early
Mathematics
childhood
DISCUSSION
Prof. David Carless

Prof. David Boud

4. Prominent authors in
feedback literacy research
showing their specificity in
feedback literacy (not featured Prof. Naomi Winstone
in the top 10 of other feedback
bibliometric research) Prof. Phil Dawson

Dr. Joanna Tai

DISCUSSION

5. Locus of research in
feedback literacy: Australia
(CRADLE, Deakin
University), the UK, China,
Hong Kong
DISCUSSION – 6. the central role of 3 articles
in the intellectual structure of FL research

Students’
feedback • Carless and Boud (2018)
literacy
Feedback
in higher • Hattie and Timperley (2007
edu
Peer • Nicol et al. (2014)
feedback
DISCUSSION

Student
● 7. Trending topics in Peer
feedback
feedback literacy feedback
research
literacy

Peer Self-
assessment assessment
DISCUSSION

This study
● 8. The rather monolithic nature One Three to

Other bibliometric feedback studies

of feedback literacy research major five
topics as compared to other cluster of clusters of
bibliometric feedback research
keywords keywords
DISCUSSION: Quo vadis feedback
literacy?

Feedback literacy research is on the

rise
• feedback literacy as the avenue for future
research in the feedback milieu (Lee, 2022)
Feedback literacy as the foundation of
feedback research
• feedback literacy as one of the conceptual
underpinnings to enrich research on written
corrective feedback (Chong, 2019)
DISCUSSION: Quo vadis feedback
literacy?

Future direction
• Online feedback
• socio-cultural, socio-emotional, and personal factors
• Dialogic feedback

More participation from other context and

countries
• More diversification needed for context (outside of
higher education) and countries (outside of Australia,
UK, Hongkong and China)
CONCLUS
ION
Summary, limitation, and
future research
06
SUMMARY
(1/2)
Source analysis
• the Assessment and Evaluation in Higher Education
journal (highest productivity and impact)
Author analysis
• Prof. David Carless (highest productivity and impact)

Document analysis
• “The development of student feedback literacy:
enabling uptake of feedback,” (Carless & Boud, 2018)
• Trending topics: “student feedback literacy”, “peer
feedback”, and “peer assessment.”
SUMMARY
(2/2)
Conceptual mapping
• research themes are still tightly bound to the same concept
(“feedback literacy”) and have not undergone significant
diversification
Intellectual mapping
• Carless and Boud (2018) – students’ feedback literacy
• Hattie and Timperley (2007) – feedback in higher education
• Nicol et al. (2014) – peer feedback
Social mapping
• Australia is the most productive country in feedback literacy
studies perhaps owing to the establishment of the CRADLE
center in Deakin University
LIMITATION

More thorough data cleaning

Enrichment with other visualization

tool (e.g. VOSViewer)

Complemented with in-depth content

analysis (systematic review)
FUTURE
RESEARCH

• can potentially unpack other

Content
aspects of the field such as article
analysis
type (conceptual vs empirical)
Other • Bibliometric study on assessment
bibliometric
literacy
study
 THANK
I am grateful for comments or questions

Imelda Gozali
Imelda.gozali.2102219@students.um.ac.id

S
CREDITS: This presentation template was created by Slidesgo, including
icons by Flaticon, and infographics & images by Freepik
Please keep this slide for attribution