Computers & Education: Hui-Chen Lin, Gwo-Jen Hwang, Shao-Chen Chang, Yaw-Don Hsu

Computers & Education 173 (2021) 104266
Contents lists available at ScienceDirect
Computers & Education

journal homepage: www.elsevier.com/locate/compedu
Facilitating critical thinking in decision making-based professional

training: An online interactive peer-review approach in a flipped
learning context
Hui-Chen Lin a, b, Gwo-Jen Hwang a, *, Shao-Chen Chang a, Yaw-Don Hsu b
a
Graduate Institute of Digital Learning and Education, National Taiwan University of Science and Technology, Taiwan
b
Department of Neurology, Tri Service General Hospital, National Defense Medical Center, Taiwan
A R T I C L E I N F O A B S T R A C T
Keywords: Designing activities for learners to deal with problems and cultivate their higher order thinking is
Applications in subject areas critical in professional training. Peer review is to allow a group of learners with similar abilities
Interactive learning environments and knowledge background to promote their higher order thinking and reflective thinking
Pedagogical issues
through mutual observation and comments. However, in the conventional peer-review approach,
Teaching/learning strategies
the interaction between assessors and assessees is one-way. That is, assessors cannot receive any
responses from assessees, so they have no opportunity to reflect on their comments. The online
interactive peer-review approach aims to enable assessees to express their points of view and to
allow assessors to understand whether their ratings and comments are accurate or helpful to
assessees. During the interaction, both assessors and assessees have the opportunity for reflection,
which then promotes the quality of ratings and comments and facilitates their learning effec
tiveness and critical thinking. The present study developed an online interactive peer-review
system and applied it to the health assessment learning course for nurse practitioners (NP). An
experiment was designed to explore the effects of different online peer-review approaches on NP
students’ learning achievement and higher order thinking. Also, the study recorded the students’
peer-review content and analyzed its effects on the quality of their peer review. The experimental
group adopted the online interactive peer-review approach, while the control group adopted the
conventional online peer-review approach in a flipped learning context. The results indicated that
integrating the online interactive peer-review learning approach could not only strengthen the NP
students’ knowledge and clinical skills, but could also significantly improve their critical thinking
tendency and reflective thinking. Moreover, based on the analytic findings of the peer-review
content, students using the online interactive peer-review approach were better able to signifi
cantly address more specific suggestions to assist their peers in improving their health assessment
performance than those using the conventional online peer-review approach.
1. Introduction
Developing students’ higher order thinking capabilities, including problem solving and critical thinking, has been regarded as a
* Corresponding author.
E-mail addresses: ceciliatsgh@gmail.com (H.-C. Lin), gjhwang.academic@gmail.com (G.-J. Hwang), shao.chen76@gmail.com (S.-C. Chang),
hsu1487@gmail.com (Y.-D. Hsu).
https://doi.org/10.1016/j.compedu.2021.104266
Received 22 June 2020; Received in revised form 21 June 2021; Accepted 22 June 2021
Available online 25 June 2021
0360-1315/© 2021 Elsevier Ltd. All rights reserved.
H.-C. Lin et al. Computers & Education 173 (2021) 104266
crucial educational objective in the 21st century. Researchers have pointed out that in the social environment with rapidly changing
information, in addition to knowledge transfer, teachers should also assist students in developing their problem-solving ability, and
guide them to collect data for their analysis, judgement, and decision making (King & Kitchener, 1994; Schommer-Aikins & Hutter,
2002). In recent years, the issues regarding cultivating students’ higher order thinking abilities such as critical thinking and problem
solving have already been extensively explored in the fields of engineering, medicine, science and social sciences (Magrabi, Pasha, &
Pasha, 2018; Maudsley & Strivens, 2000; Román-González, Pérez-González, & Jiménez-Fernández, 2017; Tümkaya, Aybek, & Aldaş,
2009).
However, in traditional instruction, students heavily rely on the knowledge delivered by the teacher, and generally have less time
to practice with assistance from the teacher in class (Hsia & Sung; 2020; Lei et al., 2019). In view of this, many teachers adopt flipped
learning, which advocates reversing the sequence of teachers’ lectures and students’ homework (Bergmann & Sams, 2012; Bond,
2020). With self-learning through instructional videos or media before the class, the time for homework and discussion is shifted to the
class so that students can have more opportunities to practice and interact with their peers and teacher (Baytiyeh & Naja, 2017; Sun &
Wu, 2016). Researchers have indicated that flipped classrooms not only engage students in active learning in the before-class stage, but
also improve their learning engagement in class (Freeman et al., 2014; Prince, 2004). A meta-analysis study conducted by Strelan,
Osborn, and Palmer (2020a), who analyzed 198 articles, further showed that flipped classrooms enable teachers to design and engage
students in well-structured in-class activities, which could promote their active learning and have a positive impact on their learning
performances.
On the other hand, scholars have emphasized that integrating proper learning strategies into flipped learning is key to intensifying
students’ higher order thinking (Chang & Hwang, 2018; Ye, Chang, & Lai, 2019). However, to enable learners to interact in an effective
manner, it is important to provide the necessary guidance. Peer review is such a learning strategy which allows learners to assess their
peers’ performance by rating or providing comments based on the rubrics provided by the teacher; it can strengthen their
problem-solving capability, promote student-centered learning, encourage active learning, and facilitate in-depth learning (Çevik,
2015; Chang, Hsu, & Jong, 2020). Andrade and Du (2005) specified that peer review enabled students to understand teachers’ per
spectives and the standards of assessment. When students learned with this approach, they could reflect on their peers’ work and
re-examine their own comprehension of the learning content. Previous studies also indicate that peer review is an effective strategy in
case analysis or courses focusing on decision making and judgment development. For example, in a health assessment activity in a
nursing course, students observe their peers’ performance and are aware of their lack of health assessment skills and judgment, which
helps promote reflection (Morrell-Scott, 2018; Solheim, Plathe, & Eide, 2017), and improves their reasoning skills, communication
skills and physical examination skills (Abelsson & Bisholt, 2017; Maas et al., 2014). Lin (2019) further denoted that using the
peer-review strategy in flipped learning could encourage peers’ reflection and improvement, which helps to enhance students’ higher
order thinking capabilities.
Despite the numerous benefits of peer review identified in these studies, peer review in conventional teaching is mostly one-way
review. That is, assessors provide ratings and comments for assessees, while assessees have no chance to express their opinions on the
ratings and comments they receive. During one-way peer review, students feel insecure about the peer review and believe that their
peers’ assessments lack objectivity (Lee & Norbaizura, 2016; Topping, 2009). Also, assessees are not able to defend their performance
in front of assessors (PlanasLladó et al., 2014). Online interactive peer review allows assessees opportunities to respond to feedback, to
clarify their performance, and to further communicate with their assessors. It enables interaction between assessees and assessors,
which can better facilitate learning effectiveness (Brinko, 1993).
Above all, online interactive peer review increases the interaction and communication between peers, provides assessees with
opportunities to respond to assessors’ comments, and allows assessors to have a better understanding of assessees’ points of view. By
increasing the communication with peers, students repeatedly think about their proposed ideas and come up with solutions, which is
conducive to facilitating learning effectiveness, critical thinking, and reflection (Filius et al., 2018). As a result, the present study
developed an online interactive peer-review system, and applied it in a health assessment course for nurse practitioners in the context
of flipped learning. Through the experiment, the study aimed to explore the following research questions:
(1) Did students using the online interactive peer-review approach outperform those using the conventional online peer-review
approach in terms of learning achievement?
(2) Did students using the online interactive peer-review approach outperform those using the conventional online peer-review
approach in terms of health assessment skills?
(3) Compared with those using the conventional online peer-review approach, did students using the online interactive peer-review
approach show higher critical thinking tendency?
approach show higher self-efficacy?
approach show higher reflective thinking?
(6) How did these two different online peer-review approaches affect the quality of students’ peer reviews?
2
2. Literature review
2.1. Flipped learning
Flipped learning is a teaching approach which shifts teachers’ traditional lectures from the in-class to the pre-class stage, so that the
teacher has more time to guide students to practice, solve their learning problems, or engage them in applying knowledge or in
discussing in class. By self-learning through instructional videos or media, this transformation allows students to have more time for
practice and interaction under the guidance and support of the teacher (Bergmann & Sams, 2012). Zainuddin and Perera (2019)
reported that engaging students in analyzing and discussing in class facilitated better interactions between them, as well as
strengthening their comprehension and the competences of applying the acquired knowledge. Additionally, this learning process
enabled students to have more responsibility and intrinsic motivation in self-learning. Heo and CHun (2018), who guided students to
propose problems, solve the problems, have group discussions, and share their reflection diaries and discussion through the online
system in a flipped mathematics classroom, reported that the approach promoted students’ learning engagement, active learning, and
self-efficacy.
Researchers have further revealed that with an effective design of learning activities, teachers can instruct students to comprehend,
analyze, and even solve problems through discussion with classmates and teachers in flipped learning. From the perspective of social
constructivism, interaction with peers implies better opportunities for sharing and constructing knowledge (Adams, 2006). This im
plies that guiding students to interact with peers with proper supports will enable them to perceive things from diverse perspectives
and make reflections, which is beneficial for them to develop higher order thinking capabilities, including problem solving and critical
thinking (Karabulut-Ilgu, Yao, Savolainen, & Jahren, 2018; Kong, 2014). These competences are particularly important for such
application domains that require making immediate decisions to solve problems such as nursing training.
In recent years, several studies have revealed the need to apply flipped learning to nursing courses (Chiou, Su, Liu, & Hwang, 2015;
Dehghanzadeh & Jafaraghaee, 2018; Park & Park, 2018). For example, Hanson (2016) implemented a group discussion of case studies
in clinical practice, in which the nursing students responded that flipped learning enabled them to have deeper understanding and
application of pharmacological knowledge to patients, and enhanced their critical thinking skills. Another study conducted by Lee and
Park (2018) also reported the importance of flipping nursing training courses for improving learners’ nursing skills and
problem-solving performances.
Furthermore, researchers have implied the challenges of implementing the flipped classroom. Strayer (2012) denoted that, without
the appropriate guiding strategies, students would feel frustrated because they were not able to connect the pre-class knowledge to the
in-class activities, which then influenced their learning performance. Despite the challenges, according to a meta-analysis study
conducted by Strelan, Osborn, and Palmer (2020b) on nearly 50 studies, students are generally more satisfied with flipped classrooms
compared to traditional approaches. Researchers have disclosed that a flipped learning environment requires more active learning
strategies in classroom teaching to enhance learning effectiveness, for example, student presentations, group problem-solving activ
ities, self- and peer-assessment, and group discussions (Kim, Kim, Khera, & Getman, 2014). Therefore, it is necessary to incorporate
effective learning strategies or tools into flipped learning (Chang et al., 2020; Ye et al., 2019).
2.2. Peer review
Peer review refers to a group of learners with a similar knowledge background who play the roles of both assessors and assessees.
They compare and analyze the advantages and disadvantages of each other’s works, comment on and question the accuracy of the
works, and advise others how to modify them. Also, because peer review allows learners to have the opportunity to compare others’
work while observing, they can make reflections on their performances, which could promote their cognitive growth through mutual
observation and learning (Topping, 1998). There are two roles in peer-review: assessors and assessees. During the peer review process,
assessors and assessees play the different roles of providing and receiving feedback. As assessors, they can provide their own ideas and
insights, while assessees can decide whether to accept their peers’ opinions and accordingly revise their works (Li, Liu, & Steckelberg,
2010). In accordance with social constructivism theory, knowledge is generated through the interaction created by social interaction.
When learners interact with each other as well as with the external world, they will show a higher level of psychological functions
(Vygotsky, Cole, John-Steiner, Scribner, & Souberman, 1978).
Many studies have verified the learning effectiveness of peer-review, for example, enhancing learners’ learning performance,
critical thinking tendency, self-efficacy and metacognition (Chang et al., 2020; Zheng, Chen, Li, & Huang, 2016), effectively helping
learners increase their problem-solving ability with the use of the peer-review learning model (Cevik, Haşlaman, & Çelik, 2015;
Elshami & Abdalla, 2017; Çevik, 2015). Researchers have pointed out that participating in peer review is helpful for students to
improve their communication skills (Lai, 2016; Solheim et al., 2017). Peer review allows students to learn from each other, promotes
their reflection and critical thinking, and improves their own performance (Solheim et al., 2017; Tornwall, 2018).
In professional training programs, such as nursing courses, learners need to face diverse problems as well as rapidly changing
knowledge; therefore, it is important to foster their higher order thinking competences, such as critical thinking and reflective
thinking, to make judgements and reflections when facing new knowledge and problems (Mann, Gordon, & MacLeod, 2009). Several
researchers have denoted that engaging nursing students in making reflections is an important way to help them gain knowledge and
learn to make decisions, which can further promote their confidence in solving problems (Chen, Chen, & Pai, 2019; Hayes, Jackson,
Davidson, Daly, & Power, 2018; O’Brien & Graham, 2020) and enables them to provide quality care for patients (Bellman, 1996;
Robinson, 2015). This implies that peer assessment is a potential strategy for promoting learners’ higher order thinking competences in
3
nursing training (Marrocco, Ginzburg, & Feder, 2019; Persson, Kvist, & Ekelin, 2015; Willemse, 2015).
Several researchers have further denoted that in the teaching of nursing skills, including skilled subcutaneous injection, intra
muscular injection, intravenous injection, tracheostomy care and tracheostomy suction, adopting peer review as the teaching strategy
can facilitate students’ autonomous learning and increase their skill proficiency and learning satisfaction (Delgado & Mack, 2002;
McAllister & Osborne, 1997; O’Brien, Talbot, & Santevecchi, 2015). For example, Sedlak and Doheny (1998) conducted a peer-review
activity in a clinical training course, and found that the approach promoted nursing students’ interactive quality; they tended to
carefully consider peers’ opinions, raise in-depth questions, clarify their points, and re-examine their nursing skill practice, which
improved their critical thinking and organizational skills. Himes and Ravert (2012) also used peer assessment in a nursing skills
training course, and found that the nursing students not only showed better patient care skills, but also had better performances in
communication, critical thinking, and clinical decision-making. Later, Lin and Shen (2013) conducted a peer-review activity to engage
nursing students in examining peers’ clinical work diaries to promote their collaboration, reflection and peer communications.
Papastavrou, Hamari, Fuster, Istomina, and Salminen (2016) designed a peer-review activity in which nursing students were asked to
review peers’ nursing journals written on social media. The experimental results showed that the approach facilitated the students’
reflections, knowledge acquisition, and critical thinking.
To sum up, in conventional peer review, assessors provide one-way ratings and comments, while assessees have no chance to
express their points of view on the feedback. For assessees, whether they are satisfied with the content of the comments and whether
the content is detailed and reasonable will arouse their in-depth thinking; they will decide whether to accept suggestions and then take
actions to change their ideas and behaviors (Huisman, Saab, Van Driel, & Van Den Broek, 2018; Zhou, Zheng, & Tai, 2020). On the
other hand, regarding assessors, the more specific and constructive the comments are and the higher the quality of their comments, the
better performance they will have (Li et al., 2010; Çevik, 2015). Through assessees’ responses, assessors can understand whether they
have offered constructive opinions; they are likely to be more confident in their acquired knowledge or to think about whether to revise
their original thoughts and judgments. Compared with assessees, assessors have better problem-solving ability through reasonable
reasoning (Çevik, 2015).
3. System framework
In the present study, an online interactive peer-review system was developed and applied to the nursing flipped learning activities.
The learning environment is illustrated in Fig. 1; it consists of a flipped learning system and an interactive peer-review system as well as
a learning management system for the teacher to manage learning materials, learning tasks, and preview-review tasks. In addition, four
databases were established to work with the systems: (1) a learning material database for keeping the instructional video data and
learning sheets; (2) a learner profile database for keeping students’ personal information and user accounts; (3) a learning portfolio
database for keeping students’ learning records and test scores; and (4) a peer-review database for keeping the students’ videos as well
as peer-review and responding records. The teacher could assign learning tasks or review tasks to individual students by referring to
their records in the learning portfolio database. The systems were developed using PHP and JavaScript with MySql. Also, the video
streaming function in this system was embedded in PHP web pages through Google YouTube API.
The flipped learning system is an online learning platform consisting of a learning module and a discussion forum. The learning
module is mainly used to enable students to watch the instructional videos and take notes in the pre-class stage of flipped learning, as
shown in Fig. 2. All the students’ annotations and the corresponding timestamps are recorded by the system. Students can review their
annotations at a later time. The discussion forum enables students to discuss with peers during the learning process.
The interactive peer-review system consists of a peer-review module and a responding module. During the peer-review process,
students are asked to upload their videos of health assessment skills to the peer-review system. The teacher then assigns the peer-
Fig. 1. The framework of the nursing health assessment flipped teaching system based on online interactive peer review.
4
Fig. 2. The interface of the learning module for the pre-class stage.
review tasks to individual students through the management systems. The pre-review module enables students to conduct online
reviews after they log in with a personal account and password. As shown in Fig. 3, the peer-review interface enables students to play
peers’ videos of health assessment skills and provide ratings and comments to peers based on the rubrics provided by the teacher.
The responding module enables students to make responses to the assessors after receiving ratings and comments from peers, as
shown in Fig. 4. Via this module, students can not only see the ratings and comments given by their peers, but can also respond to
assessors’ ratings by rating whether the assessors provided reasonable ratings (1 = very unreasonable; 4 = very reasonable) as well as
responding to the assessors’ comments offered via a dialog window. On the other hand, the students can also see peers’ responses to the
ratings and comments they provided to peers’ videos.
4. Research design
In an effort to evaluate the effectiveness of the online interactive peer-review flipped learning approach, a mixed research design
Fig. 3. The interface of the peer-review module.
5
Fig. 4. The interface of the responding module.
was adopted by conducting an experiment in a hospital to examine the influences of the two online peer-review approaches on stu
dents’ learning achievement, health assessment skill test, critical thinking, self-efficacy, reflective thinking, and the quality of peers’
comments. The study selected the health assessment training course for nurse practitioner students; the purpose of this course was to
develop nurse practitioner students’ health assessment capability. The study was approved by The Institutional Review Board (IRB) of
Fig. 5. The experiment procedure.
6
the hospital (approval number: 1-108-05-160).
4.1. Experiment procedure
The experimental procedure is shown in Fig. 5. Differing from traditional training seminars in that hospital, flipped learning was
implemented in the training program in this study; that is, all of the trainees were informed to learn via instructional videos and
complete learning sheets in the before-class stage, such that the instructor had more time to guide them to practice and apply
knowledge in the seminar.
The learning content included respiratory system assessment, cardiovascular system assessment, abdominal system assessment,
introduction of the Objective Structured Clinical Examination (OSCE) health assessment skills, and demonstration of the OSCE health
assessment skills. A total of five instructional videos, of 5–15 min each, were placed on the teaching platform. One month before the
class, students were notified by email; they started to log into the platform for study, and completed watching the videos and uploading
the worksheets 1 week before the class. All of the students participated in watching the pre-class learning videos and in completing the
flipped learning worksheets. Before the class, through the online platform, teachers could check the period of the learning time of each
student, the annotations of the online teaching materials, and the students’ worksheets, so as to understand the individual learning
status of each student.
The teacher conducted 2 days of classroom teaching in the form of a seminar. Before the first day of the course, the pre-learning test
was administered to evaluate the students’ basic knowledge of health assessment. Also, students completed the questionnaires of
critical thinking tendency, reflective thinking, and self-efficacy. During class, the students were divided into four groups, with a coach
assisting and guiding each group. The students and simulated patients conducted physical examination skills practice while other
students evaluated them using the skills checklist. Afterwards, the pre-test of the OSCE for health assessment was conducted; the test
process was filmed as the online peer-review video for this experiment. In class, the students strengthened their skill practice of
physical examination and clinical case discussion while the teacher introduced the peer-review system and rubrics.
After the first day of the course, the students were required to complete the online peer review within 1 week. At the beginning of
the peer-review activity, the teacher assigned the videos recording individual students’ OSCE process for health assessment. Each
student was assigned to review three videos. Likewise, each student would receive three ratings and comments from peers.
During the peer-review process, the experimental group adopted the online interactive peer-review approach. After receiving
comments and ratings from peers, assessees were asked to give responses to the comments and ratings, including the level of agreement
or disagreement as well as the corresponding descriptions regarding why they gave such responses to their peers. After receiving the
responses from the assessees, the assessors could determine whether to modify the ratings or comments. It should be noted that the
interactive mechanism was not used to argue who was right or wrong, but rather to give both assessors and assessees the opportunity to
make reflections.
On the other hand, the control group adopted the conventional peer-review approach. Assessees were able to see the ratings and
comments provided by their peers on the online system; however, they had no chance to respond to the ratings and comments they
received or to make clarifications.
It should be noted that the students’ ratings and comments on peers’ videos were used only for them to make reflections and would
not affect their final OSCE scores, as suggested by several previous studies (Han & Chan, 2020; Tricio, Woolford, & Escudier, 2016). To
evaluate the students’ learning performances, two experienced teachers were recruited to evaluate the students’ OSCE videos.
In the 2-day course, two clinical cases, “health assessment of the cardiovascular system” and “health assessment of the abdominal
system,” were discussed. The students were divided into groups, practiced history taking, interviewed the simulated patients, and
discussed possible differential diagnoses based on the collected health assessment. This allowed the students to learn in-depth thinking
and comprehensive analysis skills. Following that, they were asked to complete the post-learning test, the post-OSCE for health
assessment and the post-questionnaires in the second course. After the course, five students from each group were randomly chosen for
the interviews to further collect the qualitative results. Regarding the students’ peer review, the accuracy of the ratings and the content
of the comments were analyzed at the same time.
4.2. Participants
A priori power analysis was performed to estimate the sample size. A median-large effect size has been reported. Therefore, we
chose to use an effect size of f = 0.35 by referring to the suggestions of previous flipped learning studies (Strelan et al., 2020a). With an
alpha = .05 and power = .80, the total sample size was N = 67, as suggested by Faul, Erdfelder, Lang, and Buchner (2007). A total of 81
nurse practitioners registered for the course, 79 of whom signed the consent form to participate in this study. After removing three
students who did not fully take part in all the courses and two students who did not fill out the questionnaires, a total of 74 nurse
practitioners were recruited in the present study. The experimental group (n = 33) adopted the online interactive peer-review flipped
learning approach for learning, while the control group (n = 41) adopted the conventional peer-review flipped learning approach for
learning. Both of the groups were taught by the same professional teacher who had taught the health assessment course for more than 5
years.
4.3. Instruments
The instruments of the current study consisted of the pre- and post-learning achievement test, the pre- and post-OSCE, the critical
7
thinking tendency questionnaire, the reflective thinking scale, the self-efficacy questionnaire, the rubrics for peer review, the coding
for content analysis of peer review, and the interviews.
(1) Learning achievement tests
The content of the learning achievement tests included such relevant knowledge as physical assessment and differential diagnosis.
The tests consisted of the pre- and post-test. The pre-learning achievement test aimed to evaluate whether the two groups of students
possessed similar basic knowledge of health assessment; the test consisted of 20 multiple-choice questions with a perfect score of 100.
The post-learning achievement test was to assess students’ abilities of health assessment and case study analysis; the test consisted of
20 multiple-choice questions (4 points for each) and four short-answer questions (5 points for each) with a perfect score of 100. The
two tests were collaboratively set by two nursing clinical teachers with more than 5 years of teaching experience. The KR20 values of
the pre-test and post-test were 0.78 and 0.73, respectively.
(2) Health assessment skill tests
In order to understand students’ learning effectiveness of health assessment skills, medical education adopts the Objective
Structured Clinical Examination (OSCE) as the evaluation method. The OSCE was proposed by Harden in 1975; it aims to train medical
students’ clinical capabilities such as physical examination and assessment, and to simulate a clinically similar situation for students to
conduct various clinical techniques and tasks (Harden, Stevenson, Downie, & Wilson, 1975). The teacher employed objective and valid
rubrics to evaluate students’ accuracy and completeness of carrying out clinical practice through direct observation, and provided
them with timely feedback. Studies verify that OSCE is an appropriate training and assessment tool to understand the learning
effectiveness of clinical skills implemented by healthcare professionals (Deng, Fenn III, & Plake, 2019; Prince et al., 2017; Slomer &
Chenkin, 2017).
The health assessment skill tests included the pre- and post-OSCE in this study. The rubrics of the health assessment clinical skills
OSCE were developed by two nurses with more than 5 years of teaching experience in health assessment courses; the tests were
approved by the Clinical Skill Center. There were six OSCE examination rooms, including six standardized patients and six examiners.
The six standardized patients were professionals who are trained and who obtained their licenses from the Clinical Skill Center. Before
the test, they were trained for 1 h based on the content of the test questions and contexts in order to respond to the examinees’
questions with the standardized answers. All of the six examiners had more than 5 years of experience as examiners of health
assessment skill tests. Before the examination, based on the examiners’ guidelines, they reached a consensus on the evaluation criteria
of students’ performance from “fully complete” to “not complete” so as to achieve standardized scores. The test lasted for 15 min. Both
the experimental and control group adopted the same situational questions, rubrics, standardized patients, and examiners. In the pre-
OSCE, students were required to carry out medical history taking, and perform a physical examination on standardized patients who
complained of shortness of breath. The students had to synthesize the collected medical history and physical examination data,
analyze, reason, and further articulate the speculative clinical diagnosis. Examiners assessed students’ performance based on the
rubrics. As for the post-OSCE, another set of abdominal pain lesson plans was adopted.
The rubrics of OSCE skills included history taking, focused physical examination and differential diagnosis, and clinical reasoning.
The items of history taking were, for example, “whether to ask the location of abdominal pain” and “whether to ask the start time of
abdominal pain.” The items of physical examination were, for instance, “whether to perform auscultation accurately” and “whether to
perform palpation accurately.” The items of differential diagnosis were, for example, “whether to accurately identify that the patient’s
possible health problem is acute appendicitis” and “whether to say the reasons why the patient’s possible health problem is acute
appendicitis.” The total score of OSCE is 100 with history taking comprising 20 points, focused physical examination, 40 points, and
differential diagnosis and clinical reasoning, 40 points. The KR20 value of the OSCE test was 0.73.
(3) Critical thinking tendency questionnaire
The critical thinking tendency questionnaire was adapted from Chai, Deng, Tsai, Koh, and Tsai (2015) and consisted of six items, for
example, “In this learning activity, I think about whether the content I learn is correct or not” and “In this learning activity, I try to
understand the new knowledge from different perspectives.” This questionnaire adopted a 5-point Likert scale from 1 = strongly
disagree to 5 = strongly agree. The Cronbach’s alpha of this questionnaire in the original study was 0.71.
(4) Self-efficacy questionnaire
The self-efficacy questionnaire was adapted from Wang & Hwang (2012) and was composed of eight items, for instance, “I believe I
can understand the most difficult part presented in this class.” This questionnaire adopted a 5-point Likert scale from 1 = strongly
disagree to 5 = strongly agree. The Cronbach’s alpha of this questionnaire in the original study was 0.92.
(5) Reflective thinking scale
The reflective thinking scale was adapted from Kember et al. (2000) and consisted of the four dimensions of habitual action,
understanding, reflection and critical reflection. Each dimension consisted of four items with a total of 16 items. This questionnaire
8
adopted a 5-point Likert scale from 1 = strongly disagree to 5 = strongly agree. The Cronbach’s alpha values of habitual action, un
derstanding, reflection, and critical reflection in the original study were 0.62, 0.76, 0.63, and 0.68, respectively.
(6) Peer-review rubrics
The peer-review rubrics were adapted from Mavis et al. (2013) and Hettinga, Denessen, and Postma (2010) with the five di
mensions of history taking, physical examination, differential diagnosis and reasoning process, communication skills, and time
management. Based on students’ performance on health assessment skills in the videos, each dimension was given 1 to 4 points for
implementation accuracy and completeness. The peer-review rubrics are illustrated in Table 1.
(7) The coding for content analysis of peer review
The coding list for content analysis for peer review in this study was taken from Cheng, Liang, and Tsai (2015), and included
affective, cognitive, metacognitive, and irrelevant comments. Affective comments consisted of two categories: (A1) supporting and
(A2) opposing. Comments with compliments or supporting opinions such as “smooth history taking” and “great performance” were
categorized into A1. Negative comments, for example, “anxious performance” and “unstable performance,” were categorized into A2.
Cognitive comments consisted of three categories: (C1) direct correction, (C2) personal opinions, and (C3) guidance. The content of the
peer review focused on information about the accuracy of students’ performance; for instance, “When performing abdominal
palpation, the patient must have his/her feet bent” and “When performing a respiratory auscultation, a breath sound must be
completely heard,” were categorized into C1. General opinions without specific direction of revision, for example, “The content of
history taking seems to be less” and “incorrect location of breath auscultation,” were categorized into C2. Comments and feedback with
specific suggestions and guidance for revision were categorized into C3, for instance, “The location for auscultation of tricuspid area is
sternum low 1/3, not Erb” and “During medical history taking, it is confusing when inquiring in the system review; it is suggested to
perform history taking from head to toes or by systems.”
The metacognitive aspect comprised two categories: (M1) evaluating and (M2) reflecting. M1 referred to comments which verified
knowledge, skills, or selected strategies such as “Regarding the diagnosis of pulmonary embolism, the reason process should take the
collected medical history such as obesity, long-term flight and varicose veins, the patient’s occupation and a history of varicose veins
into consideration. Also, the reasoning process should be logical.” Comments that triggered students to reflect and think were cate
gorized into M2, such as “For crackles (rales), jugular vein distention, early diastolic, S3 and S4 in the breath sound, what do you think
are the main possible causes of these symptoms? It is suggested that you further think about the possible diagnosis.” Besides, comments
irrelevant to affective, cognitive and metacognitive comments were all categorized into IR. The definition of content analysis coding is
depicted in Table 2.
(8) Interviews
The interview questions referred to Hwang, Yang, Tsai and Yang (2009), and comprised a total of seven questions. They aimed to
Table 1
Peer-review rubrics.
Dimension/Rating 1 2 3 4
History taking Most of the important medical The medical history content is 1-3 errors; the medical history Accurately performed medical
history is missing. not complete enough; it lacks content is still accurate and history taking; the data is
more than 4–5 important complete. complete and accurate
data.
Physical Most of the physical examination Some physical examination Most of the physical Performed all physical
examination items are performed incorrectly or items are performed examination items are examination items (inspection,
incompletely. incorrectly or incompletely; performed correctly and palpation, percussion,
4–5 items are missing or almost completely, but 1–3 auscultation) correctly and
incorrect. items are missing or incorrect. completely
Differential Wrong diagnosis or many missing Failed to state the possible Correctly stated the possible Correctly and completely
diagnosis and items in the reasoning process diagnosis completely; 4–5 diagnosis; 1–3 items are stated the possible diagnosis
reasoning items are missing in the missing in the reasoning and the reasoning process
reasoning process but it is still process, but it is generally
complete. complete.
Communication The communication process with 4-5 pauses and unclear 1-3 pauses and unclear Professional performance;
skills the standardized patient pauses communication; ignored the communication; ignored the smooth communication with
several times and is difficult to patient’s privacy patient’s privacy standardized patients, and paid
understand; does not pay attention to the patient’s
attention to the patient’s privacy privacy and feelings.
Time management More than 4 (inclusive) items are 1-3 items are not completed Completed all the items within Time management is accurate
not completed within the time. within the time. the time; there are still more and all the items are completed
than 3 min left, which is not within the time.
fully utilized.
9
Table 2
The coding list for content analysis of peer-review.
Dimension Category Definition Example
Affective Supporting Comments with compliment or supporting opinions Smooth and great history taking
(A1)
Opposing (A2) Negative comments on students’ performance Bad and nervous performance
Cognitive Direct Focusing on the accuracy of students’ performance When performing abdominal palpation, the patient must have his/
correction (C1) in the videos (e.g., physical examination and her feet bent.
communication skills)
Personal General suggestions or opinions without specific The content of history taking seems to be less.
opinion (C2) direction of revision.
Guidance (C3) Specific suggestions, concepts or methods for the The location for auscultation of tricuspid area is sternum low 1/3.
revision of students’ performance
Metacognitive Evaluating Comments of verifying knowledge, skills or Regarding the diagnosis of pulmonary embolism, the reason
(M1) strategies process should take the collected medical history such as obesity,
long-term flight and varicose veins, the patient’s occupation and a
history of varicose veins into consideration. The reasoning process
is unclear and not logical.
Reflecting Comments that question the students’ performance For crackles (rales), jugular vein distention, early diastolic, S3 and
(M2) in the video and trigger students to reflect and think S4 in the breath sound, it is suggested that you further think about
deeply the possible health problems and diagnosis.
Irrelevant Irrelevant (IR) Irrelevant comments to affective, cognitive and
comments metacognitive comment
examine the advantages and disadvantages of online peer review from the students’ perspectives. The interview results were expected
to provide extra evidence to account for the findings in the present study and to serve as a reference for improvement in the future.
5. Experimental results
A two-way analysis of variance (ANOVA) with repeated measures was conducted to evaluate the effect of the different online peer-
review approaches and measurements across time on the dependent variables (learning achievement, health assessment skill tests,
critical thinking tendency, reflective thinking, and self-efficacy). Group is a between-subjects factor, whereas test time is a within-
subjects factor. Group included two levels (control, experimental), and test time consisted of two levels (pre-test, post-test).
5.1. Learning achievement
The descriptive statistics of learning achievement are shown in Table 3. Two-way analysis of variance (ANOVA) with repeated
measures was conducted in order to evaluate the effect of group and test time on learning achievement. The results are shown in
Table 4; group and test time have a significant interaction effect (F = 12.09, p = .001, ηp 2 = 0.144). According to the results of the
interaction effect (Fig. 6), the learning achievement score of the control group (M = 77.85, SD = 10.57) was higher than that of the
experimental group (M = 72.00, SD = 11.27) in the pre-test. For the post-test, the learning achievement score of the experimental
group (M = 72.45, SD = 9.08) was higher than that of the control group (M = 67.78, SD = 10.92).
5.2. Health assessment skill tests
The descriptive statistics of the health assessment skill tests including OSCE total scores, history taking, physical examination and
diagnosis, and reasoning, are shown in Table 5. Two-way analysis of variance (ANOVA) with repeated measures was conducted in
order to evaluate the effect of group and test time on the OSCE total scores. Results are shown in Table 6; group and test time had a
significant interaction effect (F = 5.47, p = .022, ηp 2 = 0.071). According to the results of the interaction effect (Fig. 7), the OSCE total
scores of the control group (M = 59.12, SD = 17.18) were higher than those of the experimental group (M = 55.37, SD = 9.81) in the
pre-test. For the post-test, the OSCE total scores of the experimental group (M = 74.87, SD = 9.42) were higher than those of the control
group (M = 70.43, SD = 16.05). Therefore, results suggest that students from the experimental group improved their health assessment
skills over time. Compared with students in the control group, students in the experimental group made progress in OSCE.
Table 3
Descriptive statistics for the means of learning achievement.
Group Test time Mean (SD) Std. Error 95% Confidence Interval
Lower Bound Upper Bound
Control 1 77.85 (10.57) 1.70 74.46 81.24

2 67.78 (10.92) 1.58 64.62 70.94
Experimental 1 72.00 (11.27) 1.90 68.22 75.78
2 72.45 (9.08) 1.77 68.94 75.97
10
Table 4
Repeated measures ANOVA results of learning achievement.
Source SS df MS F p η2p
group (A) 12.72 1 12.72 0.92 .762 .001
Error 15,938.33 144 110.68
test time (B) 845.79 1 845.79 10.10 .002 .123
group * test time 1013.22 1 1013.22 12.09 .001 .144
Error 6032.48 72 83.78
Fig. 6. Interaction of group and test time for learning achievement.
Table 5
Descriptive statistics for the means of health assessment skills.
Group Test times Mean (SD) Std. Error 95% Confidence Interval
TC Control 1 59.12 (17.18) 2.25 54.65 63.60

2 70.43 (16.05) 2.11 66.22 74.63
Experimental 1 55.37 (9.81) 2.50 50.38 60.36
2 74.87 (9.42) 2.35 70.18 79.56
HT Control 1 14.90 (3.34) 0.51 13.89 15.91
2 16.24 (2.63) 0.35 15.54 16.95
Experimental 1 15.18 (3.13) 0.57 14.05 16.31
2 16.47 (1.69) 0.39 15.69 17.25
PE Control 1 33.13 (4.74) 0.65 31.83 34.43
2 33.56 (3.32) 0.47 32.63 34.49
Experimental 1 31.88 (3.34) 0.73 30.43 33.33
2 34.94 (2.48) 0.52 33.91 35.97
DR Control 1 12.32 (9.0) 1.29 9.76 14.88
2 20.62 (11.24) 1.56 17.52 23.72
Experimental 1 8.31 (7.15) 1.43 5.46 11.17
2 23.46 (8.09) 1.73 20.00 26.92
Note. TC = total score, HT = history taking, PE = physical examination, DR = Diagnosis and Reasoning.
Table 6
Repeated measures ANOVA results of OSCE.
Source SS df MS F p ηp2
group (A) 4.36 1 4.36 0.02 .901 .000
Error 28,034.31 144 194.68
OSCE score (B) 8673.41 1 8673.41 77.37 .000 .518
group * OSCE score 613.52 1 613.52 5.47 .022 .071
Error 8071.66 72 112.11
11
Fig. 7. Interaction of group and test time for the OSCE.
Two-way analysis of variance (ANOVA) with repeated measures was conducted in order to evaluate the effect of group and test time
on history taking (Table 7). An interaction effect between group and test time could not be demonstrated (F = 0.01, p = .94, ηp 2 =
0.000). As the results revealed, there is no statistically significant difference between the experimental group and the control group on
history taking.
on physical examination. The results are shown in Table 8; group and test time had a significant interaction effect (F = 5.41, p = .023,
ηp 2 = 0.070). According to the results of the interaction effect (Fig. 8), the physical examination scores of the control group (M =
33.13, SD = 4.74) were higher than those of the experimental group (M = 31.88, SD = 3.34) in the pre-test. For the post-test, the
physical examination scores of the experimental group (M = 34.94, SD = 2.48) were higher than those of the control group (M = 33.56,
SD = 3.32). Therefore, results suggest that students from the experimental group improved their physical examination skill over time.
Compared with students in the control group, students in the experimental group had an increase in physical examination.
on diagnosis and reasoning. The results are shown in Table 9; group and test time have a significant interaction effect (F = 8.45, p =
.005, ηp 2 = 0.105). According to the results of the interaction effect (Fig. 9), the diagnosis and reasoning scores of the control group (M
= 12.32, SD = 9.0) were higher than those of the experimental group (M = 8.31, SD = 7.15) in the pre-test. For the post-test, the
diagnosis and reasoning scores of the experimental group (M = 23.46, SD = 8.09) were higher than those of the control group (M =
20.62, SD = 11.24). Therefore, results suggest that students from the experimental group improved their diagnosis and reasoning over
time. Compared with students in the control group, students in the experimental group had greater progress in diagnosis and
reasoning.
5.3. Critical thinking tendency
The descriptive statistics of the critical thinking tendency scores are shown in Table 10. Two-way analysis of variance (ANOVA)
with repeated measures was conducted in order to evaluate the effect of group and test time on the OSCE total scores. The results are
shown in Table 11; group and test time had a significant interaction effect (F = 4.77, p = .032, ηp 2 = 0.062). In the test time condition
(Table 16), the simple main effects of the experimental group were significant (F = 12.52, p = .001). For the experimental group, there
was a statistically significant difference between test time (pre-test and post-test). According to the results of the interaction effect
(Fig. 10), the critical thinking tendency scores of the control group (M = 4.25, SD = 0.52) were higher than those of the experimental
group (M = 4.15, SD = 0.44) in the pre-test. For the post-test, the critical thinking tendency scores of the experimental group (M =
Table 7
Repeated measures ANOVA results of history taking.
group (A) 2.33 1 2.33 0.22 .642 .003
Error 1128.55 144 7.84
OSCE- HT (B) 63.20 1 63.20 12.79 .001 .151
group * OSCE-HT 0.03 1 0.026 0.01 .942 .000
Error 355.867 72 4.943
Note. HT = history taking.
12
Table 8
Repeated measures ANOVA results of physical examination.
group (A) 0.14 1 0.14 0.01 .923 .000
Error 1893.50 144 13.15
OSCE- PE (B) 111.19 1 111.19 9.48 .003 .116
group * OSCE- PE 63.42 1 63.42 5.41 .023 .070
Error 844.33 72 11.73
Note. PE = physical examination.
Fig. 8. Interaction of group and test time for OSCE-PE.
Table 9
Repeated measures ANOVA results of diagnosis and reasoning.
Source SS Df MS F p η2p
group (A) 12.50 1 12.50 0.11 .744 .001
Error 1020.44 144 83.48
OSCE-DR (B) 5028.59 1 5028.59 99.26 .000 .580
group * OSCE-DR 428.16 1 428.16 8.45 .005 .105
Error 3647.57 72 50.66
Note. DR = Diagnosis and Reasoning.
4.44, SD = 0.49) were higher than those of the control group (M = 4.30, SD = 0.49). Therefore, results suggest that students from the
experimental group improved their critical thinking tendency over time. Compared with students in the control group, students in the
experimental group had a great increase in critical thinking tendency scores.
5.4. Reflective thinking
Descriptive statistics of reflective thinking included habitual action, understanding, reflection, and critical reflection as well as the
total reflective thinking score, as shown in Table 12. Two-way analysis of variance (ANOVA) with repeated measures was conducted in
order to evaluate the effects of group and test time on reflective thinking. The results are shown in Table 13; group and test time had a
significant interaction effect (F = 4.32, p = .041, ηp 2 = 0.057). According to the results of the interaction effect (Fig. 11), the reflective
thinking of the control group (M = 4.23, SD = 0.46) was the same as that of the experimental group (M = 4.23, SD = 0.47) in the pre-
test. For the post-test, the reflective thinking of the experimental group (M = 4.40, SD = 0.46) was higher than that of the control group
(M = 4.16, SD = 0.45). Therefore, results suggest that students from the experimental group improved their reflective thinking over
time. Compared with the students in the control group, the students in the experimental group had a great increase in their reflective
13
Fig. 9. Interaction of groups and test times for OSCE-DR.
Table 10
Descriptive statistics for the means of critical thinking tendency.
95% Confidence Interval
Group Test time Mean (SD) Std. Error Lower Bound Upper Bound
Control 1 4.25 (0.52) 0.08 4.10 4.40

2 4.30 (0.49) 0.07 4.16 4.45
Experimental 1 4.15 (0.44) 0.09 3.98 4.32
2 4.44 (0.49) 0.08 4.29 4.60
Table 11
Repeated measures ANOVA results of critical thinking tendency.
group (A) 0.01 1 0.01 0.04 .851 .000
Error 31.96 144 0.22
CT (B) 1.12 1 1.12 9.56 .003 .117
group * CT 0.56 1 0.56 4.77 .032 .062
Error 8.40 72 0.12
Note. CT = Critical thinking tendency.
thinking.
Regarding the four dimensions of reflective thinking (i.e., habitual action, understanding, reflection, and critical reflection), there
was no interaction effect between group and test time for habitual action, understanding, or critical thinking. That is, there was no
statistically significant difference between the experimental group and the control group on the dimensions of habitual action, un
derstanding, or critical thinking.
On the other hand, a significant interaction effect was found between group and test time for the dimension of reflection (F = 7.70,
p = .007, ηp 2 = 0.097) (Table 14). According to the results of the interaction effect (Fig. 12), the reflection of the control group (M =
4.37, SD = 0.46) was higher than that of the experimental group (M = 4.22, SD = 0.46) in the pre-test. For the post-test, the reflection
of the experimental group (M = 4.43, SD = 0.47) was higher than that of the control group (M = 4.21, SD = 0.50). Therefore, results
suggest that students from the experimental group improved their reflection over time. Compared with the students in the control
group, students in the experimental group had a great increase in reflection.
5.5. Self-efficacy
The descriptive statistics of self-efficacy are shown in Table 15. The self-efficacy post-test score (M = 4.02, SD = 0.62) of the control
group was higher than their pre-test score (M = 4.01, SD = 0.57). Similarly, the self-efficacy post-test score (M = 4.14, SD = 0.56) of the
experimental group was higher than their pre-test score (M = 4.10, SD = 0.53). However, the interaction effect between group and test
time was not significant (F = 0.06, p = .811, ηp 2 = 0.001) (Table 16).
14
Fig. 10. Interaction of group and test time for critical thinking tendency.
Table 12
Descriptive statistics for the means of reflective thinking.
RT Control 1 4.23 (0.46) 0.07 4.09 4.38

2 4.16 (0.45) 0.07 4.02 4.30
Experimental 1 4.23 (0.47) 0.08 4.07 4.39
2 4.40 (0.46) 0.08 4.24 4.55
HAS Control 1 4.04 (0.51) 0.09 3.86 4.21
2 4.10 (0.52) 0.09 3.93 4.28
Experimental 1 4.24 (0.61) 0.10 4.04 4.43
2 4.46 (0.64) 0.10 4.26 4.65
US Control 1 4.41 (0.59) 0.09 4.23 4.59
2 4.17 (0.53) 0.08 4.01 4.33
Experimental 1 4.39 (0.56) 0.10 4.19 4.59
2 4.35 (0.51) 0.09 4.17 4.53
RS Control 1 4.37 (0.46) 0.07 4.22 4.51
2 4.21 (0.50) 0.08 4.06 4.37
Experimental 1 4.22 (0.46) 0.08 4.06 4.38
2 4.43 (0.47) 0.09 4.26 4.60
CRS Control 1 4.12 (0.66) 0.10 3.93 4.31
2 4.15 (0.49) 0.08 4.00 4.31
Experimental 1 4.09 (0.54) 0.11 3.88 4.30
2 4.40 (0.52) 0.09 4.23 4.58
Note. RT = total reflective thinking score, HAS = habitual action sub-scale, US = understanding sub-scale, RS = reflection sub-scale, CRS = Critical
reflection sub-scale.
Table 13
Repeated measures ANOVA results of reflective thinking.
group (A) 0.51 1 0.51 1.67 .201 .023
Error 30.44 144 0.21
reflective thinking (B) 0.074 1 0.07 0.62 .432 .009
group * reflective thinking 0.509 1 0.51 4.32 .041 .057
Error 8.50 72 0.12
15
Fig. 11. Interaction of group and test time for reflective thinking.
Table 14
Repeated measures ANOVA results of reflection.
group (A) 0.05 1 0.05 0.17 .686 .002
Error 32.21 144 0.22
RR (B) 0.03 1 0.03 0.21 .651 .003
group * RR 1.22 1 1.22 7.70 .007 .097
Error 11.38 72 0.16
Note. RR = Reflective thinking-reflection sub-scale.
Fig. 12. Interaction of group and test time for reflection.
5.6. Correlations between learning achievement, health assessment skill tests, self-efficacy, critical thinking tendency, and reflective thinking
Table 17 shows the results of the correlations between health assessment knowledge, skill tests, self-efficacy, critical thinking, and
reflective thinking. The results show a significant correlation between the health assessment knowledge test and reflective thinking (r
= 0.424, p < .05), with a moderate positive correlation (Cohen, 1988). This means that students with higher levels of reflective
thinking are likely to have better learning outcomes. This result is consistent with the research of Rotz and O’Neill (2020) who
indicated that enhancing students’ reflective thinking was helpful to their learning outcomes. The results also reveal a significant
16
Table 15
Descriptive statistics for the means of self-efficacy.
Control 1 4.01 (0.57) 0.09 3.84 4.18

2 4.02 (0.62) 0.09 3.83 4.20
Experimental 1 4.10 (0.53) 0.10 3.90 4.29
2 4.14 (0.56) 0.10 3.94 4.34
Table 16
Repeated measures ANOVA results of self-efficacy.
group (A) 0.42 1 0.42 0.95 .333 .013
Error 46.62 144 0.29
SE (B) 0.03 1 0.03 0.13 .719 .002
group * SE 0.01 1 0.01 0.06 .811 .001
Error 14.99 72 0.21
Note. SE = self-efficacy.
Table 17
Correlation analysis between learning achievement, health assessment skill tests, self-efficacy, critical thinking tendency, and reflective thinking.
Learning achievement Health assessment skill tests Self-efficacy Critical thinking tendency Reflective thinking
Learning achievement 1 .027 -.018 .097 .424a

Health assessment skill tests .027 1 .180 -.058 .068
Self-efficacy -.018 .180 1 .350a .329
Critical thinking tendency .097 -.058 .350a 1 .511**
Reflective thinking .424a .068 .329 .511** 1
a p
<.05, **p<.01.
Table 18
Correlation between peer-review ratings and the experts’ ratings.
Group Rater Mean SD r
Experimental (n = 33) Peer-review 2.64 0.41 .60a

Experts 2.36 0.45
Control (n = 41) Peer-review 3.03 0.42 .50a
Experts 2.65 0.54
a p
<.01.
Table 19
The number of affective, cognitive, and metacognitive comments in the peer review.
Dimension Experimental (n = 33) Control (n = 41)
Affective Supporting (A1) 45 5.26% 49 5.73%

Opposing (A2) 11 1.29% 13 1.52%
Cognitive Direct correction (C1) 111 12.98% 61 7.13%
Personal opinion (C2) 214 25.03% 178 20.82%
Guidance (C3) 101 11.81% 56 6.55%
Metacognitive Evaluating (M1) 7 0.82% 3 0.35%
Reflecting (M2) 3 0.35% 0 0.00%
Irrelevant comments (IR) 0 0.00% 3 0.35%
Total 492 57.54% 363 42.46%
17
correlation between reflective thinking and critical thinking (r = 0.511, p < .01), with a large positive correlation (Cohen, 1988). This
finding is consistent with the research of Cheng, Huang, Yang, and Chang (2020) and Chen, Chang, and Pai (2018), who indicated that
self-reflective thinking and critical thinking generally influence each other.
5.7. Peers’ and the teacher’s ratings
Table 18 illustrates the health assessment skill scores rated by the experimental group, the control group, and the experts. The
results indicated that the peer-review scores of the experimental group and control group were positively correlated with the experts’
scores with r = .60 (p < .01, r2 = 0.36) and r = 0.50 (p < .01, r2 = 0.28), respectively.
5.8. Quality of peer-review content
Table 19 lists the total number of peer-review codings as well as the number and percentage of comments in each dimension in this
study. The total number of peer-review comments in the experimental group (492) was more than that of the control group (363).
Among all the dimensions in the comments, cognitive comments accounted for the majority (84.33%), followed by affective comments
(13.80%), and metacognitive comments (1.52%); this was in line with previous studies (Cheng et al., 2015; Lin, 2018).
Fig. 13 depicts the ratio difference in the quality of peer-review content provided by the two groups with the different approaches.
More affective comments were found in the control group, including supporting comments (A1) (the experimental group = 45 (5.3%);
the control group = 49 (5.7%)) and opposing comments (A2) (the experimental group = 11 (1.3%); the control group = 13 (1.5%)). In
the current study, the types of comments students provided the most were cognitive comments, including direct correction (C1) (the
experimental group = 111 (13.0%); the control group = 61 (7.1%)), personal opinion (C2) (the experimental group = 214 (25%); the
control group = 179 (20.8%) and guidance (C3) (the experimental group = 101 (11.8%); the control group = 56 (6.6%)). Further
analysis pointed out that the experimental group provided more cognitive comments than the control group. In the whole activity, few
metacognitive comments (M1 or M2) or irrelevant comments (IR) could be found. The study only discovered a few metacognitive
comments relevant to applying knowledge, skills, or strategies (M1) (the experimental group = 7 (0.8%); the control group = 3
(0.4%)).
5.9. Interviews
After the learning activity, we randomly chose five students from each group to take part in 30-min qualitative interviews with the
aim of understanding their perceptions of the learning activity. The following is a summary of the interview results. Three stages based
on the grounded theory were applied in the analysis of the interview data, namely open coding, axial coding, and selective coding
(Glaser & Strauss, 1967). The purpose of the interview was to understand the students’ ideas and suggestions regarding using the
different peer-review approaches, and as the basis for further improvement.
(1) Interviews of the experimental group
From the interview results of the experimental group, in the online interactive peer-review flipped learning approach, the students
mainly had several perceptions, including “strengthen learning,” “improve health assessment skills,” “promote reflection,” and
Fig. 13. A comparison of the peer-review content in the peer-review activity with the different approaches.
18
“increase decision-making ability.”

With regard to “strengthen learning,” many students mentioned that the online interactive peer-review flipped learning approach
could strengthen their learning and enable them to gain more knowledge and skills through their interaction with classmates. For
example, E01 expressed, “During the rating process, I can learn from the parts that are easy for us to forget or make mistakes. I think the
effect is good. Because I know that my classmates will read my comments, I watched the video very seriously and I had deeper im
pressions.” E03 claimed, “I used to think about where I hadn’t done well after practicing the skills, and my memory was always unclear.
Rating others allows me to understand others’ advantages when conducting OSCE. In this way, I can learn from others’ practices. I can
deepen my impressions by reviewing each other and looking back on my own video. Also, I can see my classmates’ suggestions on how
I can improve.” EN05 stated, “I can confirm the unclear points by replying to my classmates’ comments. When I reply to the comments,
I will go back to the content of the pre-class instructional video to check whether I have made a mistake and to strengthen my memory.”
In terms of “improve health assessment skills,” several students indicated that in the online interactive peer-review flipped learning
approach, their health assessment skills could be enhanced by watching pre-class instructional videos, by strengthening skills practice
in class, and by rating peers, understanding rubrics and repeatedly watching videos. For instance, E01 expressed, “The pre-class
instructional video allows me to quickly organize the key points. Also, the rubrics allow me to clearly know my strengths, weak
nesses, and completeness of each dimension. I will know more which part to strengthen during the practice, which is very helpful for
the skill test.” E02 mentioned, “Although I already know the key points of physical assessment through the pre-class instructional
video, rating my classmates can help me better understand the completeness and accuracy of the techniques. By watching others’
practice from different perspectives and attitudes, when providing comments, I will introspect and think more about the perspectives
as the rater. I feel more engaged and impressed. In addition, I can repeatedly watch and revise, which makes me more skilled.” E04
stated, “I am clearer about the structure of the OSCE, and I am more impressed with the operation of each part, especially for physical
examination.”
As for “promote reflective thinking,” many interviewees pointed out that in the online interactive peer-review flipped learning
approach, aside from providing and accepting comments based on the rubrics, playing the role of assessees to respond to comments
could enable them to reflect on what needed to be improved at the same time. For example, E01 expressed, “After watching the video
and the advice provided by my classmates, I found that my history-taking process was really incomplete and the physical examination
was not very accurate.” E03 implied, “With the rubrics, I can know the way of correctly rating and see my mistakes more clearly. I can
reflect on my mistakes through comments from others.” E04 mentioned, “We do not watch a demonstration video; instead, we must
give ratings and comments to each other. Therefore, we can learn the advantages and disadvantages from our classmates more
seriously by watching the videos, and I will also think about whether I do the same thing.”
With regard to “increase decision making ability,” in the online interactive peer-review flipped learning approach, students made
sure whether their knowledge and application were reasonable based on their peers’ responses, or revised their original ideas and
judgments through their peers’ suggestions. For example, E03 conveyed, “My classmates’ opinions let me know more about the dif
ferential diagnosis. I only saw the obvious symptoms of the patient, but I need to take the patient’s occupation into account because he
may have venous thrombosis. When I asked the patient about his occupation, I just wanted to ask the TOCC; I didn’t think carefully
about the correlation between the patient’s illness and occupation.” E04 revealed, “I think the suggestions given by my classmates are
valuable. We can learn from each other, including history-taking techniques and thinking directions of differential diagnosis; it is very
helpful for us.” E05 disclosed, “I learned how to quickly and logically carry out the history taking through peers’ videos. I learned how
to manage the time, or I would not be able to diagnose and deal with it when I heard the bell ringing. So, I watched the video
repeatedly, and mastered the history taking and physical examination techniques. Then, I could have more time to think about the
differential diagnosis for patients.”
(2) Interviews of the control group
Based on the interview results of the control group, with the online peer-review flipped learning approach, the students had several
major perceptions, including “increase health assessment skills” and “promote reflective thinking.”
In terms of “increase health assessment skills,” several interviewees mentioned that the online peer-review flipped learning
approach could allow them to adjust the learning time, repeatedly watch and observe using cellphones or tablet computers, and
enhance their health assessment skills. For instance, CN01 conveyed, “The configuration of the rubrics can let us know where we need
to improve. We can learn more assessment skills from others’ performance such as medical history inquiry. We cannot conduct the
history taking in the same way for every symptom, or we will not get the information.” CN02 expressed, “Watching the instructional
video can allow us to freely adjust the class time. I think I learned the most about the accuracy and smoothness of physical examination.
With the busy life, I can repeatedly learn from the video and be more proficient in the parts of OSCE that I do not understand.” CN03
mentioned, “You can directly imitate and learn the good assessment skills without time and space limitation. You can watch the movie
a few times more and enhance your own skills.”
With regard to “promote reflective thinking,” the control group favored the online peer-review approach; by commenting on other
students’ videos, they could reflect on the advantages and disadvantages of their clinical skills and increase their learning effectiveness.
For instance, CN02 stated, “Watching peers’ videos can correct my deficiencies or keep my advantages, and we learn from each other.
This kind of teaching approach is different from the previous inflexible approach. It can strengthen our logical thinking. Based on the
ratings and suggestions given by different people, I can deepen my impressions, and I will not make the same mistakes again.” CN03
conveyed, “I watched the film carefully, and I found that I could see many of my deficiencies when compared with my classmates’
performance.” CN04 mentioned, “I usually practice on my own and do not know my problems. When I watched the video, I found that I
19
was so nervous that I spoke too low. The examiner might not hear what I said. When I carried out the history taking, I did not speak
fluently. I could learn from others from the video.” CN04 claimed, “This class made me discover and reflect on myself. Usually, we are
used to the standardized working mode. Practicing to look at one thing from different angles will help us produce more ideas.”
(3) Discussion of the interview results
The key points from the two groups’ interviews were summarized. The control group expressed the advantages of flipped learning
and mentioned that they increased their learning efficiency through time management and autonomous learning. As for online peer-
review, reviewing others’ videos provided them with an excellent opportunity for reflective thinking, and enhanced their skill learning
effectiveness. When the experimental group adopted the online interactive peer-review learning approach, the students could increase
their in-depth learning by responding to the comments. Besides, they could have a greater learning effect on how to accurately collect
data, make judgements and make decisions on differential diagnosis based on the interaction of online reviews. As could be seen,
integrating the online interactive peer review and interactive discussion with peers had considerable benefits for critical thinking and
decision making.
(4) Other problems or suggestions
Regarding the online interactive peer-review flipped learning activity, the majority of students possessed a positive attitude;
however, some students addressed some suggestions, for instance, technical problems relevant to the peer-review system. Two stu
dents conveyed that they were still not familiar with the interfaces of the peer-review system, and hoped to receive more explanation of
the operation of the system webpage operations. Furthermore, the angle of the camera on the wall affects the viewing position of the
physical examination. Two students complained that the viewing angle was restricted, suggesting that the film could be taken from the
examiner’s location. Also, two students mentioned and hoped to obtain the teacher’s rating and comments. As pointed out by previous
studies, students pay a great deal of attention to teachers’ feedback and opinions. In particular, they hope to receive compliments from
teachers (Bader, Burner, Hoem Iversen, & Varga, 2019; Ghahari & Farokhnia, 2018). Moreover, the students mostly agreed that the
online interactive peer review could be applied to the health assessment flipped learning course. It is suggested that this teaching
approach be adopted in courses relevant to skill practice and actual practice such as Advanced Cardiac Life Support (ACLS) and clinical
communication.
6. Discussion and conclusions
In this learning activity, the online interactive peer-review flipped learning approach was proposed and applied to a case study
analysis activity in a nursing course. The findings indicated that employing the online interactive peer-review approach had significant
benefits for the students’ learning achievement, health assessment skills, critical thinking tendency, and reflective thinking.
6.1. Discussion
Regarding the first research question, although the experimental group had higher post-test scores than the control group, the
significant effect was because the control group performed worse on the post-test compared to their pre-test. This implies that the
proposed approach at least prevented the students from performing worse when facing the more challenging tasks. As indicated by
several researchers, more peer interactions between assessors and assessees would generally enable them to perceive the learning
targets from diverse views, which could prevent them from missing important factors that might affect their judgement (Nicol &
Macfarlane-Dick, 2006; Yu, 2011). In addition, the students knew that their comments would be rated and commented on by the
assessees, so they gave comments and suggestions more seriously. While increasing the conversation on comments between peers, the
assessors made efforts to search for relevant data, which could also prevent them from making serious mistakes.
Regarding the second research question, in addition to being proficient in physical assessment techniques, the most important
features of the OSCE were evaluation of students’ ability to discover problems, and conducting comprehensive analysis, judgment, and
decision making based on the collected information (Deng, Fenn, & Plake, 2019; Harden et al., 1975; Prince et al., 2017). The current
study found that students participating in the online interactive peer-review flipped learning activity performed better on OSCE total
scores than those participating in the conventional peer-review activity. There were three dimensions in OSCE. The experimental
group showed better performances in the “physical examination” and “clinical reasoning” dimensions than the control group, while no
significant difference was found between their scores in the “history-taking” dimension. Regarding the “physical examination”
dimension, several interviewees in this study conveyed that repeatedly watching peers’ OSCE videos could increase their skill learning
opportunities. Previous research has also indicated that, when watching peers’ skill performance, students can clearly identify their
own problems and practice more toward the correct direction: moreover, they can learn from their peers when providing and receiving
peers’ constructive suggestions and opinions (Cushing, Abbott, Lothian, Hall, & Westwood, 2011; Rush, Firth, Burke, & Marks-Maran,
2012; White, Ross, & Gruppen, 2009). In this study, the assessors and assessees in the online interactive peer-review group had more
opportunities to remind each other than those in the control group regarding their physical examination performance. As indicated by
several researchers, both assessors and assessees could benefit from learning from each other during the peer-review process (Maas
et al., 2014; Tornwall, 2018). In terms of the “clinical reasoning” dimension, the interactions between the assessors and assessees can
not only reduce their doubt and misunderstanding regarding the assessment results (Adachi, Tai, & Dawson, 2018), but also provide
20
more opportunities for them to discuss and make reflections on their comments to peers. This could engage them in deep and reflective
thinking as well as thinking diversely, and hence improve their clinical reasoning performance. In terms of the “history taking”
dimension, there was no significant difference between the scores of the two groups. This could be due to the fact that the items for
assessing the “history taking” dimension were generally in the category of memorization rather than comprehension, while more
interactions between assessors and assessees did not contribute much in this regard.
Regarding the third research question, the results were consistent with those of previous studies. For example, Chang et al. (2020)
pointed out that integrating peer review in a virtual reality (VR) activity in a science class could not only increase students’ learning
performance but also enhance their self-efficacy and critical thinking tendency. In addition, adopting peer review to learn English
speaking in a spherical video-based virtual reality (SVVR) context could significantly increase students’ critical thinking ability (Chien,
Hwang, & Jong, 2020). Critical thinking ability is one of the higher order thinking capabilities. In peer review, learners learn to
comment on works from different perspectives; the process of commenting involves logical thinking and skills in order to make
reasonable statements or evaluations (Hovardas, Tsivitanidou, & Zacharia, 2014). The interactive peer review approach provided
opportunities to think deeply about peers’ responses to their own comments. Consequently, there was a positive influence on students’
critical thinking tendency.
In terms of the fourth research question, there was no statistically significant difference in the self-efficacy of the two groups.
According to previous studies, peer-review has different results in terms of enhancing learners’ self-efficacy. Some studies have
indicated that applying peer review helps students reflect, and improves their self-efficacy as well as their learning performance (Hsia,
Huang, & Hwang, 2016). Yet, some studies found no correlation between students’ learning performance and their self-efficacy
(Lauder et al., 2008). Researchers specify that self-efficacy is related to self-evaluation; students who give themselves higher scores
may possess stronger self-efficacy (Papinczak, Young, Groves, & Haynes, 2007). Also, based on such factors as professionalism and
psychology, learners trust teachers’ comments more than peers’ reviews (Ghahari & Farokhnia, 2018). In the present study, the teacher
provided verbal feedback after the OSCE, so the students found it difficult to memorize the feedback. Some interviewees mentioned
and hoped that the teacher’s ratings and comments could be included in the experiment. In the future, if self-assessment and online
teachers’ comments and feedback can be included in peer review for students to repeatedly read the teachers’ feedback, it may be
possible to increase the learners’ self-efficacy and help them obtain greater learning benefits (Cushing et al., 2011).
With regard to reflective thinking, the present study verified that students using the online interactive peer-review flipped learning
activity performed better reflective thinking than those using the conventional peer-review activity, thus answering the fifth research
question. After the course, the slope of the increase in the total score of the experimental group indicated that their reflective thinking
was somewhat higher than that of the control group. However, only in one of the four dimensions, the reflection sub-scale, was the
magnitude of the score increase of the experimental group higher than that of the control group after learning. For the other three
dimensions (i.e., habitual action, understanding, and critical reflection), no significant difference was found between the two groups in
the magnitude of score increase after learning. It was challenging to provide comments for peers; the peer-review process enabled the
students to understand their advantages and disadvantages more clearly. Previous research has already corroborated that peer review
reinforces learners’ self-reflection (Elshami & Abdalla, 2017; Rush et al., 2012). When learners observe their peers’ performance in
simulated situations and evaluate other students’ skills, they think about their own health assessment skills through self-awareness and
conversation with peers (Ramm, Thomson, & Jackson, 2015). In this study, most of the students revealed in the interviews that peer
review promoted their reflective thinking. Students who adopted the online interactive peer review offered and obtained more
guidance and cognitive comments through the interaction. When responding to comments, the opportunity for communication and
conversation was increased. Students thought, organized, revised again, and generated new knowledge concepts based on peers’
perspectives. As a result, the online interactive peer-review activity assisted students in improving their reflective thinking. These
higher order thinking abilities are exactly what is required for nurse practitioners in the training course of health assessment.
The sixth research question explored the influences of two different online peer-review approaches on the quality of students’ peer-
review content. Regarding the accuracy of peer ratings, the ratings of the two groups and the experts were significantly correlated
(Chien, Hwang, & Jong, 2020; Hsia, Huang, & Hwang, 2016). Compared with the ratings from students in the conventional
peer-review activity, the ratings from students in the online interactive peer-review activity had a higher correlation with those of the
experts. Nevertheless, based on the r2 explanatory power, 0.3 < r2 < 0.5 is often regarded as low explanatory power. Hence, the
correlation between the experts’ and peers’ ratings should be explained in a conservative way. Besides, based on the analytic results of
the peer-review content, students adopting the online interactive peer-review flipped learning activity produced better quality peer
review than those adopting the conventional peer-review flipped learning activity, especially for cognitive comments. The findings
substantiated that students in the online interactive peer-review activity devoted themselves more to providing comments. By pointing
out students’ skill performance in the videos, providing personal opinions and specific suggestions for improvement, both assessors
and assessees had the opportunity to think deeply, integrate and modify, and then generate new knowledge concepts (Cheng et al.,
2015). According to the major findings in this study, the online interactive peer review enabled learners to participate in learning more
actively. Because of more interaction, students watched the videos more carefully and tried hard to give comments, which was re
flected in the increase in the quantity and quality of the comments. The findings were similar to those of previous studies; compared
with affective comments, cognitive comments accounted for the majority of the peer-review comments (Cheng et al., 2015; Cheng &
Hou, 2015; Lin, 2018).
Both the experiment findings and the students’ interviews verified the concepts proposed by the researchers. That is, in the learning
process, students can complete the learning content more effectively and improve their health assessment skills as well as higher order
thinking capabilities through the online interactive peer-review flipped learning approach. Previous research has indicated that
increasing interaction has a positive effect on students’ learning results (Fulantelli, Taibi, & Arrigo, 2015). Students analyze and
21
synthesize the acquired knowledge and skills through interaction with peers, and use critical thinking to guide students to collect data
for analysis, judgment, and decision making so as to cultivate their problem-solving ability.
From the correlation analysis results, there was a significant positive correlation between critical thinking and reflective thinking as
well as reflective thinking and the health assessment knowledge test. From the perspective of meta-cognition, engaging students in the
interactive peer-review activity enabled them to be aware of their own learning status and to make reflective thinking via reviewing
peers’ work and receiving peers’ responses to their reviews, as well as receiving and commenting on peers’ reviews. As indicated by
prior studies, effective peer interactions and knowledge sharing could improve students’ critical thinking and reflective thinking
abilities (Belcher, Hall, Kelley, & Pressey, 2015; Lin, 2019). Such a process involving self-awareness and reflective thinking enables
them to perceive things from a wider perspective for making objective judgments as well as finding effective ways to improve their
learning performances. Several researchers have pointed out that knowing one’s own status and making reflections are the key to
improving one’s learning performances and competences of making correct decisions (Cheng et al., 2020; Zarifsanaiey, Etemadi, &
Rezaee, 2018). Therefore, it is worth conducting follow-up studies to further examine the impacts of applying the interactive
peer-review approach to other professional training programs aimed at fostering learners’ critical thinking and decision-making
abilities.
6.2. Limitations and suggestions
The present study has some limitations. First of all, the flipped learning and online interactive peer review were only implemented
for 2 weeks; the duration of the intervention was thus very short. Also, the study only explored nurse practitioners in a health
assessment course; the results may not be generalized to other learners or courses.
Nevertheless, the current study initially verified the influences of the online interactive peer review on nurse practitioner students’
learning effectiveness in the health assessment flipped learning course. In addition to the two groups’ knowledge, skills, and higher
order thinking, the study also compared the differences in the quality of peer-review content under different learning approaches. The
online interactive peer-review approach proposed in the current study can be applied to other case analysis or project-based learning
activities in the future, for example, communication skills, case reports, and advanced cardiac lifesaving (ACLS). Based on the current
findings, the study addresses the following suggestions for relevant studies in the future:
(1) Due to the short experiment in this study, it is recommended that the time of the experiment be extended in the future to
compare peer-review learning methods with different approaches or to explore the change in quality and learning effectiveness
at different points in time.
(2) It is suggested that teachers’ comments and feedback be added to the peer review and that their effects on students’ learning
effectiveness be further investigated in the future.
(3) Good rubrics increase students’ rating ability. In the future, participating in designing rubrics can be added to explore their
influences on learning effectiveness.
(4) It is recommended that the effects of students’ self-assessment and teachers’ assessment on students’ self-efficacy be examined.
(5) It is recommended that the influences of the peer-review learning approach on students in different groups, such as learners
with high (low) learning achievements, or students with high (low) learning motivation be further explored, as this could assist
students in achieving more effective learning.
Author statement
Hui-Chen Lin: Formal analysis, Investigation, Data Curation, Writing - Original Draft, Writing - Review & Editing. Gwo-Jen
Hwang: Conceptualization, Methodology, Software, Writing - Review & Editing. Shao-Chen Chang: Software, Data Curation, Writing
- Original Draft. Yaw-Don Hsu: Methodology, Validation, Writing - Review & Editing.
Acknowledgements
This study is supported in part by the Ministry of Science and Technology of Taiwan under contract numbers MOST-109-2511-H-
011-002-MY3 and MOST-108-2511-H-011-005-MY3.
References
Abelsson, A., & Bisholt, B. (2017). Nurse students learning acute care by simulation–Focus on observation and debriefing. Nurse Education in Practice, 24, 6–13.
Adachi, C., Tai, J. H. M., & Dawson, P. (2018). Academics’ perceptions of the benefits and challenges of self and peer assessment in higher education. Assessment &
Evaluation in Higher Education, 43(2), 294–306.
Adams, P. (2006). Exploring social constructivism: Theories and practicalities. Education, 34(3), 243–257.
Andrade, H., & Du, Y. (2005). Student perspectives on rubric-referenced assessment. Practical Assessment, Research and Evaluation, 10(3), 1–11.
Bader, M., Burner, T., Hoem Iversen, S., & Varga, Z. (2019). Student perspectives on formative feedback as part of writing portfolios. Assessment & Evaluation in Higher
Education, 44(7), 1017–1028.
Baytiyeh, H., & Naja, M. K. (2017). Students’ perceptions of the flipped classroom model in an engineering course: A case study. European Journal of Engineering
Education, 42(6), 1048–1061.
Belcher, A. R., Hall, B. M., Kelley, K., & Pressey, K. L. (2015). An analysis of faculty promotion of critical thinking and peer interaction within threaded discussions.
Online Learning, 19(4), n4.
22
Bellman, L. M. (1996). Changing nursing practice through reflection on the roper, logan and tierney model: The enhancement approach to action research. Journal of
Advanced Nursing, 24(1), 129–138.
Bergmann, J., & Sams, A. (2012). Flip your classroom: Reach every student in every class every day. Washington, DC: International Society for Technology in Education.
Bond, M. (2020). Facilitating student engagement through the flipped classroom approach in K-12: A systematic review. Computers & Education, 151, 103819.
Brinko, K. T. (1993). The practice of giving feedback to improve teaching: What is effective? The Journal of Higher Education, 64(5), 574–593.
Çevik, Y. D. (2015). Assessor or assessee? Investigating the differential effects of online peer assessment roles in the development of students’ problem-solving skills.
Computers in Human Behavior, 52, 250–258.
Cevik, Y. D., Haşlaman, T., & Çelik, S. (2015). The effect of peer assessment on problem solving skills of prospective teachers supported by online learning activities.
Studies In Educational Evaluation, 44, 23–35.
Chai, C. S., Deng, F., Tsai, P. S., Koh, J. H. L., & Tsai, C. C. (2015). Assessing multidimensional students’ perceptions of twenty-first-century learning practices. Asia
Pacific Education Review, 16(3), 389–398.
Chen, S. Y., Chang, H. C., & Pai, H. C. (2018). Caring behaviours directly and indirectly affect nursing students’ critical thinking. Scandinavian Journal of Caring
Sciences, 32(1), 197–203.
Chang, S. C., Hsu, T. C., & Jong, M. S. Y. (2020). Integration of the peer assessment approach with a virtual reality design system for learning earth science. Computers
& Education, 146, 103758.
Chang, S. C., & Hwang, G. J. (2018). Impacts of an augmented reality-based flipped learning guiding approach on students’ scientific project performance and
perceptions. Computers & Education, 125, 226–239.
Chen, F. F., Chen, S. Y., & Pai, H. C. (2019). Self-reflection and critical thinking: The influence of professional qualifications on registered nurses. Contemporary Nurse,
55(1), 59–70.
Cheng, K. H., & Hou, H. T. (2015). Exploring students’ behavioural patterns during online peer assessment from the affective, cognitive, and metacognitive
perspectives: A progressive sequential analysis. Technology, Pedagogy and Education, 24(2), 171–188.
Cheng, Y. C., Huang, L. C., Yang, C. H., & Chang, H. C. (2020). Experiential learning program to strengthen self-reflection and critical thinking in freshmen nursing
students during COVID-19: A quasi-experimental study. International Journal of Environmental Research and Public Health, 17(15), 5442.
Cheng, K. H., Liang, J. C., & Tsai, C. C. (2015). Examining the role of feedback messages in undergraduate students’ writing performance during an online peer
assessment activity. The Internet and Higher Education, 25, 78–84.
Chien, S. Y., Hwang, G. J., & Jong, M. S. Y. (2020). Effects of peer assessment within the context of spherical video-based virtual reality on EFL students’ English-
Speaking performance and learning perceptions. Computers & Education, 146, 103751.
Chiou, S. F., Su, H. C., Liu, K. F., & Hwang, H. F. (2015). Flipped classroom: A new teaching strategy for integrating information technology into nursing education.
Journal of Nursing, 62(3), 5–10.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd). Hillsdale, NJ: erlbaum.
Cushing, A., Abbott, S., Lothian, D., Hall, A., & Westwood, O. M. (2011). Peer feedback as an aid to learning–what do we want? Feedback. When do we want it? Now!
Medical Teacher, 33(2), e105–e112.
Dehghanzadeh, S., & Jafaraghaee, F. (2018). Comparing the effects of traditional lecture and flipped classroom on nursing students’ critical thinking disposition: A
quasi-experimental study. Nurse Education Today, 71, 151–156.
Delgado, C., & Mack, B. (2002). A peer-reviewed program for senior proficiencies. Nurse Educator, 27(5), 212–213.
Deng, B., Fenn, N. E., III, & Plake, K. S. (2019). Impact of a teaching objective structured clinical examination (TOSCE) on student confidence in a pharmacy skills
laboratory. Currents in Pharmacy Teaching and Learning, 11(2), 145–154.
Elshami, W., & Abdalla, M. (2017). Diagnostic radiography students’ perceptions of formative peer assessment within a radiographic technique module. Radiography,
23(1), 9–13.
Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G* power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences.
Behavior Research Methods, 39(2), 175–191.
Filius, R. M., de Kleijn, R. A., Uijl, S. G., Prins, F. J., van Rijen, H. V., & Grobbee, D. E. (2018). Strengthening dialogic peer feedback aiming for deep learning in SPOCs.
Computers & Education, 125, 86–100.
Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., et al. (2014). Active learning increases student performance in science, engineering,
and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410–8415.
Fulantelli, G., Taibi, D., & Arrigo, M. (2015). A framework to support educational decision making in mobile learning. Computers in Human Behavior, 47, 50–59.
Ghahari, S., & Farokhnia, F. (2018). Peer versus teacher assessment: Implications for CAF triad language ability and critical reflections. International Journal of School
& Educational Psychology, 6(2), 124–137.
Glaser, B., & Strauss, A. (1967). The discovery of grounded theory. Chicago, IL: Aldine.
Han, Z., & Chan, A. (2020). Pharmacy students’ perceptions toward peer assessment and its use in teaching patient presentation skills. Currents in Pharmacy Teaching
and Learning, 12(2), 228–236.
Hanson, J. (2016). Surveying the experiences and perceptions of undergraduate nursing students of a flipped classroom approach to increase understanding of drug
science and its application to clinical practice. Nurse Education in Practice, 16(1), 79–85.
Harden, R.t. M., Stevenson, M., Downie, W. W., & Wilson, G. (1975). Assessment of clinical competence using objective structured examination. British Medical
Journal, 1(5955), 447–451.
Hayes, C., Jackson, D., Davidson, P. M., Daly, J., & Power, T. (2018). Pondering practice: Enhancing the art of reflection. Journal of Clinical Nursing, 27(1–2),
e345–e353.
Heo, H., & CHun, B. A. (2018). Improving the higher order thinking skills using flipped learning: Focused on the in-class activities with problem posing and solving.
Asia Life Sciences Supplement, 15(4), 2187–2199.
Hettinga, A. M., Denessen, E., & Postma, C. T. (2010). Checking the checklist: A content analysis of expert-and evidence-based case-specific checklist items. Medical
Education, 44(9), 874–883.
Himes, D. O., & Ravert, P. K. (2012). Situated peer coaching and unfolding cases in the fundamentals skills laboratory. International Journal of Nursing Education
Scholarship, 9(1).
Hovardas, T., Tsivitanidou, O. E., & Zacharia, Z. C. (2014). Peer versus expert feedback: An investigation of the quality of peer feedback among secondary school
students. Computers & Education, 71, 133–152.
Hsia, L. H., Huang, I., & Hwang, G. J. (2016). Effects of different online peer-feedback approaches on students’ performance skills, motivation and self-efficacy in a
dance course. Computers & Education, 96, 55–71.
Hsia, L. H., & Sung, H. Y. (2020). Effects of a mobile technology-supported peer assessment approach on students’ learning motivation and perceptions in a college
flipped dance class. International Journal of Mobile Learning and Organisation, 14(1), 99–113.
Huisman, B., Saab, N., Van Driel, J., & Van Den Broek, P. (2018). Peer feedback on academic writing: Undergraduate students’ peer feedback role, peer feedback
perceptions and essay performance. Assessment & Evaluation in Higher Education, 43(6), 955–968.
Hwang, G. J., Yang, T. C., Tsai, C. C., & Yang, S. J. (2009). A context-aware ubiquitous learning environment for conducting complex science experiments. Computers
& Education, 53(2), 402–413.
Karabulut-Ilgu, A., Yao, S., Savolainen, P., & Jahren, C. (2018). Student perspectives on the flipped-classroom approach and collaborative problem-solving process.
Journal of Educational Computing Research, 56(4), 513–537.
Kember, D., Leung, D. Y., Jones, A., Loke, A. Y., McKay, J., Sinclair, K., … Wong, M. (2000). Development of a questionnaire to measure the level of reflective
thinking. Assessment & Evaluation in Higher Education, 25(4), 381–395.
Kim, M. K., Kim, S. M., Khera, O., & Getman, J. (2014). The experience of three flipped classrooms in an urban university: An exploration of design principles. The
Internet and Higher Education, 22, 37–50.
23
King, P. M., & Kitchener, K. S. (1994). Developing reflective judgment: Understanding and promoting intellectual growth and critical think in adolescents and adults. San
Francisco: Jossey-Bass.
Kong, S. C. (2014). Developing information literacy and critical thinking skills through domain knowledge learning in digital classrooms: An experience of practicing
flipped classroom strategy. Computers & Education, 78, 160–173.
Lai, C. Y. (2016). Training nursing students’ communication skills with online video peer assessment. Computers & Education, 97, 21–30.
Lauder, W., Holland, K., Roxburgh, M., Topping, K., Watson, R., Johnson, M., et al. (2008). Measuring competence, self-reported competence and self-efficacy in pre-
registration students. Nursing Standard, 22(20), 35.
Lee, K. L., & Norbaizura, M. N. (2016). Assessment for Learning: Students’ perception on peer review in a private university. Assessment for Learning Within and Beyond
the Classroom, 199–210.
Lee, M. K., & Park, B. K. (2018). Effects of flipped learning using online materials in a surgical nursing practicum: A pilot stratified group-randomized trial. Healthcare
Informatics Research, 24(1), 69–78.
Lei, C. U., Yau, C. W., Lui, K. S., Tam, V., Yuen, A. H., & Lam, E. Y. (2019). Designing instructional videos and class work activities: Teaching internet of things via
flipped classroom. International Journal Of Mobile Learning And Organisation, 13(4), 392–411.
Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational
Technology, 41(3), 525–536.
Lin, G. Y. (2018). Anonymous versus identified peer assessment via a Facebook-based learning application: Effects on quality of peer feedback, perceived learning,
perceived fairness, and attitude toward the system. Computers & Education, 116, 81–92.
Lin, C. J. (2019). An online peer assessment approach to supporting mind-mapping flipped learning activities for college English writing courses. Journal of Computers
in Education, 6(3), 385–415.
Lin, K. Y., & Shen, Y. F. (2013). The nursing students’ attitude toward using blogs in a nursing clinical practicum in taiwan: A 3-R framework. Nurse Education Today,
33(9), 1079–1082.
Maas, M. J., Sluijsmans, D. M., van der Wees, P. J., Heerkens, Y. F., Nijhuis-van der Sanden, M. W., & van der Vleuten, C. P. (2014). Why peer assessment helps to
improve clinical performance in undergraduate physical therapy education: A mixed methods design. BMC Medical Education, 14(1), 117.
Magrabi, S. A. R., Pasha, M. I., & Pasha, M. Y. (2018). Classroom teaching to enhance critical thinking and problem-solving skills for developing IOTApplications.
Journal of Engineering Education Transformations, 31(3), 152–157.
Mann, K., Gordon, J., & MacLeod, A. (2009). Reflection and reflective practice in health professions education: A systematic review. Advances in Health Sciences
Education, 14(4), 595–621.
Marrocco, G. F., Ginzburg, E., & Feder, S. L. (2019). Seeing from the eyes of the beholder: A pilot program for learning through reflection in graduate nursing
education. Nursing Education Perspectives, 40(2), 113–115.
Maudsley, G., & Strivens, J. (2000). Promoting professional knowledge, experiential learning and critical thinking for medical students. Medical Education, 34(7),
535–544.
Mavis, B. E., Wagner, D. P., Henry, R. C., Carravallah, L., Gold, J., Maurer, J., … Saxe, A. (2013). Documenting clinical performance problems among medical
students: Feedback for learner remediation and curriculum enhancement. Medical Education Online, 18(1), 20598.
McAllister, M., & Osborne, Y. (1997). Peer review: A strategy to enhance cooperative student learning. Nurse Educator, 22(1), 40–44.
Morrell-Scott, N. (2018). Confidence and deeper learning: Simulated practice within pre-registration nurse education. British Journal of Nursing, 27(14), 827–830.
Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher
Education, 31(2), 199–218.
O’Brien, B., & Graham, M. M. (2020). BSc nursing & midwifery students experiences of guided group reflection in fostering personal and professional development.
Part 2. for discussion and innovation section. Nurse Education in Practice, Article 102884.
O’Brien, T., Talbot, L., & Santevecchi, E. (2015). Strengthening clinical skills using peer-student validation. Nurse Educator, 40(5), 237–240.
Papastavrou, E., Hamari, L., Fuster, P., Istomina, N., & Salminen, L. (2016). Using blogs for facilitating and connecting nurse educator candidates. Nurse Education
Today, 45, 35–41.
Papinczak, T., Young, L., Groves, M., & Haynes, M. (2007). An analysis of peer, self, and tutor assessment in problem-based learning tutorials. Medical Teacher, 29(5),
e122–e132.
Park, E. O., & Park, J. H. (2018). Quasi-experimental study on the effectiveness of a flipped classroom for teaching adult health nursing. Japan Journal of Nursing
Science, 15(2), 125–134.
Persson, E. K., Kvist, L. J., & Ekelin, M. (2015). Analysis of midwifery students’ written reflections to evaluate progression in learning during clinical practice at
birthing units. Nurse Education in Practice, 15(2), 134–140.
Planas Lladó, A., Soley, L. F., Fraguell Sansbelló, R. M., Pujolras, G. A., Planella, J. P., Roura-Pascual, N., … Moreno, L. M. (2014). Student perceptions of peer
assessment: An interdisciplinary study. Assessment & Evaluation in Higher Education, 39(5), 592–610.
Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223–231.
Prince, L. K., Campbell, R. C., Gao, S. W., Kendrick, J., Lebrun, C. J., Little, D. J., … Saddler, M. (2017). The dialysis orders objective structured clinical examination
(OSCE): A formative assessment for nephrology fellows. Clinical Kidney Journal, 11(2), 149–155.
PrinceRobinson, J. (2015). Reflection, return to practice and revalidation. Nursing Older People, 27(6), 31–37.
Ramm, D., Thomson, A., & Jackson, A. (2015). Learning clinical skills in the simulation suite: The lived experiences of student nurses involved in peer teaching and
peer assessment. Nurse Education Today, 35(6), 823–827.
Román-González, M., Pérez-González, J.-C., & Jiménez-Fernández, C. (2017). Which cognitive abilities underlie computational thinking? Criterion validity of the
computational thinking test. Computers in Human Behavior, 72, 678–691.
Rotz, M. E., & O’Neill, D. D. (2020). Exploring second professional year pharmacy student reflections after an interprofessional shadowing experience with nursing
students in critical care units: A qualitative study. Journal of Interprofessional Care, 1–8.
Rush, S., Firth, T., Burke, L., & Marks-Maran, D. (2012). Implementation and evaluation of peer assessment of clinical skills for first year student nurses. Nurse
Education in Practice, 12(4), 219–226.
Schommer-Aikins, M., & Hutter, R. (2002). Epistemological beliefs and thinking about everyday controversial issues. Journal of Psychology, 136(1), 5–20.
Sedlak, C. A., & Doheny, M. O. B. (1998). Peer review through clinical rounds: A collaborative critical thinking strategy. Nurse Educator, 23(5), 42–45.
Slomer, A., & Chenkin, J. (2017). Does test-enhanced learning improve success rates of ultrasound-guided peripheral intravenous insertion? A randomized controlled
trial. AEM Education and Training, 1(4), 310–315.
Solheim, E., Plathe, H. S., & Eide, H. (2017). Nursing students’ evaluation of a new feedback and reflection tool for use in high-fidelity simulation–Formative
assessment of clinical skills. A descriptive quantitative research design. Nurse Education in Practice, 27, 114–120.
Strayer, J. F. (2012). How learning in an inverted classroom influences cooperation, innovation and task orientation. Learning Environments Research, 15(2), 171–193.
Strelan, P., Osborn, A., & Palmer, E. (2020a). The flipped classroom: A meta-analysis of effects on student performance across disciplines and education levels.
Educational Research Review, 100314.
Strelan, P., Osborn, A., & Palmer, E. (2020b). Student satisfaction with courses and instructors in a flipped classroom: A meta-analysis. Journal of Computer Assisted
Learning, 36(3), 295–314.
Sun, J. C. Y., & Wu, Y. T. (2016). Analysis of learning achievement and teacher–student interactions in flipped and conventional classrooms. International Review of
Research in Open and Distance Learning, 17(1), 79–99.
Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational Research, 68(3), 249–276.
Topping, K. J. (2009). Peer assessment. Theory Into Practice, 48(1), 20–27.
Tornwall, J. (2018). Peer assessment practices in nurse education: An integrative review. Nurse Education Today, 71, 266–275.
24
Tricio, J. A., Woolford, M. J., & Escudier, M. P. (2016). Fostering dental students’ academic achievements and reflection skills through clinical peer assessment and
feedback. Journal of Dental Education, 80(8), 914–923.
Tümkaya, S., Aybek, B., & Aldaş, H. (2009). An investigation of university students’ critical thinking disposition and perceived problem solving skills. European
Journal of Educational Research, 36, 57–74.
Vygotsky, L. S., Cole, M., John-Steiner, V., Scribner, S., & Souberman, E. (1978). The development of higher psychological processes. Mind & Society, 1–91.
Wang, S. L., & Hwang, G. J. (2012). The role of collective efficacy, cognitive quality, and task cohesion in computer-supported collaborative learning (CSCL).
Computers & Education, 58(2), 679–687.
White, C. B., Ross, P. T., & Gruppen, L. D. (2009). Remediating students’ failed OSCE performances at one school: The effects of self-assessment, reflection, and
feedback. Academic Medicine, 84(5), 651–654.
Willemse, J. J. (2015). Undergraduate nurses reflections on Whatsapp use in improving primary health care education. Curationis, 38(2), 1–7.
Ye, X. D., Chang, Y. H., & Lai, C. L. (2019). An interactive problem-posing guiding approach to bridging and facilitating pre-and in-class learning for flipped
classrooms. Interactive Learning Environments, 27(8), 1075–1092.
Yu, F. Y. (2011). Multiple peer-assessment modes to augment online student question-generation processes. Computers & Education, 56(2), 484–494.
Zainuddin, Z., & Perera, C. J. (2019). Exploring students’ competence, autonomy and relatedness in the flipped classroom pedagogical model. Journal of Further and
Higher Education, 43(1), 115–126.
Zarifsanaiey, N., Etemadi, S., & Rezaee, R. (2018). E-portfolio based learning: Implementation and evaluation. Journal of Advanced Pharmacy Education & Research|
Oct-Dec, 8(S2), 171.
Zheng, L., Chen, N.-s., Li, X., & Huang, R. (2016). The impact of a two-round, mobile peer assessment on learning achievements, critical thinking skills, and meta-
cognitive awareness. International Journal of Mobile Learning and Organisation, 10(4), 292–306.
Zhou, J., Zheng, Y., & Tai, J. H. M. (2020). Grudges and gratitude: The social-affective impacts of peer assessment. Assessment & Evaluation in Higher Education, 45(3),
345–358.
25

Computers & Education: Hui-Chen Lin, Gwo-Jen Hwang, Shao-Chen Chang, Yaw-Don Hsu

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Computers & Education: Hui-Chen Lin, Gwo-Jen Hwang, Shao-Chen Chang, Yaw-Don Hsu

Uploaded by

Copyright:

Available Formats

Computers & Education 173 (2021) 104266

Contents lists available at ScienceDirect

Computers & Education

Facilitating critical thinking in decision making-based professional

2.1. Flipped learning

2.2. Peer review

Fig. 3. The interface of the peer-review module.

Fig. 4. The interface of the responding module.

Fig. 5. The experiment procedure.

the hospital (approval number: 1-108-05-160).

4.1. Experiment procedure

(1) Learning achievement tests

(2) Health assessment skill tests

(3) Critical thinking tendency questionnaire

(4) Self-efficacy questionnaire

(5) Reflective thinking scale

(6) Peer-review rubrics

(7) The coding for content analysis of peer review

5.1. Learning achievement

5.2. Health assessment skill tests

Lower Bound Upper Bound

Control 1 77.85 (10.57) 1.70 74.46 81.24

Fig. 6. Interaction of group and test time for learning achievement.

Lower Bound Upper Bound

TC Control 1 59.12 (17.18) 2.25 54.65 63.60

Fig. 7. Interaction of group and test time for the OSCE.

5.3. Critical thinking tendency

Note. HT = history taking.

Note. PE = physical examination.

Fig. 8. Interaction of group and test time for OSCE-PE.

Note. DR = Diagnosis and Reasoning.

5.4. Reflective thinking

Fig. 9. Interaction of groups and test times for OSCE-DR.

Control 1 4.25 (0.52) 0.08 4.10 4.40

Note. CT = Critical thinking tendency.

Lower Bound Upper Bound

RT Control 1 4.23 (0.46) 0.07 4.09 4.38

Note. RR = Reflective thinking-reflection sub-scale.

Fig. 12. Interaction of group and test time for reflection.

Lower Bound Upper Bound

Control 1 4.01 (0.57) 0.09 3.84 4.18

Learning achievement 1 .027 -.018 .097 .424a

Experimental (n = 33) Peer-review 2.64 0.41 .60a

Affective Supporting (A1) 45 5.26% 49 5.73%

5.7. Peers’ and the teacher’s ratings

5.8. Quality of peer-review content

(1) Interviews of the experimental group

“increase decision-making ability.”

(2) Interviews of the control group

(3) Discussion of the interview results

(4) Other problems or suggestions

6. Discussion and conclusions

6.2. Limitations and suggestions

You might also like