Problems of Education in The 21st Century, Vol. 78, No. 1, 2020

Problems of Education in the 21st Century, Vol. 78, No.
1, 2020
ISSN 1822-7864 (Print) ISSN 2538-7111 (Online)
PROBLEMS OF EDUCATION
IN THE 21st CENTURY
Scientia Socialis, Ltd., a member of the Lithuanian Publishers Association and a member of
Publishers International Linking Association, Inc. (Crossref) in cooperation with SMC „Scientia
Educologica“, Lithuania, The Associated Member of Lithuanian Scientific Society, European
Society for the History of Science (ESHS) and ICASE (International Council of Associations for
Science Education)
The articles appearing in this journal are indexed, listed, and abstracted in:
Emerging Sources Citation Index (ESCI) (Web of CNKI (China National Knowledge Infrastructure)
Science | Clarivate Analytics) Scholar
SCOPUS Turkish Education Index
Index Copernicus Information Matrix for the Analysis of Journals
EBSCO: Education Research Complete (MIAR)
Education Research Index Database Coverage List Directory of Open Access Scholarly Resources
“Core” coverage (ROAD)
Central & Eastern European Academic Source Eurasian Scientific Journal Index
Database Coverage List (EBSCO-CEEAS) Crossref
JournalSeek ProQuest
Ulrich`s Periodicals Directory Europub: Academic and Scholarly Research
Contemporary Science Association/AAP databases Publication Center
Open Academic Journals Index (OAJI) The Scientific Literature Database (Scilit)
GIGA Information Centre (Electronic Journals
Library)
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
2 Publisher
Scientia Socialis, Ltd. in cooperation with SMC „Scientia Educologica“, Lithuania, the Associated
Member of Lithuanian Scientific Society, the Association of Lithuanian Serials, European Society
for the History of Science (ESHS) and International Council of Associations for Science Education
(ICASE)
Editors-in-Chief
Dr., Prof. Agnaldo Arroio, University of Sao Paulo, Brazil
Dr., Prof. Vincentas Lamanauskas, Šiauliai University & SMC „Scientia Educologica“, Republic
of Lithuania
Editorial Board
Dr., Prof. Boris Aberšek, University of Maribor, Slovenia
Dr., Prof. Saleh A. Alabdulkareem, King Saud University, Saudi Arabia
Dr. Monica Baptista, University of Lisbon, Portugal
Dr., Prof. Martin Bilek, Charles University, Czech Republic
Dr., Prof. Andris Broks, University of Latvia, Latvia
Dr., Prof. Marco Antonio Bueno Filho, Federal University of ABC, Brazil
Dr. Paolo Bussotti, University of Udine, Italy
Dr., Prof. Muammer Calik, Trabzon University, Turkey
Dr., Prof. Margaret Chan Kit Yok, MARA University of Technology (UiTM), Malaysia
Dr., Angela James, University of Kwazulu – Natal, South Africa
Dr., Prof. Vladimir S. Karapetyan, Armenian State Pedagogical University named after
Kh. Abovyan, Armenia
Dr., Prof. Kuo-Hung Huang, National Chiayi University, Taiwan
Dr. Milan Kubiatko, Jan Evangelista Purkyně University in Ústí nad Labem, Czech Republic
Dr., Prof. Miroslaw Kowalski, University of Zielona Gora, Poland
Dr. Todar T. Lakhvich, Belarusian State Medical University, Republic of Belarus
Dr. Eleonora Melnik, Petrozavodsk State University, Republic of Karelia, Russia
Dr., Prof. Danuše Nezvalova, Palacky University, Czech Republic
Dr. Osman Pekel, Suleyman Demirel University, Turkey
Dr., Prof. Yuriy Pelekh, Rivne State University of Humanities, Ukraine
Dr., Prof. Raffaele Pisano, University of Lille, France
Dr., Prof. Katarzyna Potyrala, Pedagogical University of Cracow, Poland
Dr. Costin Pribeanu, Academy of Romanian Scientists, Romania
Dr. Agneta Simeonsdotter Svensson, University of Gothenburg, Sweden
Dr. Uladzimir K. Slabin, University of Oregon, USA
Dr. Laima Railienė, Scientific Methodical Centre „Scientia Educologica“, Republic of Lithuania
Dr., Prof. Borislav V. Toshev, Sofia University, Bulgaria
Dr., Prof. Milan Turčani, Constantine the Philosopher University, Slovakia
Dr., Prof. Nicos Valanides, Educational Robotics and Science Organization, Cyprus
Copyright of this issue is the property of Scientia Socialis, Ltd, Lithuania. By virtue of their appearance in
this open access journal, articles are free to use, with proper attribution, in educational and other non-
commercial settings
Index Copernicus (IC™ Value): 100.00 (2018)
ICDS (Secondary Composite Index Broadcasting): 9.5 (2018).
CGIJ OAJI: 0.350 (2018).
H Index: 3 (2018)
Problems of Education in the 21st Century is an international, periodical, peer reviewed scientific journal,
issued by the Scientia Socialis Ltd. in cooperation with SMC „Scientia Educologica“.
Address: Scientia Socialis, Ltd.

Donelaičio Street 29, LT-78115 Siauliai, Lithuania
E-mail: problemsofeducation@gmail.com
Phone: +370 687 95668
WEB page: http://www.scientiasocialis.lt/pec ISSN 1822-7864 (Print) ISSN 2538-7111 (Online)
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Contents 3
Editorial
A FEW REFLECTIONS ON CONTEMPORARY SCIENCE EDUCATION AND

EDUCATIONAL RESEARCH
Katarzyna Potyrała ................................................................................................... 4
Articles
STUDENTS’ ATTITUDES TO SELECTED SUBJECTS IN SECONDARY

ECONOMIC EDUCATION
Kateřina Berková, Kristýna Krejčová, Jaromír Novák .......................................................... 9
IMPLEMENTATION OF MOBILE LEARNING USING A SOCIAL NETWORK

PLATFORM: FACEBOOK
John, G. Chaka, Irene Govender ...................................................................................24
THE RELATION BETWEEN SCHOOL CULTURE SUB-CATEGORIES AND

EXPECTED RESULTS OF LEARNING PROCESS
Ludvík Eger, Michaela Prášilová ................................................................................... 48
COMPETENCY-BASED ASSESSMENT AND THE USE OF PERFORMANCE-BASED

EVALUATION RUBRICS IN HIGHER EDUCATION: CHALLENGES
TOWARDS THE NEXT DECADE
Katherina Gallardo ................................................................................................... 61
MATHEMATICAL LITERACY PROFICIENCY DEVELOPMENT BASED ON CONTENT,

CONTEXT, AND PROCESS
Jailani Jailani, Heri Retnawati, Nidya F. Wulandari, Hasan Djidu ..............................................80
EFFECT OF LEARNING CLIMATE, THINKING PATTERN, AND CURIOSITY

ON ACADEMIC PERFORMANCE IN HIGHER EDUCATION
Ali Maksum, Fifukha D. Khory ..................................................................................... 102
A BIBLIOMETRIC ANALYSIS OF THE USE OF OPEN SOURCE SOFTWARE

IN EDUCATIONAL CONTEXTS
Orfa Nidia Patiño Toro, Yesenia Acevedo Correa, Alejandro Valencia-Arias, Martha Benjumea-Arias ... 114
Information
INSTRUCTIONS FOR AUTHORS ............................................................................. 129

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
4
A FEW REFLECTIONS ON
CONTEMPORARY SCIENCE EDUCATION
AND EDUCATIONAL RESEARCH
Katarzyna Potyrała
Pedagogical University of Krakow, Poland
E-mail: katarzyna.potyrala@up.krakow.pl

Educational tendencies are certain regularities outlined in the development of ideas
related to education and upbringing. They show some perspectives that can be predicted by
describing models, planning research on future processes as well as taking preventive actions.
Trends are more, than tendencies, related to the style of some types of research or their directions
conditioned by social demand or the specificity of cultural changes at a given time and place.
Contemporary science education tendencies are largely due to the globalization process,
technology development, multiculturalism and human impact on the natural environment.
The old slogans such as “Live globally, act locally”, “Sustainable development in practice” or
“Multicultural path to success” still seem to be a big challenge. We see a number of problems
that remain unsolved, and even grow, in the situation of rapid development of knowledge,
social change and global climate issues.
In response to various calls, local organizations, business, NGOs and other institutions
have initiated a big number of activities for natural environment. However, it seems that the
school and university do not comment them, rarely undertake joint research problems and
challenges facing education. Inclusion, social participation, augmented reality relations, out-
door science education, sustainable development, competence and metacognition can still
be found on the “top of the top” list regarding the need for educational research. It seems
that school is too overwhelmed by the progress and speed of change, teachers are obliged to
implementation a narrowly understood curriculum. Observations and interviews with teachers
show that they are more interested in the degree of acquiring programme material by students,
than thoroughly thought-out work on the development of students’ interest in creative activity,
the necessity in it, which predetermines the development of creative abilities (Uzakbaeva at
al. 2013). In general, additional activities can only be taken as part of external projects whose
beneficiary is the school community but an external institution there is a leader.
Still is current that taking into account the broad context of education, awareness of its
variability, areas of various tensions, definitely cut off the presented multi-context approach to
teaching practice from those views that consider teaching in a contextual, narrow categories of
craft, proficiency or advanced technology (Gołębniak, 1998).
For educators, the research related to their practical skills, students’ achievements and
the impact of teachers’ activities on the education of the young generation are often closer to
them. These studies are equally important, especially in the time of the educational system
reform and program changes. Particularly important are the research on the selection, structure
of the teaching and learning content, ways of transformation of university knowledge at lower
stages of education, the impact of media on teaching and learning outcomes, new ways of
evaluating the teaching process and the „competences of the future” of teachers and students
(e.g. creativity, communication). Formation of creative competence of the person is one of
the contemporary problems for both the modern society as a whole and for the individual, in
particular for the educational system (Uzakbaeva at al, 2013). Communicative competence of
https://doi.org/10.33225/pec/20.78.04 ISSN 1822-7864 (Print) ISSN 2538-7111 (Online)

Katarzyna POTYRAŁA. A few reflections on contemporary science education and educational research
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
the teachers includes effectively solving skills of social and psychological problem situations 5
in the professional and personal spheres (Yermentaeyeva at al, 2014). These skills seem
crucial today, and practice requires strengthening with responsible research in this area and the
connection with teacher training. Ngema and Lekhethothe (2019, p. 759) underline centrality
of well-qualified, competent teachers to learners’ academic success and the need for teacher
training institutions to improve their admissions selectivity and for school authorities to be
selective when appointing teachers.
In 2009, Henry Jenkins, in the report connected with the digital media and learning
(Confronting the Challenges of Participatory Culture Media Education for the 21st Century)
wrote about educational challenges related to problem solving, team work related to the tasks
and developing new knowledge, for example during Wikipedia or digital games using. He
indicated some benefits regarding these forms of participation culture, including the possibility
of using each other, diversifying cultural expression, developing skills valued in the modern
world and workplaces, and a more intense concept of citizenship. We are coming back to these
ideas today by popularizing gamification, new learning programs through games and more and
more frequent testing of such ideas during experimental classes. Some authors, from years,
see great potential in games in particular for science education, for example Honey and Hilton
(2011, p.1) suggest that ‘they enable learners to see and interact with representations of natural
phenomena that would otherwise be impossible to observe’.
Gamification in education cannot be contrary to cooperative (social) learning, it cannot
focus only on cooperation with a computer. Too often today we only feel the declarative
commitment of young people to social issues. Especially on social networks, all you have to do
is mark interest in the action or click “I like it”.
According to Jenkins, access to the participation culture functioning as a new form of
hidden curriculum, shapes the successes of young people. Some scientists say, however, that
children and youth, by achieving these key skills and competences, interact too much with
popular culture. There are some fears which suggest the need for pedagogical research, namely:
gaps and appearances of participation, unequal access to opportunities, experiences, skills and
knowledge that will prepare young people for full participation in tomorrow’s world, and the
challenges connected with learning and recognizing the ways (manipulations), through which
the media shape the perception of the world. An important issue is also ethical challenges: the
collapse of traditional forms of vocational training and socialization, as well as the traditional
norms of social life, new ethical standards, the fall of traditionally understood authorities
(teachers, parents). Jenkins (2009, p. 13) says that schools and out-of-school programs must
devote more attention to supporting what we call new media skills: cultural competences and
social skills that young people need in a new media environment. Participation culture transfers
concentration from individual expression to community involvement. New skills include
social skills developed through collaboration and they are based on knowledge and research,
technical and critical thinking skills acquired during school science education. Experts agree
that critical-thinking behaviors imply an open-minded tendency to analyze, synthesize, and
evaluate information in order to solve problems and make resolutions (Alghafri & Ismail,
2014). Working on free social networks is a great way to understand „our times” (Giordan,
2018). Giordan asks: Are there „good reasons“ to completely ban smartphones at school? [...]
Why immediately fall into fear, anxiety and even fear? And if we look at this tool again? Why
not make it a subject of study? Or maybe we will put a smartphone in the school curriculum?
Students perceive the presence of other people (friends) in their educational experience
as an important part of this experience, and they perceive the virtual environment as a natural
space for collective pro-social activities. The school should use those skills that are considered
key in a knowledge-based society, and not ignore the needs of students in learning and building
relationships (Potyrała, 2019).
ISSN 1822-7864 (Print) ISSN 2538-7111 (Online) https://doi.org/10.33225/pec/20.78.04

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
6 Modern science education, based on the idea of education for action and in action must
take into account a lot of educational approaches and contexts, the diversity of communication
practices and cooperation between people involved in the processing of scientific information
in the real and virtual world (Potyrala, 2017, p. 281)
It is still current to confront educational models described by didactics a few or
several years ago with the current educational reality. This is an interesting area of research
- educational reconstruction model (Kattmann et al.), allosteric model (Giordan), The Action
for Climate Empowerment (UNESCO 2016) and others. How do these models and guidelines
for accelerating solutions through education, training and awareness-raising work in the face
of changes in the modern world, overproduction of information, broadly understood informal
education and the dominant role of new media?
And what about neurodidactics, a relatively young discipline, some kind of bridge
between neuroscience and didactics, “didactics of the 21st century”? (Petlák, 2012). According
to Petlák (2012), many aspects of neuropedagogy and neurodidactics are very interesting
especially in the context of the effective learning but on the other hand one can remark, that
both of the neurosciences introduced to a pedagogy do not ensure an affective learning, because
it is not determined by the neuronal factors only, but other factors too. Many scientists comment
the positions represented by neurodidactics and the status of neuropedagogy, we need to take
a closer look at this in the context of in-depth interdisciplinary research. Undoubtedly, new
tools used by teams of neurodidactics allow for conducting promising pedagogical research
increasing the level of understanding of the conditions of the learning and teaching process.
We need to undertake also more research on alternative models of science and
environmental education in augmented reality and education for science culture and sustainable
development. Current research reveals numerous antinomies and, among others, opens a broader
discussion on human addiction to nature and its simultaneous destruction and even destruction.
Nature also appears as the work of a man with new technologies, but also as a creation of social
activities. All threads reveal the cultural entanglement of man in matters of nature, which he
is, and which he changes (Popczyk, 2011). The natural environment influences the shape of
civilization in a fundamental, not incidental way. In addition, creating the environment is limited,
and thus illusory - a civilization perceived as independent of natural conditions is contrary to
its own nature (Diec, 2011). Ecological awareness, combined with aesthetic sensitivity, can
help to recognize and respect nature, both intact by man and that in which man interferes. To
see beauty, you need to understand the natural relationships and relationships of man with
the environment in which he lives (Głutkowska-Polniak, 2011). Don’t let Greta Thunberg’s
message to go unnoticed by science teachers and researchers in science and environmental
education.
It is often underlined that sustainable development means a new way of thinking about
the world, and despite conceptual and political uncertainty is a concept, despite doubts about
the instruments to be adopted for its implementation, despite the difficulties with articulation
and determining the appropriate executive scales, despite the many actors and the ubiquitous
conflicts that characterize their relations, sustainable development has acquired a global
dimension that is necessary to emphasize the importance of certain contradictions. Educational
research is a basic condition for raising the level of educational practice at all levels of teaching
and learning. These studies should be some starting point for science and environmental
awareness shaping and examining educational trends and their social implications.
Let me quote the words from the preface to the study Rethinking Education Towards a
global common good? published by UNESCO in 2015 (p.3):

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
The world is changing – education must also change. Societies everywhere are undergoing 7
deep transformation, and this calls for new forms of education to foster the competencies that
societies and economies need, today and tomorrow. This means moving beyond literacy and
numeracy, to focus on learning environments and on new approaches to learning for greater
justice, social equity and global solidarity. Education must be about learning to live on a planet
under pressure. It must be about cultural literacy, on the basis of respect and equal dignity,
helping to weave together the social, economic and environmental dimensions of sustainable
development.

References
Alghafri, A. S. R., & Ismail, H. N. B. (2014). The effects of integrating creative and critical thinking on
school students’ thinking. International Journal of Social Science and Humanity, 4(6), 518-525.
https://doi.org/10.7763/IJSSH.2014.V4.410
Diec, J. (2011). Przyroda a cywilizacja [Nature and civilization]. Kultura współczesna, 2, 25-33
Kattmann, U., Duit, R., Gropengießer, H., & Komorek, M. (1997). Model rekonstrukcji dydaktycznej -
ramy dla badań naukowych i rozwoju dydaktycznego [Didactic reconstruction model - framework
for scientific research and didactic development]. Journal of Science Didactics, 3 (3), 3-18.
Giordan, A. (1996). From constructivism to allosteric learning model. In UNESCO Conference on Science
Education 2000+. https://www.andregiordan.com/apprendre/From-constructivisme-to-allosteric-
learning-model1.pdf
Giordan, A. (2018). Le smartphone, un outil d’apprentissage... Educavox: Le média des acteurs de l’Ecole
[The smartphone, a learning tool ... Educavox: The media for school stakeholders]. https://www.
educavox.fr/accueil/debats/le-smartphone-un-outil-d-apprentissage
Głutkowska-Polniak, A. (2011). Przyroda w ujęciu estetyki ekologicznej Allena Carlsona [Nature in
terms of ecological aesthetics by Allen Carlson]. Kultura Współczesna, 2, 34-42
Gołębniak, B. D. (1998). Zmiany edukacji nauczycieli: wiedza–biegłość–refleksyjność [Changes in
teacher education: knowledge – fluency – reflectivity]. Edytor.
Hilton, M., & Honey, M. A. (2011). Learning science through computer games and simulations.
Washington, DC: National Academies Press.
Fleischman, K., & Ariel, E. (2016). Gamification in science education: Gamifying learning of microscopic
processes in the laboratory. Contemporary Educational Technology, 7(2), 138-159. https://files.
eric.ed.gov/fulltext/EJ1106923.pdf
Jenkins, H. (2009). Confronting the challenges of participatory culture media education for the 21st century.
http://digilib.umpalopo.ac.id:8080/jspui/bitstream/123456789/315/1/%5BHenry_Jenkins%5D_
Confronting_the_Challenges_of_Part%28BookZZ.org%29.pdf
Ngema, M., & Lekhetho, M. (2019). Principals’ role in managing teacher professional development
through a training needs analysis. Problems of Education in the 21st Century, 77(6), 758-772.
https://doi.org/10.33225/pec/19.77.758
Petlák, E. (2012). Neuropedagogika i neurodydaktyka-tendencje XXI wieku [Neuropedagogy and
neurodidactics-trends of the 21st century]. Chowanna, 39(2), 59-66.
Popczyk, M. (2011). Przemyśleć przyrodę [Think about nature]. Kultura współczesna, 2, 21-24.
Potyrała, K. (2017). iEdukacja. Synergia nowych mediów i dydaktyki. Ewolucja, antynomie, konteksty
[iEducation. Synergy of new media and didactics. Evolution, antinomies, contexts]. Wydawnictwo
Naukowe Uniwersytetu Pedagogicznego.
Potyrała, K. (2019). Internet i uczestnictwo społeczne w świetle relacji uczeń-nauczyciel [Internet and
social participation in the light of student-teacher relations]. In J. R. Paśko (Ed.), Nauczyciel i uczeń
we współczesnej szkole [Teacher and student in contemporary school] (pp. 11-22). Wydawnictwo
Małopolskiej Wyższej Szkoły Ekonomicznej.
Rethinking education towards a global common good? (2015). UNESCO. https://unevoc.unesco.org/e-
forum/RethinkingEducation.pdf
Uzakbaeva, S., Baimukhanbetov, B., Berkimbaev, K., Mukhamedzhanov, B., & Pralieva, R. (2013). To
the problem of forming creative competence of future teachers. Creative Education, 4(03), 234-
240. http://dx.doi.org/10.4236/ce.2013.43034

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
8 Yermentaeyeva, A., Aurenova, M. D., Uaidullakyzy, E., Ayapbergenova, A., & Muldabekova, K. (2014).
Social intelligence as a condition for the development of communicative competence of the future
teachers. Procedia-Social and Behavioral Sciences, 116, 4758-4763. https://doi.org/10.1016/j.
sbspro.2014.01.1021
Received: December 22, 2019 Accepted: January 30, 2020
Cite as: Potyrała, K. (2020). A few reflections on contemporary science education and
educational research. Problems of Education in the 21st Century, 78(1), 4-8. https://doi.
org/10.33225/pec/20.78.04
Katarzyna Potyrała Dr hab., Professor, Head of Department of Didactics and Methodology of

Educational Research, Institute of Educational Sciences, Pedagogical University
of Krakow, ul Ingardena 4, 30-060 Krakow, Poland.
E-mail: katarzyna.potyrala@up.krakow.pl
Website: https://scholar.google.com/citations?user=y5ogVosAAAAJ&hl=pl

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
STUDENTS’ ATTITUDES TO SELECTED 9
SUBJECTS IN SECONDARY
ECONOMIC EDUCATION
Kateřina Berková, Kristýna Krejčová
University of Economics, Czech Republic
E-mail: katerina.berkova@vse.cz, kris.krejcova@gmail.com
Jaromír Novák
University of Economics in Bratislava, Slovakia
E-mail: jaromir.novak@euba.sk
Abstract
Students’ motivation is one of the key factors that determine their school success. It is closely linked
to their attitudes toward individual school subjects. This research analyzes four crucial dimensions
of students’ subject-related attitudes (practical importance, difficulty, popularity and curriculum
contents attractiveness) in the secondary economic education in the Czech Republic and Slovakia in the
international context. The research focused on the four dimensions of students’ attitudes (n = 573) toward
four major school subjects (economics, accounting, mathematics and the mother tongue) at business
academies. It analyzes the data from the questionnaires using the Spearman Correlation Coefficient, the
non-parametric Kruskal-Wallis test and the modification of Tukey’s method for multiple comparisons. The
results show that the popularity of a school subject significantly correlates with its curriculum contents
attractiveness and practical importance. However, the difficulty of school subjects did not influence their
popularity in a negative way, which opens several questions for further research in the area.
Keywords: economic education; students’ attitudes; motivation in economic school subjects; school
subject popularity; school subject difficulty.
Introduction
Educational processes at all levels and types of school have one general goal – to maximize
school success and educational attainment of an individual. Students’ general cognitive abilities,
general motivational factors and special, subject-related interests are the main determinants of
their potential to prosper in these processes (Schiefele, Krapp, & Winteler, 1992). Two previous
studies (Berková & Krejčová, 2016; Berková, Krejčová, Králová, Krpálek, Krpálková Krelová,
& Kolářová, 2018) examined the role of cognitive abilities and their relation to motivation,
learning outcomes and attitudes to a teacher. According to their findings, the motivation of the
average students increases thanks to their teachers’ expertise, their ability to develop thinking
and exposition to the curriculum, whereas the above-average students perceived only teachers’
expertise and exposition to the curriculum as motivational. The research in the previous article
provided the basis for the current research, which deals especially with students´ interests and
subject-related attitudes.
General definitions of motivation refer to the sum of processes or dynamism that energize
and target behavior of an individual to reach something desirable or avoid something undesirable
(Plháková, 2010). Ceauşu (2018) stated that motivation was a result of the encounter between
the biologic and the social and operated as a dynamic and plastic system providing social-
adaptive program of the subject. An individual specification of motivational processes arises
from the unique structure of human needs (Maslow, 1954).

Kateřina BERKOVÁ, Kristýna KREJČOVÁ, Jaromír NOVÁK. Students’ attitudes to selected subjects in secondary economic education
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
10 Considering practical implications of the theoretical construct, motivation is observable

in a form of specific relations to various dimensions of reality. In educational processes, the
subject-related interests interact with emotional phenomena such as a sense of school belonging
(Ferreira, Cardosob, & Abrantesc, 2011) and are transformed into self-regulative processes that
significantly modify learning efficiency (Popa, 2015). The concept of academic self-efficacy
(Pajares, 1996) expressed the interplay of all these processes. Reflecting the importance of self-
view for students´ academic success, the research presented in this article builds on students’
self-reflective statements. Tossavainen and Juvonen (2015) used this research approach when
they examined the motivation of primary and secondary school students to the subjects of
mathematics and music. They described the students’ learning motivation with respect to their
enjoyment of the subjects and their perceptions about the importance and usefulness of the
subjects. Their research also investigated the respondents’ view of the difficulty of the subjects,
the popularity of the subjects, the students’ self-competence in the subjects, and their gender.
The research revealed a relation between the students’ attitudes to the subject of mathematics
and their learning motivation. The research of Huber and Seidel (2018) also supported this
approach to the research of motivation to learn. The researchers concluded that teachers tended
to overlook the diversity of students’ attitudes, perceiving them mostly in simplistic, general
categories.
There are close relations between students’ attitudes to the subjects they study and their
learning motivation. Attitudes to a subject and the overall level of motivation may determine
learning outcomes more intensively than the level of intelligence even in the education of
math (Murayama, Pekrun, Lichtenfeld, & Vom Hofe, 2013). Therefore, an extreme set of mind
such as math anxiety may have a distortive impact on educational attainment (Luttenberger,
Wimmer, & Paechter, 2018). Tossavainen and Juvonen (2015) confirmed in their research that
the students’ attitudes to one subject (math) can affect attitudes to another subject (music). Their
methodology and research results were inspirational for the current research, which examines
the relations between the students’ attitudes toward the chosen economic and non-economic
subjects.
Subject-related attitudes are largely influenced by the approach of a teacher who modulates
students’ behavior engagement and thus influences learning outcomes (Leon, Medina-Garrido,
& Núñez, 2017), using specific techniques such as deep reasoning questions and feedback
(Jurik, Gröschner, & Seidel, 2014). The study by Prokop, Tuncer, and Chudá (2007) found
that a teacher´s identity also strongly affects the students’ attitudes toward the teacher. Adu,
Galloway, and Olaoye (2014) did research in the field of economic education and proved that
students’ perception of their teachers influences their attitude toward economics.
Identification and measuring the influence of students’ attitudes to their school
subjects, their teachers and the overall educational process have become an important part of
educational research. Related studies usually focus on subject-specific areas of teaching and
learning at schools. For example, several studies examined students’ attitudes toward science
(school subjects such as biology, geography and so like), in the Czech and Slovak educational
environments (Kubiatko, Janko, & Mrázková, 2012; Prokop, Tuncer, & Chudá, 2007). The
results of the former research showed that the popularity of geography in the Czech Republic,
as well as the year of study, significantly influenced the perception of the subject. Other studies
stressed the link between the perception of a subject and usefulness of the acquired knowledge
for students’ lives etc.
Many researchers have studied learning motivation as well as students’ attitudes towards
school subjects and their school success. However, they are still the subject of discussions and
their relations are not clear. The OECD study (2003, p. 81) states: “it can be argued that attitudes
should relate to achievement, an interesting question of causality arises here because it is not
obvious if success in learning engenders better attitudes or if a positive attitude to school is

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
effectively a factor in motivating students to learn.” According to PISA results (OECD, 2003), 11
the students may be motivated to study mathematics by its perceived importance to future
education or to careers. The current empirical research builds on these results and extends the
methodology to finding relations between the chosen economic and non-economic subjects.
The current empirical research is based on PISA results (OECD, 2012), which show
students’ solidarity to their schools throughout 28 countries around the whole world. The Czech
Republic and Slovakia prove to be significantly under the OECD average. The PISA results
were also used in a comparative study of Bruner, Keller, Wenger, Fischbach, and Lüdtke (2018)
dealing with domain-general and domain-specific measures of achievement, learning strategies,
engagement and motivation. Having compared responses of 15-year-old students at schools
from 81 countries, they found substantial differences in achievement. Interestingly, differences
in learning strategies, motivation and engagement were less significant. We can presume that
research conclusions about motivation tend to have a more general validity than conclusions
about achievement. Therefore, the research into students’ attitudes to school subjects and their
influence on the students´ motivation to learn can bring useful recommendations for pedagogical
practice.
This research focuses primarily on students’ attitudes toward two major economic
subjects (economics and accounting) studied at business academies (i. e. the most common
type of school in economic secondary education in both the Czech Republic and Slovakia) in
comparison with their attitudes to two other major non-economic subjects (mathematics and
the mother tongue) studied at the same schools. Hausmannová (2015) researched some aspects
of students’ attitudes to major economic-related school subjects at business academies in the
Czech Republic on a sample of 1,733 respondents. Economics proved to be a more popular
subject than accounting, and it was regarded as considerably less difficult and slightly more
important for the students´ future careers than accounting. The methodology of this research
builds on the research by Hrabal and Pavelková (2010), who suggested using six dimensions
for measuring students’ attitudes, half of which examined students’ perception of the evaluated
subject and the other half reflected selected aspects of students’ self-evaluation.
Research Aim
The current research deals with students’ interests and subject-related attitudes. The
aim of the research was to find how students perceive the subjects of economics, accounting,
mathematics and their mother tongue (i.e. the Czech and Slovak languages) as regards their
practical importance, popularity, levels of difficulty and attractiveness of their curriculum
contents. Based on the aforementioned approach to the research problem, the correlations
between these four dimensions of students’ attitudes for each of the subjects were analyzed
separately. Furthermore, the similarities and differences in the dimensions of students’ attitudes
between the selected subjects were identified.
Research Hypotheses
The hypotheses that result from the aim of the research were as follows:
1. There is a positive dependency between popularity, importance and curriculum
contents attractiveness of each subject.
2. A negative dependence proves to exist between difficulty and other dimensions of
the students’ attitude (popularity, importance and curriculum contents attractiveness)
to each selected subject.
3. Students perceive the selected subjects – as regards the monitored dimensions of
their attitudes – differently.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
12 Research Methodology
General Background
This empirical research was a partial section of the comprehensive research which
focused on the attitude of secondary school students to the selected subjects in study programs
related to economics as well as the factors influencing students’ motivation in connection with
the selected subjects: economics; accounting; mathematics; and the mother tongue (the Czech
or Slovak languages, which are very similar regarding both grammar and vocabulary). The
examples of good practice showed that students had very similar feelings for mathematics and
accounting and their school success was very similar in both of these subjects. Given that
economics and accounting are the main graduation subjects at business academies, they are both
very important for the attempts to improve the quality of teaching and learning and to prepare
the students for the final examinations at this type of school. The empirical research examined
whether the students’ attitudes to the subjects were independent or whether there were some
links between the students’ attitudes towards the subjects (Tossavainen & Juvonen, 2015). A
deeper knowledge of these contexts is very helpful for improving the processes of teaching and
learning in these economic subjects, for which a certain level of knowledge of mathematics
and the mother tongue is vital. Accounting uses many mathematical operations and forms a
logical system. Both accounting and economics use specific terminology, so it is important that
students have a good command of their mother tongue in order for them to understand and be
able to discuss the subject matter correctly. For the above reasons, the combination of these four
subjects was chosen for the analyses.
The research was based on two comprehensive types of research conducted in the
Czech Republic (Hrabal & Pavelková, 2010; Hausmannová, 2015), the results of which may
be, considering population, regarded as generally valid; they pertain to students’ attitudes to
individual subjects both on the general level and in the context of study programs related to
economics. Their approach to the research problem inspired the chosen research design as well.
Sample
The research comprised secondary schools from the Czech Republic and Slovakia,
which guarantee and provide high-quality economic education within the scope of a study
program at a business academy. The sample of secondary schools was created on the basis
of a premeditated selection whose main criterion was a close cooperation of the schools with
the two main scientific workplaces in the area – the University of Economics in Prague and
the University of Economics in Bratislava. The respective cooperation pertains to provision
of pedagogical internships to future teachers – students of the study program of Teaching of
Economic Subjects. Therefore, the teachers in these schools are supposed to approach the
educational process in an innovative manner and exploit not only traditional but also modern
teaching methods. They should show the university students examples of good teaching
practice and base the educational process more on experience-oriented, active learning. The
aforementioned educational requirements are further reinforced with respect to the changing
strategy of educational policies in both countries. They reflect contemporary evolution of the
educational contents and ways of teaching and learning. As regards secondary school students,
their education is supposed primarily to develop competencies required for their active civil,
professional and personal lives. This should lead to the increasing students’ interest in the
study field through formation of positive attitudes to the key subjects of the study program and
reaching the educational objectives and related competencies in an efficient, targeted manner.
It has been proven (Berková, Krejčová, Králová, Krpálek, Krpálková Krelová, & Kolářová,

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
2018) that the students of business academies with higher motivation can develop their higher 13
cognitive skills better.
The total number of 7 secondary schools from the capital cities of both countries (Prague
and Bratislava) participated in the research. The main research comprised 577 students in their
last (fourth) or penultimate (third) years of study. The number of valid observations was 573,
which was due to some failures to answer questions and provision of ambiguous answers.
Therefore, 4 respondents were excluded from the overall research sample. The sample of
students was also created on the basis of intentional selection. This method was applied in order
to obtain answers from students who already have enough experience with economic education,
thus being able to evaluate their attitude to selected subjects in economic education more
impartially. Representation of the students from the Czech Republic is lower and constitutes
28.6% of the sample. The Slovak students represent 71.4% of the sample. The research sample
consists of 392 women (i.e. 68.4%) and 181 men (i.e. 31.6%). The respondents, who were
third-year and fourth-year students of secondary schools, were 17 – 19 years old. The data were
analyzed comprehensively, for the sample as a whole. The results from both countries were
evaluated and processed together, considering the almost identical educational procedures and
objectives in the respective area. For the structure of the research sample, see Table 1.
Table 1
Structure of the research sample
Group Absolute frequency Relative frequency (%)

3rd year, Czech Republic 83 14.5
4th year, Czech Republic 81 14.1
3rd year, Slovakia 206 36.0
4th year, Slovakia 203 35.4
Total 573 100
Instrument and Procedures
The research was conducted using the questioning method. As the research was
quantitative, the questionnaire for the respective purpose was created based upon two
questionnaires previously used for similar purposes (Berková, Krejčová, Králová, Krpálek,
Krpálková Krelová, & Kolářová, 2018; Berková, Borůvková, & Lízalová, 2018). It was
distributed to the secondary schools and among the students in the written form from April
to June 2018. The collection of data at each of the selected schools was performed by the
researchers in person or with the assistance of the secondary school teachers. This approach
guaranteed a higher rate of return. It was recommended that the students should respond
as impartially as possible, specifying that the collected data were subject to a high level of
confidentiality and anonymity while being used solely for the publishing of research papers.
The questionnaire contained five main sections pertaining to the (a) student’s general
motivation within the scope of economic education; (b) student’s attitudes to the teacher; (c)
student’s attitudes to the selected subjects (economics, accounting, mathematics and the mother
tongue); (d) time necessary for the student’s preparation at home; and (e) demographic data
about the respondent (i.e. gender, secondary school, grades from the selected subjects, interest
in the chosen study field, and factors influencing the student’s choice of secondary school). The
questionnaire was not newly developed, but it was based on the structure of the questionnaires
used in the previous two research studies in the areas of economics-related secondary education
(Berková, Krejčová, Králová, Krpálek, Krpálková Krelová, & Kolářová, 2018) and tertiary
education (Berková, Borůvková, & Lízalová, 2018). For this reason, no validation and feasibility

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
14 tests were performed. The reliability of the questionnaire was verified in the pilot phase. Due
to the fact that the research is extensive and comprises a high number of variables, this paper
focuses solely on a single part of the research – student’s attitudes to the selected subjects and
search for the interrelations between the attitudes and their dimensions.
The following four dimensions of students’ attitudes were determined: practical
importance of the subject; popularity of the subject; attractiveness of the curriculum contents
of the subject; and its perceived level of difficulty. The students evaluated these dimensions of
their attitudes not only for the two major subjects in their economic education, i.e. economics
and accounting. They also expressed their opinions about two major non-economic subjects
in which it was possible to find similarities with the two economic subjects, i.e. mathematics
and the mother tongue (Czech, Slovak). mathematics was selected considering the fact that
accounting represents a discipline based mainly on the logical system exploiting mathematical
algorithms. On the contrary, economics is a socio-scientific discipline based more on verbal
logic and description of economic phenomena, so it could be presumed that students’ attitudes
to this subject might be related to their attitudes toward their mother tongue. Both of these
subjects are significantly influenced by the students’ literacy, ability to understand texts, search
for information and work with it.
The data representing ordinal variables as the measurement of attitudes was based on
the scale featuring values between –2 and 2. The value of –2 represented the lowest intensity
of attitude while 2 represented the highest intensity of a student’s attitude to the subject and
its dimensions. As regards the practical importance, popularity and curriculum contents
attractiveness of a subject, a higher value represents a more positive evaluation presented by a
student. In case of the difficulty of a subject, a higher value represents a higher level of demand,
i.e. a more negative attitude.
Data Analysis

The obtained data were analyzed using the Spearman Correlation Coefficient (r). This
test was applied with respect to the unproven normality of data and nature of the applicable
variables. The above-mentioned test allowed for the determination of relations between
students’ attitudes to each of the selected subjects.
The analysis of similarities/differences within and between the two pairs of the subjects
(economics versus the mother tongue and accounting versus mathematics) was performed
with respect to the fact that data normality had not been proven using the non-parametric
Kruskal-Wallis test. A paired comparison test (a modification of Tukey’s method for multiple
comparisons) – the Tukey-Kramer Procedure – was used in order to determine the subjects in
which students’ attitudes showed similarities and differences.
Null (statistical) hypotheses were defined as follows:
1. No positive dependence proves to exist between popularity, importance and
attractiveness of each subject.
2. No negative dependence proves to exist between difficulty and other dimensions of
attitudes (popularity, importance and attractiveness) to each selected subject.
3. No differences prove to exist in the students’ perception of the selected subjects as
regards the analyzed attitudes.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Research Results 15
Correlation of the Importance, Popularity and Attractiveness in Case of Each Subject

according to the Students’ Attitudes
The Spearman Correlation Coefficient revealed correlations between the three dimensions
of the students’ attitudes to the selected subjects of economics, accounting, mathematics and the
mother tongue (the Czech and Slovak languages).
As regards economics, the research revealed a positive correlation of practical importance,
popularity and attractiveness of the curriculum contents – at the significance level of 5%. This
means that if the subject of economics features practical importance for students, it is also
attractive – and vice versa.
In the case of economics, the strength of the positive correlation was as follows:
popularity – attractiveness (r = .67; p < .01); practical importance – popularity (r = .529; p <
.01); practical importance – attractiveness (r = .478; p < .01). The detailed results are presented
in Table 2.
Table 2
Correlation of the attitudes to economics
Importance Popularity Difficulty Attractiveness
p rs p rs p rs p rs
Importance .001 .529 .720 -.015 .001 .478
Popularity .001 .529 .0356 -.088 .001 .670
Difficulty .720 -.015 .0356 -.088 .509 -.028
Attractiveness .001 .478 .001 .670 .509 -.028
In the case of accounting, the research revealed a positive correlation of reviewed

attitudes – practical importance, popularity and attractiveness of the curriculum contents – at the
significance level of 5%. The research proved that if the subject of accounting is popular with
secondary school students, it also has practical importance and attractiveness, which applies in
any given order – like in the case of economics. However, the correlations are stronger than
in the case of economics, as documented by the following values of the Spearman Correlation
Coefficient for popularity – attractiveness (r = .75; p < .01); practical importance – popularity
(r = .595; p < .01); practical importance – attractiveness (r = .628; p < .01). For more details,
see Table 3.
Table 3
Correlation of the attitudes to accounting

p rs p rs p rs p rs
Importance .001 .595 .265 .047 .001 .628
Popularity .001 .595 .770 -.012 .001 .750
Difficulty .265 .047 .770 -.012 .413 .413
Attractiveness .001 .628 .001 .750 .413 .413

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
16 As regards mathematics, the research proved – at the significance level of 5% – major

interrelations between: popularity – attractiveness (r = .735; p < .01); practical importance –
popularity (r = .638; p < .01); practical importance – attractiveness (r = .655; p < .01). It may
be stated that if mathematics is popular with students, it is also considered to be attractive and
important for their practical life. This applies in any order. For more details, see Table 4.
Table 4
Correlation of the attitudes to mathematics
p rs p rs p rs p rs
Importance .001 .638 .0003 .150 .001 .655
Popularity .001 .638 .040 .086 .001 .735
Difficulty .0003 .150 .040 .086 .004 .118
Attractiveness .001 .655 .001 .735 .004 .118
As regards the mother tongue, the following correlations prove to be generally valid at
the significance level of 5%: popularity – attractiveness (r = .67; p < .01); practical importance –
popularity (r = .494; p < .01); practical importance – attractiveness (r = .471; p < .01). This
case also documented the fact that if the mother tongue is considered to be a popular subject,
it is also attractive for the students and they perceive its curriculum contents as practically
important. For more details, see Table 5.
Table 5
Correlation of the attitudes to the mother tongue
p rs p rs p rs p rs
Importance .001 .494 .001 .133 .001 .471
Popularity .001 .494 .009 .109 .001 .670
Difficulty .001 .133 .009 .109 .001 .064
Attractiveness .001 .471 .001 .670 .001 .064
The strongest correlation of the analyzed dimensions of students’ attitudes was found for
the pair of popularity and curriculum attractiveness of each subject. At the significance level of
5% we reject the null hypothesis 1 on the non-existence of positive dependence of popularity,
practical importance and attractiveness in case of each subject.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Correlation of the Difficulty with Other Dimensions of the Students’ Attitudes (Importance, 17
Popularity and Attractiveness) in Case of Each Subject
In the case of economics, the negative correlation was proven (at the significance level
of 5%) with a single pair of reviewed attitudes – difficulty and popularity (p = .035). As regards
interpretation of the significance of relations based on the p-value, it may be generally stated
that if the subject is difficult for students, it is less popular (and vice versa). However, the value
of the correlation coefficient is very low; it does not reach even the lowest required value of .4
(r = –.088). For the detailed results of p-values and correlation coefficients, see Table 2.
In the case of accounting, no negative correlation between difficulty and other dimensions
of the students’ attitudes to the subject was proven at the significance level of 5%. For more
details (i.e. p-values of individual attitudes and correlation coefficients), see Table 3.
The subject of mathematics – unlike both economic subjects – features completely
different results. At the significance level of 5% the research proved the correlation of the
difficulty of the subject with the other attitudes – its practical importance (p = .0003); popularity
of the subject (p = .04); and attractiveness of the curriculum contents of the subject (p = .004).
The Spearman Correlation Coefficient was used to determine a positive correlation between
the difficulty of the subject of mathematics and the other dimensions of attitudes. However,
the correlation coefficient values are very low and do not reach the required minimum value of
.4 – in case of the following combinations of the analyzed dimensions of students’ attitudes: (a)
difficulty – practical importance (r = .15); (b) difficulty – popularity (r = .086); (c) difficulty –
attractiveness (r = .118). For the detailed results, see Table 4. It is, however, to be stated that a
rather surprising result pertains to the positive correlation between the difficulty of the subject
of mathematics and the other dimensions of attitudes toward this subject – unlike the subject
of economics in case of which a negative correlation was determined between its difficulty
and popularity (even though it was very small). The positive correlation means that even if the
subject of mathematics is difficult for students, it is popular, attractive and practically important
at the same time.
In the case of the mother tongues (the Czech and Slovak languages), the research proved –
at the significance level of 5% – certain correlations between difficulty and the other dimensions
of attitudes, i.e. difficulty and practical importance of the subject (p = .001); difficulty and
popularity of the subject (p = .009); and difficulty and attractiveness of the curriculum contents
of the subject (p < .01). The Spearman Correlation Coefficient showed positive correlations –
like in the case of mathematics. However, the values of correlation coefficients are very low
and, once again, they do not reach the minimum value of .4, in case of which the result might
be considered as generally valid. The values for individual combinations of the dimensions of
students’ attitudes were as follows: (a) difficulty and practical importance of the subject (r =
.133); (b) difficulty and popularity (r = .109); (c) difficulty and attractiveness (r = .064). For
more details, see Table 5.
In the case of the mother tongue, the research also revealed positive correlation of its
difficulty with the other dimensions of attitudes. It means that if students perceive the mother
tongue as a difficult school subject, it is also popular and practically important. As regards both
mathematics and the mother tongue, the aforementioned result might be explained by the fact
that these subjects are generally important for everyone. Students are aware of the fact that
they will need both these subjects – to a certain extent – not only for their professional careers,
but also in their everyday lives. Therefore, they might associate them with the importance
which reinforces the difficulty of these subjects. This importance and difficulty are given by the
depth of the curriculum contents, as these subjects are not discussed only on a superficial level.
Furthermore, it might result from the fact that these subjects already form a part of primary
education, so students are already familiar with their fundamentals and might regard the present

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
18 teaching level as difficult (they can see how much progress they made, and what the subjects
actually comprise, etc.). Practical importance might also be emphasized by the fact that the
school-leaving examination in the mother tongue is compulsory and a compulsory examination
in mathematics is being planned; furthermore, these subjects are taught in all types of primary
and secondary schools. This proves to be contrary to economics (in this case, the negative
correlation was proven for the pair of difficulty and popularity of the subject), which might
be related to the position of the discipline in the society. Based on the aforementioned facts, it
might be stated that in case of mathematics and the mother tongue students show more tolerance
to their difficulty than in case of economics (due to their position in the society).
At the 5% level of significance, the null hypothesis 2 was not rejected. The negative
dependence was found between the difficulty and popularity of the subject of economics.
Similarities and Differences of the Selected Subjects with Respect to the

Analyzed Dimensions of the Students’ Attitudes
Firstly, a review of significant differences between subjects based on each analyzed

dimension of students’ attitudes was conducted on the level of n = 573. The results document
the fact that the students evaluated the subjects differently with respect to their practical
importance, popularity, difficulty and curriculum contents attractiveness; nevertheless, certain
similarities of students’ attitudes toward the subjects were found as well.
The differences between the subjects with respect to the students’ attitudes (p < .01) were
found on the general level – at the significance level of 5%. In addition, the differences between
the subjects applicable to the partial dimensions of students’ attitudes were determined using
mean values. For the respective results, see below:
The students perceive the practical importance of the subject differently for mathematics
and the mother tongue. The economics-related subjects differ from non-economic subjects,
but without differences in importance between them. At the significance level of 5%, students
perceive the practical importance of economics (mean value: .923) and accounting (mean
value: .777) identically while the mother tongue (mean value: .335) and mathematics (mean
value: –.754) are perceived completely differently. Furthermore, students regard economics as
the most practically important subject while mathematics ranks last. Even though mathematics
is – in comparison with the other subjects – evaluated as the least important for the economic
practice, the difficulty of this subject is, by contrast, in a positive correlation with its popularity
and practical importance; however, this correlation value is very low. That might be related to
the position of this subject in people’s everyday lives, their motivations and set objectives for
the future (i.e. students expect more extensive and deeper knowledge in mathematics than in
economics-related subjects). On the other hand, in the context of economic practice, students
do not perceive the subject of mathematics as so significant (to the same extent) as economics-
related subjects.
The students perceived the popularity of the subjects differently within the group of
the economic subjects, i. e. there was a difference between economics (mean value: .377) and
accounting (mean value: .187). The students regard economics as more popular than accounting.
In addition, the popularity of economics also differs from mathematics (mean value: –.729)
and the mother tongue (mean value: .101). As regards the subject of accounting, a difference
was determined solely in comparison with the subject of mathematics. Similarly, a difference
in popularity was revealed between mathematics and the mother tongue. The similarity was
proven for the popularity of the subjects of accounting and the mother tongue. These two
subjects are on the identical level of students’ popularity.
The students perceive the difficulty differently between mathematics and the other
subjects. The students regard mathematics as the least difficult (mean value: –.268). Considering

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
the aspect of statistical significance, the research proved similarity between the economics- 19
related subjects; according to the mean values, economics (mean value: .094) proves to be
less difficult for the students than the subject of accounting (mean value: .157). Furthermore,
the research revealed a similarity in the students´ attitudes between economics and the mother
tongue (mean value: –.024). It may be stated that there is parity between both of the economics-
related subjects, and between the subject of economics and the mother tongue. As regards its
difficulty, the subject of mathematics proves to differ from the other subjects.
As for the attractiveness of the curriculum contents, all of the analyzed subjects are
perceived differently. This documents the fact that each subject, i.e. economics (mean value:
.490), accounting (mean value: .20), mathematics (mean value: –.010), and the mother tongue
(mean value: –.08), has a different level of interest for the students. The most interesting subject
is economics. For a comprehensive review of correlations between the subjects by mean values,
see Table 6.
Table 6
Rank of the subjects according to the students’ attitudes
Rank Importance Popularity Difficulty Attractiveness
1. Economics Economics Economics Economics
2. Accounting Accounting Accounting Accounting
3. Mother tongue Mother tongue Mother tongue Mother tongue
4. Mathematics Mathematics Mathematics Mathematics
At the 5% level of significance, the null hypothesis 3 was not rejected. The differences
were found in the case of the attractiveness of the curriculum content. As regards the curriculum
contents attractiveness, the students perceived the subjects rather differently.
Discussion
The aim of this empirical research was to find how students perceived the subjects
of economics, accounting, mathematics and their mother tongue (i.e. the Czech and Slovak
languages) as regards their practical importance, popularity, levels of difficulty and attractiveness
of their curriculum contents. The research primarily focused on the analysis of the correlations
between these four dimensions of the students’ attitudes toward each of the subjects separately.
Furthermore, it identified the similarities and the differences in the dimensions of the students’
attitudes between the selected subjects.
The practical importance, popularity and curriculum contents attractiveness within
each individual subject feature positive correlations – as regards the subjects of economics,
accounting, mathematics and the mother tongue. The correlations are the strongest in the cases of
mathematics and accounting, followed by economics and the mother tongue. The difficulty does
not correlate with these dimensions of students’ attitudes in the case of accounting. As regards
economics, its difficulty features a very low level of negative correlation with its popularity;
however, the respective correlation is very weak. On the contrary, a positive correlation between
difficulty and the other attitudes was determined in the cases of mathematics and the mother
tongue. Nevertheless, the correlation values were very low. The subject of economics proved
to be the most practically important, popular and attractive for the students. The similarity
between economics-related subjects was significant as regards their practical importance and

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
20 difficulty. Considering popularity, a certain similarity was identified between the subjects of
accounting and the mother tongue. As for difficulty, a certain similarity was identified between
the students’ attitudes toward the mother tongue and economics. Considering attractiveness, all
the subjects proved to differ from one another significantly.
Some of these findings might be considered as quite surprising. Greater difficulty of
a school subject could be supposed to lead to its decreasing popularity among students, and
vice versa. The results of the research show that this does not necessarily have to be the case,
definitely not with our respondents. Hrabal and Pavelková (2010) even argue against some
teachers’ attempts to make their subject more attractive and popular with students by lowering
the requirements in terms of the set objectives and learning outcomes, or on the contrary, warn
the teachers not to demotivate students by making their subject too difficult. In this respect, it
could be considered as a desired state to find that students do not automatically regard difficulty
and popularity of a subject as opposites. We agree with Hausmannová (2015) that if teachers
set favorable conditions in the process of teaching and learning, students can regard even most
difficult tasks as relatively easy to accomplish. We can conclude that it is of vital importance
for teachers to reach such a level of difficulty of their subjects that is appropriate for their
students (i.e. the highest possible level of difficulty that cannot be increased any more without
a considerable decrease in students’ motivation). This is also in line with the facts found out by
Latham and Locke (1991). In that case, students will regard even a relatively difficult subject
as popular and attractive. Whether the students regard their subjects as popular and attractive
is linked with their motivation (Tossavainen & Juvonen, 2015). These relations were also
examined by PISA (OECD, 2003). In these aspects, a new question arises to verify the link
between the popularity of the subject and the students´ motivation to learn. In this respect, this
current research brings new knowledge and opens up new research questions.
An overall preference of economics was proven also by the research of Hausmannová
(2015). Economics was perceived as more popular, less difficult and slightly more important
than accounting. The researcher proved negative dependency between the popularity and the
difficulty of both these subjects. In this current research, this relation was confirmed only
in the case of economics; what is more, with a very low value of the correlation coefficient.
The students’ perception of the popularity and difficulty of a school subject does not have to
be contradictory if a teacher sets a level of difficulty that is appropriate in terms of students’
knowledge and skills, and provides the class with adequate conditions of teaching and learning.
Considering the inner factors of school success, the subject-specific interests interplay
with general motivational variables and cognitive abilities (Schiefele, Krapp, & Winteler,
1992). This current research revealed relations between practical importance, popularity and
curriculum contents attractiveness. This conclusion leads to a recommendation for teachers to
show practical importance of the contents of their subjects to their students. In this way, the teacher
may significantly influence the subject’s popularity and attractiveness. This interconnectedness
of a teacher’s impact on students’ subject interests via stimulating behavioral engagement was
described by Leon, Medina-Garrido, and Núñez (2017) and Berková and Krejčová (2016).
A comparison of the results of the studies dealing with students’ attitudes and a teacher’s
influence on them leads to a remarkable paradox. Although the motivational factors may affect
learning outcomes more than a level of intelligence even in Math (Muryama et al., 2015),
teachers tend to overview a diversity of students’ attitudes, perceiving them too simplified
(Huber & Seidel, 2018). Considering these relations, it is not surprising that educational
outcomes may be radically transformed by undesirable psychological states such as math anxiety
(Luttenberger, Wimmer, & Paechter, 2018). The research depicted important dimensions of
subject-related attitudes that should be reflected by teachers. Moreover, the findings of specific
relations between the attitudes toward different subjects (e.g. the similarity of accounting and
the mother tongue in terms of popularity, the similarity of economic-related subjects regarding
their practical importance and difficulty) may also be used in the area of career counselling.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
The novelty of the research presented in this article was in dividing students’ attitudes 21
toward school subjects into four categories. The main contribution of the research can be seen
especially in the context of the study plans for the students in economic study fields. However,
the results of the current research may be transferred to further research of students´ attitudes
and motivation in other study fields as well.
Conclusions
The research analyzed four important dimensions of students’ attitudes to the selected
school subjects and their interrelations. The research results proved that the popularity of a
school subject, which is one of the crucial factors of students’ motivation influencing their
tendency to deep learning significantly correlates with its curriculum contents attractiveness
and practical importance. However, it does not necessarily have to be negatively influenced
by the difficulty of the subject although several previous research studies showed that it often
happens. Therefore, further research should focus more on the main factors that influence how
students perceive the difficulty of a school subject. It might be interesting to analyze to what
extent it is dependent on the level of requirements set by the teacher or how the difficulty of a
school subject is influenced by the way they perceive the difficulty of other subjects. Another
aspect worth analyzing is to what extent a teacher can influence the perceived difficulty of a
subject by creating a stimulating learning environment, or by showing students the practical
importance of the subject matter. It might also be interesting to research whether the perceived
difficulty of a school subject influences the students’ results in the subject and vice versa.
Positive students´ attitudes to school subjects stimulate their motivation to study the
subject matter more deeply. Therefore, it seems important for teachers to pay closer attention to
their students´ attitudes and their analysis with a view to influencing them with their choice of
appropriate learning outcomes and attractive curriculum contents as well as suitable teaching
and learning activities. Together with creating a supportive learning environment in class, they
can positively influence their students’ attitudes, thus increasing their learning motivation,
which will finally lead to their better school results.
Acknowledgements
This research was supported by the Project No. IGS VŠE F1/7/2018, the Project VŠE
No. IP 100040 and the Project KEGA No. 005EU-4/2019.
References
Adu, E. O., Galloway, G., & Olaoye, O. (2014). Teachers’ characteristics and students’ attitude towards
economics in secondary schools: Students’ perspectives. Mediterranean Journal of Social
Sciences, 5(16), 455–462. https://doi.org/10.5901/mjss.2014.v5n16p455
Berková K., Borůvková J., & Lízalová L. (2018). Motivation of students of economic and technical study
programmes as a tool of competitiveness of universities and colleges: Empirical study. Journal
on Efficiency and Responsibility in Education and Science, 11(4), 72–77. https://doi.org/10.7160/
eriesj.2018.110401
Berková, K., Krejčová, K., Králová, A., Krpálek, P., Krpálková Krelová, K., & Kolářová, D. (2018).
The Conceptual four-sector model of development of the cognitive process dimensions in
abstract visual thinking. Problems of Education in the 21st Century, 76(2), 121–139. http://www.
scientiasocialis.lt/pec/node/1116
Berková, K., & Krejčová, K. (2016). Effect of teachers’ abilities on students’ motivation with varying
levels of intellectual abilities in the economics. Journal on Efficiency and Responsibility in
Education and Science, 9(3), 81–87. https://doi.org/10.7160/eriesj.2016.090304

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
22 Brunner, K., Keller, U., Wenger, M., Fischbach, A., & Lüdtke, O. (2018). Between-school variation in
students’ achievement, motivation, affect, and learning strategies: Results from 81 countries for
planning group-randomized trials in education. Journal of Research on Educational Effectiveness,
11(3), 452–478. https://doi.org/10.1080/19345747.2017.1375584.
Ceauşu, F. (2018). Effects of motivation on the learning activity. Revista Romaneasca pentru Educatie
Multidimensionala, 10(2), 6–27. https://doi.org/10.18662/rrem/42
Ferreira, M., Cardosob, A. P., & Abrantesc, J. L. (2011). Motivation and relationship of the student
with the school as factors involved in the perceived learning. Procedia – Social and Behavioral
Sciences, 29, 1707–1714. https://doi.org/10.1016/j.sbspro.2011.11.416
Hausmannová, H. (2015). Postoje žáků obchodních akademií k ekonomickým předmětům: diplomová
práce. [Business academy students´ attitudes toward economic subjects: diploma thesis]. Prague:
Vysoká škola ekonomická v Praze.
Hrabal, P., & Pavelková, I. (2010). Jaký jsem učitel. [What kind of a teacher I am]. Prague: Portál.
Huber, S. A., & Seidel, T. (2018). Comparing teacher and student perspectives on the interplay of
cognitive and motivational affective student characteristics. PLoS ONE, 13(8), e0200609. https://
doi.org/10.1371/journal.pone.0200609
Jurik, V., Gröschner, A., & Seidel, T. (2014). Predicting students’ cognitive learning activity and intrinsic
learning motivation: How powerful are teacher statements, student profiles, and gender? Learning
and Individual Differences, 32, 132–139. http://dx.doi.org/10.1016/j.lindif.2014.01.005
Kubiatko, M., Janko, T., & Mrázková, K. (2012). The influence of gender, grade level and favourite
subject on Czech lower secondary school pupils’ perception of geography. International Research
in Geographical and Environmental Education, 21(2), 109–122. https://doi.org/10.1080/103820
46.2012.672675
Latham, G. P., & Locke, E. A. (1991). Self-regulation through goal setting. Organizational Behavior and
Human Decision Processes, 50, 212-247.
Leon J., Medina-Garrido, E., & Núñez, J. L. (2017). Teaching quality in math class: The development
of a scale and the analysis of its relationship with engagement and achievement. Frontiers in
Psychology, 8, 895.
Luttenberger, S., Wimmer, S., & Paechter, M. (2018). Spotlight on math anxiety. Psychology Research
and Behavior Management, 11, 311–322.
Maslow, A. H. (1954). Motivation and Personality. New York: Harper and Row.
Murayama, K., Pekrun, R., Lichtenfeld, S., & Vom Hofe, R. (2013). Predicting long-term growth in
students’ mathematics achievement: The unique contributions of motivation and cognitive
strategies. Child Development, 84(4), 1475–1490.
Tossavainen, T., & Juvonen, A. (2015). Finnish primary and secondary school students’ interest in music
and mathematics relating to enjoyment of the subject and perception of the importance and
usefulness of the subject. Research Studies in Music Education, 37(1), 107-121.
Pajares, F. (1996). Self-efficacy beliefs in academic settings. Review of Educational Research, 66(4),
543578. https://doi.org/10.3102/00346543066004543
Plháková, A. (2010). Učebnice obecné psychologie. [Textbook of general psychology]. Prague: Academia.
Popa, D. (2015). The relationship between self-regulation, motivation and performance at secondary
school students. Procedia – Social and Behavioral Sciences, 191, 2549–2553. https://doi.
org/10.1016/j.sbspro.2015.04.410
Prokop, P., Tuncer, G., & Chudá, J. (2007). Slovakian students’ attitudes toward biology. Eurasia Journal
of Mathematics, Science and Technology Education, 3(4), 287–295.
OECD. (2003). Mathematics teaching and learning strategies in PISA, PISA: OECD Publishing, 176
p. Retrieved 10/11/2019 from: https://www.oecd-ilibrary.org/docserver/9789264039520-en.
pdf?expires=1573914856&id=id&accname=guest&checksum=49A7472FD3E181C4CFDDF-
53CAF767209
Schiefele, U., Krapp, A., & Winteler, A. (1992). Interest as a predictor of academic achievement: A
meta-analysis of research. In K. A. Renninger, S. Hidi, & A. Krapp (Eds.). The role of interest in
learning and development (pp. 183–212). Hillsdale, NJ, US: Lawrence Erlbaum Associates.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
23
Received: September 20, 2019 Accepted: January 15, 2020
Cite as: Berková, K., Krejčová, K., & Novák, J. (2020). Students’ attitudes to selected
subjects in secondary economic education. Problems of Education in the 21st Century, 78(1),
9-23. https://doi.org/10.33225/pec/20.78.09
Kateřina Berková Ph.D., Assistant Professor at Department of Economic Teaching Methodology,

(Corresponding author) Faculty of Finance and Accounting, University of Economics, Prague, W.
Churchilla Sq. 4, Prague, 130 67, Czech Republic.
E-mail: katerina.berkova@vse.cz
Website: https://ffu.vse.cz/english/departments/economic-teaching-methodology
ORCID: 0000-0002-6158-0171
Kristýna Krejčová Ph.D., Assistant Professor at Department of Economic Teaching Methodology,

Faculty of Finance and Accounting, University of Economics, Prague, W.
Churchilla Sq. 4, Prague, 130 67, Czech Republic.
E-mail: kris.krejcova@gmail.com
Website: https://ffu.vse.cz/english/departments/economic-teaching-methodology
ORCID: 0000-0002-7909-4649
Jaromír Novák PhD., Assistant Professor at Department of Pedagogy, Faculty of National

Economy, University of Economics in Bratislava, Dolnozemská cesta 1, Bratislava
852 35, Slovakia
E-mail: jaromir.novak@euba.sk
Website: https://nhf.euba.sk/en/departments/department-of-pedagogy/about-
department
ORCID: 0000-0002-4666-9047

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
24
IMPLEMENTATION OF MOBILE LEARNING
USING A SOCIAL NETWORK PLATFORM:
FACEBOOK
John, G. Chaka, Irene Govender
University of KwaZulu-Natal, South Africa
E-mail: chakajohn@yahoo.com, govenderi4@ukzn.ac.za
Abstract
This research set out to determine the use of a social networking platform, Facebook, as a medium for
learning and teaching. Using a combination of the unified theory of acceptance and use of technology
(UTAUT) model, Information Systems (IS) success model and the educational use of Facebook theory,
a conceptual framework was developed for implementation of mobile learning and teaching. Structural
equation modelling (SEM) was used to determine a model for the implementation of mobile learning
and teaching. A sample of 330 respondents was used in this research. The data were analysed using
structural equation modelling. The aim was to determine to what extent mobile learning is acceptable to
colleges of education in Nigeria. Lecturers from a federal college of education in Nigeria were used to
test the implementation of mobile learning using Facebook as a medium of communication. Four courses
were selected as part of the intervention programme. The research revealed that m-learning significantly
improves the teaching and learning conditions in colleges of education, specifically by reducing the
inadequacies of physical facilities, and by improving the reading culture and performance of students.
The results further indicate that most of the stakeholders are satisfied with the benefits of m-learning and
wish to continue using the Facebook and their mobile phones.
Keywords: educational system, quantitative approach, mobile learning, Facebook, SEM.
Introduction

The United Nations Education, and Scientific Corporation – UNESCO (as cited in Otoo
(2015)) has stated that Nigeria leads the world in terms of the number of people that have
difficulty in accessing education. The inadequacy of the teaching and learning environments in
most educational institutions especially at the tertiary level may have aggravated the situation.
Most higher education institutions in the country lack the needed facilities, for example
classrooms, books and other relevant educational materials (Asiyai, 2013; Ilogho, 2015; Jaja,
2013; Okojie, 2013). In addition, most Nigerian higher education institutions still depend on
time-honoured teaching and learning methods, which are largely constrained by using the
physical infrastructure such as classrooms, and libraries, which are not always suitable for
teaching and learning. This makes it difficult for the Nigerian education institutions to cater to
the increasing number of students who seek entrance to tertiary education, which gave impetus
for this research.
Based on the positive results obtained from preliminary studies (Chaka & Govender,
2017a; Chaka & Govender, 2017b), which revealed that both lecturers and students were accepting
the use of mobile devices to teach and learn, it became necessary to test the implementation of
mobile learning to see how the acceptance and success measures work together. This research
ascertains the viability and success of mobile learning in improving the teaching and learning
conditions in Colleges of Education. Specifically, the Facebook application was used as the

John, G. CHAKA, Irene GOVENDER. Implementation of mobile learning using a social network platform: Facebook
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
mobile learning medium. Most stakeholders in Colleges of Education have been using this 25
application as ascertained from a preliminary study (Chaka, 2018). This research reports on
part of a larger research study (Chaka, 2018).
Literature Review
Related Literature
Serbessa (2006, p. 129) described the time-honoured teaching method as the “chalk and
talk” lecture approach where the students are passive recipients of knowledge. The teacher is
at the centre of the teaching process using a rigid curriculum structure to teach all students in
the same manner (Ahmed, 2013; Bishara, 2015).The problem with this approach is that access
to education, depends on availability of physical infrastructure such as classrooms while
learning depends on the teachers’ competence. Thus, Clark and Ausukuya (2013) reported
that the insufficient number of teachers in most higher education institutions has resulted in
teacher-student ratios as high as 1:356 in some cases, as against the recommended ratio of
1:25. This has affected not just the quality of training and performance of existing students,
but also the enrolment of new students thereby increasing the access gap to education in the
country. Klopfer, Osterweil, Groff and Haas (2009) argued that social networks have all the
characteristics and potential of achieving good learning, as users are now creating, learning,
and communicating in new ways. Thus, since time, lack of skills and lack of money constitute
the major barriers to the use of conventional Virtual Learning Environment (VLEs) (Walker
et al., 2014), the application of social networking sites in education seems to address these
challenges. Corroborating this position, Selwyn (2012) stated that, perhaps the driving force
behind the educational use of social networking sites is the nature of students that are enrolling
in the institutions. Selwyn (2012, p. 2) further described the incoming students as those “who
know nothing other than a life with the internet.”
In the light of the above, Alsereihy and Al Youbi (2014) have asserted that with the
exponential increase in the popularity of social networking media, higher education institutions
are beginning to embrace their applications for various educational purposes. They maintain
that a number of Universities in the Western world today have included social networking
sites as part of their platform to facilitate communication, research and collaboration
among stakeholders. Thus, with the social interactive, communication and collaborative
features of Social Networking Sites (SNS) which are lacking in most conventional learning
management systems (LMSs), social networking sites are believed to be well-positioned to
support teaching and learning (Buzzetto-More, 2012; Mazman & Usluel, 2010). They further
argued that social networking sites, specifically Facebook, have the capacity to facilitate
online discussions and collaboration among students, and between students and teachers, in
addition to providing opportunities for the sharing of ideas, educational content as well as
enhancement of communication among stakeholders. This is very important in light of the fact
that the current generation of students are digital natives. This compels learning to be socially
and digitally driven by ubiquitous technologies (Buzzetto-More, 2012). They maintain that
since social networking sites are being used extensively by many people, their application
to education is long overdue in view of the numerous advantages including learner-centred,
collaboration, and active participation among others. They further stated that most people
have a preference for communicating online instead of the traditional face-to-face approach.
Yapici and Havedanli (2014), maintain that since most teachers and students are users of social
networks, with Facebook having the largest population (e.g. 34% of Facebook users in Turkey
are higher education students), the application of social networks in facilitating teaching and
learning will certainly be rewarding. In this sense, Ahn (2011) argued that the youth of today

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
26 communicate and learn more from outside the school premises. Thus, engaging in interactions
using social networking sites is viewed as a form of literacy practice which contributes to
learning. According to Adewole and Fakorede (2013), appropriation is a skill which enables
the communication of ideas by remixing content from different sources. They posit that social
networking sites provide the tools to enable students to gather, synthesize and remix content,
thereby acquiring skills and knowledge. Similarly, Clark and Ausukuya (2013), stated that
networking is the capacity to search, integrate, and disseminate information, which social
networking sites provide.
In view of this, the next section examines the extent to which social networking sites
have been useful in teaching and learning.
Application of Social Networking in Teaching and Learning
In what appears like the first stage of studies to establishing a link between the use
of SNS and educational performance, Karpinski and Duberstein (2009) conducted a survey
among students of Ohio Dominican University to ascertain if there was a link between the
use of Facebook and educational performance. Although part of the findings of Karpinski and
Duberstein (2009) indicated that Facebook users had lower performance than non-Facebook
users, their study did not suggest that Facebook use led to lower performance, as there could
be other factors such as personality traits. Building on this, Pasek, More, and Hargittai (2009)
examined the association between social networking use and learning outcomes among
undergraduate students, and they found that Facebook usage had no link with GPA. They
therefore suggested further research to explore the association between Facebook and academic
performance of students.
Laying a theoretical foundation on the application of SNS in education, Mazman and
Usluel (2009) proposed a model for the educational use of social networking sites. Their model
was conceived from previous innovation and technology acceptance models. Following this
initial conception, Mazman and Usluel (2010) came up with a structural model to explain
how users can use social networking sites specifically Facebook for educational purposes.
The researchers tested out their model and found that user-purposes and adoption processes
accounted for 50% of the variance in educational use of Facebook while Facebook adoption
processes explained 86% of user purposes. In other words, while Facebook adoption processes
alone accounted for 45% of variance in educational use of Facebook, the joined effects of
Facebook adoption processes and user purposes accounted for 50% of the variance in education
use of Facebook.
Additionally, in a bid to understand the efficacy of social networking sites as instructional
tools, Buzzetto-More (2012) examined the perceptions of management students who have
already completed courses. As part of his methodology, the researcher integrated Facebook use
into some courses that were taken by management students for at least one academic semester.
Facebook was used to create course groups for the courses involved and all students who had
registered for the various courses were required to join. The findings from their study revealed
that 33% of the participants indicated that they stay online for 3-8 hours per week engaged in
social networking sites, and that 47% of them agreed that Facebook develops interpersonal
relationships. In addition, Buzzetto-More (2012) found that social networking sites especially
Facebook can strengthen relationships within learning communities as 47 per cent of the
students agreed that Facebook can enhance their learning process. The report further reveals
that 50% of the students see Facebook as a good tool to engage in learning. To further verify
the use of SNS, Shembilu (2013) examined the benefits of social networking in education
in Tanzania. The study found that the majority of the students (96%) use social networking
sites to socialize with peers while a reasonable number (63%) use social networking sites

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
for sharing academic-related information. Shembilu (2013) also found out that Facebook 27
was the most preferred social networking site while blogs ranked second. The study further
revealed that 65% of the respondents feel that social networking sites can serve both social
and academic needs of students. Regarding measures that needed to be put in place to facilitate
the educational use of SNS, sensitization of students and educators was ranked first (87%)
followed by provision of Internet access (68%), and distribution of Internet and ICT facilities
nationwide (42%). Forcing government to provide free Internet centres was ranked last (12%).
Shembilu (2013) concluded that although participants see the educational use of social
networking sites as promising, and that there was some evidence of the educational use of
SNS in Tanzania, not much sensitization had been done to widen the scope of usage. In
addition, challenges of ICT infrastructure and cost of Internet bandwidth served as obstacles.
Similarly, in order to assess the viability of SNS sites in facilitating teaching and learning,
Dunn (2013) focused on the application of mobile and portable devices. Dunn (2013) found
that the majority of students (92%) use one form of social networking medium or the other,
with Facebook (86%) being the most popularly used. The study further indicated that 87%
of subscribers use SNS for personal use, 33% for networking with other professionals while
24% use it for research-related activities. Regarding the belief of students in the helpfulness
of social networking sites for learning, his study reveals that 75% of the students found social
networking sites very helpful in learning. In a more recent study, Gülbahar (2014) found that
students have positive perceptions on the use of social media in education but are, however
not aware of the various facilitating tools that these technologies offer. A very recent study by
Gorghiu, Iordache, Pribeanu, and Lamanauskas (2018) revealed that if students use Facebook
for learning purposes, it is mainly for sharing learning material and collaboration.
Alsereihy and Al Youbi (2014) argued that the positive reports that are emanating
from the western world on the usage and application of social networking sites in education
cannot not be generalized to other countries. In the light of this, they examined the usage of
social networking sites by higher education institutions in Saudi Arabia. Alsereihy and Al
Youbi (2014) found among other things that the use of social networking applications was
prevalent in major Universities in Saudi Arabia. They further found that Facebook was leading
in terms of usage and awareness, but that the use of social networking sites as collaborating
and educational tools was underutilized amidst privacy and security concerns.
Despite the promises of social networking sites and mobile learning in facilitating
teaching and learning, no known study has integrated the use of mobile learning with social
networking sites in Nigeria, especially in the context of Colleges of Education. Some studies
(such as Adedoja et al., 2013; Osang & Ngole, 2014; Shaibu & Mike, 2014) have investigated
mobile learning in Universities but not with social networking sites as the learning platform.
The situation in other higher education institutions remains unclear in view of different
affordances, which are key determinants of any research output (Traxler, 2007; Thomas et
al., 2013). Colleges of Education are vital, based on their role as teacher training institutions,
but are most affected in terms of the challenges that are confronting teaching and learning
in Nigerian higher education institutions (Ogunyinka et al., 2015; Tella, 2011). Therefore,
exploring the use of mobile learning in improving the teaching and learning conditions in
higher education institutions, specifically Colleges of Education has become imperative.
In view of the affordability and commonality of SNS, their use as the learning platform is
considered.
Hence, this research was guided by the following research questions:
1. To what extent do the acceptance factors (performance expectancy, effort expectancy,
social influence, mobile learning conditions, anxiety & personal innovativeness)
jointly with user satisfaction influence the behavioural intention of participants to
use Facebook as a medium to teach?

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
28 2. To what extent do mobile learning conditions, behavioural intention, and user

satisfaction influence the actual use of mobile learning?
3. How does the actual use of mobile learning and user satisfaction result in an improved
educational system?
Conceptual Framework
In order to guide this research towards achieving its aim, an appropriate conceptual
framework is required. Most of the studies reviewed in the literature made use of the unified
theory of acceptance and use of technology (UTAUT) or variations of the theory to understand
the acceptance or intention and usage of mobile learning in higher education. However, one
limitation of UTAUT in relation to this study is that UTAUT only measures the likelihood
of users to accept a technology and the users’ intended usage behaviour of the technology
(Venkatesh et al., 2003). In other words, UTAUT is mainly suitable for use prior to the
application or implementation of a technology. By implication, UTAUT alone is unable to
address the objectives of this study. Three models – the UTAUT model, the IS success model,
and the educational use of Facebook model are considered critical in explaining the acceptance
and use of mobile learning, as well as educational use of social networking sites. While the
IS success model is considered suitable in view of its capacity to measure the success of the
application of information systems in organisations, the educational use of Facebook model
brings into the study constructs that are peculiar to the use of Facebook.
Figure1
Conceptual framework of the research
Acceptance Constructs Success Constructs

Performance Communication & Collaboration &
Expectancy Interaction Resource
Sharing
Effort
Expectancy Actual Use of
Behavioural Mobile
Social Influence Intention Learning Less Demand
on Teacher
Less Demand
Anxiety on Physical
Facilities
Improved
Personal Educational Improved
Information System
Innovativeness Reading
Quality Culture
System
Mobile Learning Improved
Quality Performance
Conditions
Service User
Quality Satisfaction
Student-
centric
Learning

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Research Methodology 29
Research Approach and Design
The quantitative approach followed the survey research strategy. This was to enable
the possibility of generalizing the findings of the research to the entire population. The survey
design was considered more feasible in this research in view of its advantages, most especially
cost saving (Kumar, 2011). In addition, it was easier and advantageous to measure the attributes
of a population from a smaller group (Creswell, 2014). This was imperative since Nigeria is one
of the largest countries in Africa (landmass of 923,768 km²).
Research Site
The research was conducted in a College of Education in Nigeria, specifically in the

North-central geopolitical zone of the country. This zone was selected in view of its cosmopolitan
nature. Apart from being the zone where the nation’s capital (Abuja) is located, the zone is in
the centre of the country and draws from the diverse cultures of almost all the ethnic groups in
Nigeria.
Intervention - Preparation of Lecturers for Implementation

As part of the implementation process, the following guidelines were specifically
provided to lecturers to follow:
1. Create a group in Facebook and name the group after the course that you are teaching
(e.g. call the group DBE 221) if DBE 221 is the course to be taught;
2. Enrol your students as members of the group. Also add me (researcher) as a member
of the group;
3. All activities should be performed using mobile devices (phones and possibly
tablets) only.
4. Engage your students in various teaching and learning activities such as:
a. Discussion forum: Post a question or topic that is relevant to the course and
allow students to discuss it freely while the teacher facilitates.
b. Sharing and collaboration: Encourage students to do more research into the
topic and share relevant links with colleagues in the group. The teacher can
also share relevant links, documents, files, videos, etc. in the groups.
c. Assessment: Teachers can post assignment as a document or file to the group
and advise students to submit the same through a private message to them.
d. Motivation: Teachers should encourage participation of students by awarding
marks to very active participants and to quality contributors in the group.
e. Ubiquitous: since mobile devices are being used, participants should be
engaged with the exercises at anytime and anywhere.
Population and Sample
Lecturers from a College of Education were used to test the use of mobile learning by
using the technology to supplement the teaching of four selected courses (CS 412, DBE 221,
GUC 205, SSE 313). One college was used to pilot the implementation of the use of social
network site for effective management and monitoring of the exercise. Since the researcher was
a member of staff, it was easy to gain access to the system as well as obtain the cooperation
of the staff and students. Kumar (2011) posits that bias can be introduced by a researcher if it
becomes difficult to penetrate the population.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
30 Four lecturers from four of the five schools in the college were part of the intervention.
The schools included were School of Arts and Social Sciences, School of Education, School
of Sciences, and School of Vocational and Technical Education. Based on Creswell’s (2014,
p.93) list of ethical issues and possible ways of addressing them, this study adhered to the
following ethical procedure: Permission was sought and granted from the National Commission
for Colleges of Education (NCCE) - to undertake the study in Colleges of Education. A written
approval was then granted by the College of Education. For all individuals that were involved
in the study, their consent was obtained through a letter that was attached to the data collection
instrument.
The researcher used his experience as an insider to select lecturers that were more
knowledgeable in the use of computer and social networking applications. A social networking
site (Facebook) was used as the teaching and learning platform. This is in view of the availability
of the application, its user friendliness and compatibility with most mobile devices, and the
fact that most members of the colleges were already using it. Based on Krejcie and Morgan’s
(1970) table and Kumar’s (2011) suggestion, a study sample of 330 was considered adequate
for the implementation of the research. A sample of 330 students was used in this study. The
data were analysed using structural equation modelling.
Data Collection Instruments

The questionnaire used consisted of two sections – Section ‘1’ obtained demographic
information while Section ‘2’ elicited information regarding the use of mobile learning in line
with the constructs contained in the conceptual framework of the research. The items in the
questionnaire were measured on a 5-point Likert scale ranging from 1 = strongly disagree to 5
= strongly agree. Different components of the questionnaire items were adapted from various
sources. The first set of 26 items which measured the acceptance of mobile learning using a
social networking site (Facebook) were adopted from Venkatesh et al., (2003) and Abu-Al-
Aish and Love (2013). The second set of 48 items which measured the actual use and success
of mobile learning were adopted from a number of sources (Alharbi & Drew, 2014;Delone &
McLean, 2003; Lin et al., 2016). All the scales used in the research had good internal consistency
reliability ratings, with most of them above .9.
Data Analysis

Categorical demographic variables were analysed or explored from different perspectives
using cross-tabulations, while continuous variables were analysed using descriptive statistics
including mean and standard deviation. Structural equation modelling (SEM) was used to test
the fitness of the model to the data and to explore various relationships in the conceptual model
in order to provide answers to the research questions. SEM was considered suitable to use in
the implementation stage to enable the researcher test the conceptual framework which was a
new conception (Kumar, 2011; Pallant, 2011).
A total number of 330 questionnaires were distributed to the respondents using the
physical (face-to-face) method out of which 324 were retrieved. Three of the questionnaires
were later discarded for inconsistency resulting in 321 useable questionnaires and a final
response rate of 97%.
Research Results
Descriptive Results
The descriptive results emerged from the analysis of demographic information. Table 1
shows a cross-tabulation of stakeholders grouped by gender.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Table 1 31
Analysis of respondents by gender
Gender
Male Total
Female
Student 184 133 317

Stakeholders' group
Lecturer 4 0 4
Total 188 133 321
From Table 1, it is apparent that the target population consisted of more males than
females, in particular there were only male lecturers who participated in this research.
Table 2 gives some indication of the characteristics of the participants.
Table 2
Analysis of demographic information for continuous variables
N Min Max Mean SD Skewness Kurtosis
Age 321 19 52 26.98 5.156 1.561 .136 3.575 .271

Use of mobile
device 321 1 12 4.12 2.355 .797 .136 .208 .271
experience
Valid N (listwise) 321
Table 2 shows that the minimum age of the respondents is 19 years while the maximum
age is 52 years. The average age was 26.98. From the Table, it is gathered that the age of
the respondents is positively skewed; meaning that majority of the respondents lie below the
average age of 26.98. Table 2 further reveals that the respondents have used mobile devices for
a minimum of one year and a maximum of 12 years.
Quantitative Data Analysis Procedure Using SEM
SEM using IBM Amos version 23 was used to analyse the data by exploring various
relationships that exist in the proposed or conceptual framework. The primary goal of SEM
is to determine and validate a proposed causal process or a model. In other words, the model
used in the research was specified as the conceptual framework, subject to confirmation. We
have a covariance matrix to serve as our dataset, which is based on the sample of collected
measurements. The empirical question of SEM is therefore whether the proposed model
produces a population covariance matrix that is consistent with the sample covariance matrix.
Hence the model specified/proposed needs to be validated/ tested.
Therefore, using SEM is considered suitable to test the conceptual model, which is a new
formulation. The interplay between different variables provides the answers to the research
questions. In order to apply SEM, the data must satisfy a number of assumptions. Thus, as part
of the process of analysis, the data in this study was first cleaned up and tested for the non-
violation of the assumptions of SEM.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
32 Testing for non-violation of assumptions
To conduct SEM, the data must meet two criteria, completeness and normality. There
was no need to adopt any ad hoc measure since no missing data was present.
Normality was checked by looking at skewness and kurtosis for all variables in the
model. The normality of the data on the dependent and/or mediating variables in the model
(behavioural intention, and user satisfaction) were found to appear normal (skewness <=3 and
Kurtosis <=10) (Pallant, 2011). This was done by assessing the effect of 5% trimmed mean on
actual values. Skewness, and kurtosis of each measured (observable) dependable variable as
shown in Table 3. The 5% trimmed mean refers to the computed mean excluding 5% of the
upper (largest) and lower (smallest) values of the distribution.
Table 3
Trimmed mean, skewness and kurtosis statistics for dependent variables
Descriptive
Statistic Std. Error
Mean 7.07 .101
Lower Bound 6.88
95% Confidence Interval for Mean
Upper Bound 7.27
5% Trimmed Mean 7.15
Median 7.00
Variance 3.307
Std. Deviation 1.818
Minimum 2
BI
Maximum 10
Range 8
Interquartile Range 2
Skewness -.654 .136
Kurtosis .157 .271
Mean 10.35 .144
Lower Bound 10.06
95% Confidence Interval for Mean
Upper Bound 10.63
5% Trimmed Mean 10.42
Median 11.00
Variance 6.652
Std. Deviation 2.579
Minimum 3
US
Maximum 15
Range 12
Interquartile Range 3
Skewness -.509 .136
Kurtosis -.042 .271

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Sample Size 33
Sample size is also an important factor in SEM. Although there may be no consensus
regarding the sample size appropriate for SEM analysis (Sivo et al., as cited in (Hoe, 2008)), it
is suggested that large samples of say 200 and above may result in the needed level of statistical
power with any given model (Schreiber, Nora, Stage, Barlow, & King, 2006). Hence, the sample
size of 321 utilized in this study is considered appropriate for applying SEM.
Exploratory Factor Analysis and Validation of Measurement Model
As part of the refinement process of the measurement model, exploratory factor

analysis is used to assess the factor loadings of all items that estimate each construct. Due
to the complexity of the conceptual model of the study, each of the 4 sub-models were taken
individually and tested for ‘uni-dimentionality’ and cross-loading. Reliability and validity were
calculated and, based on these results, adjustments were made. These adjustments included
adding in covariance indicators to error variances and removing any measured items with low
loadings or indications of cross-loading.
After the sub-models were treated individually and the ‘best’ sub-models achieved, they
were taken together in pairs and then three at a time and the same process was carried out.
Information regarding each construct in the measurement model with the number of factors
entered and retained is presented in Table 4.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
34 Table 4
Information on various constructs in the measurement model
Number of items retained

Construct Initial number of items
Sub-model A
Performance expectancy 4 2
Effort expectancy 4 3
Social Influence 4 2
Anxiety 3 3
Personal innovativeness 3 2
Mobile learning conditions 4 3
2
Behavioural intention 4
Sub-model B
Communication & interaction 5 3
2
Collaboration & resource sharing 4
Sub-model C
Information quality 7 5
System quality 5 4
Service quality 6 5
Student-centric learning 3 2
3
User satisfaction 3
Sub-model D
Less demand on teacher 3 2
Less demand on physical facilities 3 3
Improved reading culture 5 3
Improved performance 4 2
Fitness of the Model
To assess the fitness of the model, fit indices were used. Since it is not necessary or
realistic to report all the fit indices produced by SEM application, Hooper et al., (2008) have
recommended that, in addition to Chi-square value, its degree of freedom and the p-value, a
variety of the indices which tests different aspects of the model fit should also be reported. In
accordance with this discussion, this research reports at least one fit index from each test group.
The chi-square value (CMIN or χ2), degree of freedom (df), chi-square value over degree of
freedom (CMIN or χ2 / df), RMSEA, GFI, IFI, TLI, and CFI were reported for each model test.
For the integrated model proposed in this research, it was found that the best fit indices
that could be obtained without dropping any of the constructs were not entirely within the
prescribed values as can be seen in Table 5.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Table 5 35
Overall fit indices of the proposed measurement model
Fit Index Observed Value
CMIN 1543.415
df 1040
CMIN/Df 1.484
GFI .847
IFI .909
TLI .890
CFI .906
RMSEA .039
As can be seen from Table 5, with the exception of GFI and TLI which are adequate or
marginally acceptable, all the indices indicate a good fit (Barrett, 2007; Hoe, 2008). The model
was then tested for reliability and validity, which are presented in Table 6.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
36 Table 6
Reliability and validity measures of the proposed measurement model
Cronbach's
Construct Items Loadings CR AVE MSV MIC
alpha
Performance expectancy PE2 0.854 0.747 0.765 0.625 0.228 0.410
PE3 0.698
Effort expectancy EE1 0.700 0.695 0.687 0.423 0.394 0.419
EE2 0.612
EE3 0.653
Social Influence SI2 0.656 0.684 0.690 0.530 0.228 0.381
SI3 0.792
Mobile learning conditions MLC1 0.690 0.694 0.688 0.425 0.394 0.382
MLC2 0.670
MLC3 0.613
Anxiety AX1 0.589 0.718 0.723 0.469 0.032 0.458
AX2 0.788
AX3 0.661
Personal innovativeness PI1 0.819 0.757 0.761 0.615 0.119 0.484
PI2 0.746
Behavioural intention BI1 0.798 0.657 0.661 0.497 0.262 0.369
BI2 0.616
Information quality IQ1 0.538 0.803 0.795 0.441 0.408 0.387
IQ2 0.681
IQ3 0.678
IQ4 0.770
IQ5 0.639
System quality SQ1 0.452 0.648 0.664 0.402 0.209 0.339
SQ2 0.642
SQ3 0.573
SQ5 0.682
Service quality SVQ2 0.647 0.745 0.765 0.395 0.375 0.357
SVQ3 0.640
SVQ4 0.600
SVQ5 0.637
SVQ6 0.616
User satisfaction US1 0.859 0.819 0.853 0.659 0.408 0.402
US2 0.776
US3 0.804
Student-centric learning SCL1 0.722 0.751 0.750 0.601 0.251 0.498

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
SCL2 0.836 37
Less demand on teacher LDT2 0.712 0.71 0.712 0.553 0.458 0.452
LDT3 0.774
Less demand on physical
LDF1 0.761 0.765 0.764 0.520 0.461 0.521
facilities
LDF2 0.720
LDF3 0.683
Improved reading culture IRC2 0.672 0.761 0.759 0.513 0.456 0.407
IRC3 0.785
IRC4 0.700
Improved performance IP1 0.743 0.659 0.665 0.499 0.358 0.377
IP2 0.667
Communication & interaction CMI1 0.648 0.724 0.722 0.465 0.449 0.374
CMI2 0.694
CMI3 0.716
Collaboration & resource sharing CLR1 0.828 0.728 0.735 0.583 0.449 0.438
CLR2 0.694
Reliability
The reliability or internal consistency of a scale ascertains how measures of the same
constructs hang together (Pallant, 2011). Internal consistency is indicated if Cronbach’s alpha
values are greater than 0.7. From Table 6 above, it is observed that while some variables indicate
acceptable and good reliability, others (effort expectancy, social influence, mobile learning
conditions, behavioural intention, system quality and improved performance) have marginally
acceptable Cronbach alpha coefficients (Chinomona & Cheng, 2013).
However, it has been observed that Cronbach’s alpha coefficients are sensitive to the
number of items in a scale (Gliem & Gliem, 2003), such that scales having fewer than ten
items often result in Cronbach’s alpha coefficients as low as 0.5 (Pallant, 2011). While some
researchers (such as (Briggs & Cheek, 1986)) suggested that a scale with mean inter-item
correlations (MIC) coefficients in the range of 0.2 to 0.4 is reliable, others (for example (Clark
& Watson, 1995)) recommended a range of 0.15 to 5.0 depending on the nature of construct
being measured. Since the number of items used in the various subscales in this study is below
ten, the mean inter-items correlation for each construct is also computed and considered. In this
case, all constructs are considered reliable.
Validity
In this research, convergent or construct validity and discriminant validity were used
to measure the validity of the scale. Convergent validity ascertains the level of agreement
between items that measure the same construct (Guo, Aveyard, Fielding, & Sutton, 2008).
Convergent validity is assessed by observing factor loadings, composite reliability (CR) values,
and average variance extracted (Field, 2009; Hair et al., 2009). These measures determine if
variables converge at some point. The rule of thumb is that CR values should be greater than
0.6, and average variance extracted or estimated (AVE) should be greater than 0.5 (Hair et al.,
2009). Additionally, factor loadings should be greater than 0.5.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
38 Table 6 shows that all factors have acceptable loadings, just as all constructs have acceptable
CR. Similarly, most of the constructs indicate acceptable values for AVE (> 0.5), while others
(with AVE > 0.4) are marginally acceptable (Chinomona & Cheng, 2013). Service quality (SQ)
with AVE < 0.4 seems to be the only concern in this case.
Discriminant Validity
Discriminant validity measures the distinctiveness of constructs in a given model (Guo

et al., 2008). In this research, discriminant validity is assessed by ascertaining the maximum
shared variance in each construct, which should be less than AVE. As can be clearly seen from
Table 6, all constructs in the study passed the discriminant validity test.
Multicollinearity
A check to ensure the non-violation of the assumptions of multicollinearity was carried

out. Multicollinearity ascertains the extent to which the various constructs are related to or
measure the same concept/object. Specifically, the tolerance and variance inflation factor (VIF)
values in the coefficient table were inspected against the prescribed values (tolerance value <
0.10, or a VIF value >10 for the presence of multicollinearity) (Pallant, 2011). Results indicate
a non-violation of multicollinearity assumption as the lowest tolerance value was 0.647 and the
highest VIF value was 1.415.
Improvement of the Measurement Model and Results
After exhausting all the possible avenues towards improving the model, the model
still did not fit the data well. A further improvement on the model was not possible without
dropping some variables. To improve the model in this regard while ensuring the retention of
most vital components of the model, it became imperative to drop anxiety from the acceptance
measures, and then system quality and students’-centric learning from the success measures.
These changes yielded some more improvements on the model. The final measurement model
and fit indices are presented in Figure 2 and Table 7 respectively.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Figure 2 39
Measurement model for the acceptance and success of m-learning in colleges of education
The measurement model shown in figure 2 consists of different components including

rectangular boxes, circle/ellipses, error terms, non-causal or non-directional relationships
(correlations), causal relationships or directional relationships (single headed arrows). The
rectangular boxes represent observed or measured variables. In this case, they represent
measured items (questions) that were specified in the measuring instrument. The components
represented as circles/ellipses are latent or unobserved variables including exogenous or
independent variables (IVs) and endogenous or dependent (DVs). These variables were
measured through the variables in rectangular boxes as indicated by the path coefficients (single
headed arrows that flow from the IVs/DVs to the measured variables). The circles/ellipses with
numbers that are preceded by the letter ‘e’ are error terms or residuals. They represent omitted
causes or errors on each measured variable. The double-headed arrows connecting two error
terms represent the residuals’ covariance. All unobserved variables are interrelated with each
other using double-headed arrows called correlations.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
40 Table 7
Final fit indices of the proposed measurement model
CMIN 1094.109
df 718
CMIN/Df 1.524
GFI .868
IFI .920
TLI .901
CFI .917
RMSEA .040
Table 7 indicates that with the exception of CFI which has a marginally acceptable index,
all the other indices are within the prescribed thresholds (Hoe, 2008; Tabachnick & Fidell,
2007)2007. On this basis, the measurement model is considered acceptable.
The Structural Model
Based on the acceptability of the measurement model, confirmatory factor analysis

was applied to the structural model. Composite scores for each construct were calculated by
averaging measured variables (questions) identified in the measurement model analysis. These
were then used in the model to test with SEM. As can be seen in Figure 2, Mobile Learning
Conditions (MLC) had two direct relationships – one relationship with Behavioural Intention
(BI), and the second relationship with Actual Use of Mobile Learning (AUML).
An assessment of the structural model showed that the model has adequate or marginally
acceptable fit. It was observed from the model that if the relationship between Personal
Innovativeness (PI) and BI is removed, the relationships between MLC and BI as well as that
between MLC and AUML become significant. Furthermore, if PI is completely removed from
the model, the model’s fit improves negligibly. The action further lowers the variance in BI
explained.
However, since dropping PI from the model did not result in any significant improvement,
PI was retained in the model. The structural model is presented in Figure 3.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Figure 3 41
Structural model for the acceptance and success of m-learning
In the structural model, the rectangular boxes represent observable IVs and DVs, while
the circles/ellipses represent latent variables. Table 8 presents the overall fit indices of the
model.
Table 8
Overall fit indices for the structural model

CMIN 205.573
df 66
CMIN/df 3.115
GFI .922
IFI .884
TLI .836
CFI .881
RMSEA .081
Table 8 reveals that RMSEA has a mediocre or average fit (MacCallum, Widaman,
Preacher, & Hong, 2001) while IFI, TLI, CFI have marginally acceptable or adequate fit
indices. GFI on the one hand has an acceptable index. This suggests that the model is marginally
acceptable or adequate.
In addition, although the indices shown in Table 8 may suggest that the model has
marginal or mediocre fit, Schunn and Wallach (2005) caution that goodness-of-fit measures for
a model should not be regarded as alpha-levels. They maintain (p.117) that “a model should
not be thrown out simply because it does not exceed some arbitrarily defined threshold for

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
42 goodness-of-fit”. Rather, goodness-of-fit should be compared to previous ones in the domain. In

other words, a relatively weak fit to the data can be acceptable in terms of newer models. Based
on this argument, this structural model is considered acceptable since it is a new formulation,
even though derived from three already validated theories. Further research should be carried
out to improve the model.
The model indicates that behavioural intention (BI) is the main variable that is used
to measure the acceptance of mobile learning. BI depends on five independent variables (PE,
EE, SI, PI, and MLC). BI relates directly with two success measures – AUML and US, while
MLC also relates directly to AUML. These relationships represent the operational interplay
between the acceptance of mobile learning and its success. AUML is a latent variable, which is
observed or manifests itself through two variables – Communication and Interaction (CMI), and
Collaboration and Resource Sharing (CLR). The overall dependent variable that represents the
net benefit of the study is Improved Educational System (IMP-ED_SYS). This variable is also
a latent variable, which is observed or manifests itself through four other variables (LDT, LDF,
IRC, and IP). The endogenous variable User Satisfaction (US) depends on two independent
variables – Information Quality (IQ) and Service Quality (SVQ). US plays a mediating role
between these two variables and the other three endogenous variables (BI, AUML, and IMP_
ED_SYS). Table 9 presents the regression estimates from the model.
Table 9
Regression weights of the structural model
Estimate S.E. C.R. P Label

US <--- IQ .251 .033 7.653 *** par_9
US <--- SVQ .250 .033 7.563 *** par_10
BI <--- EE .060 .046 1.293 .196 par_5
BI <--- SI .167 .049 3.436 *** par_6
BI <--- PI .163 .058 2.792 .005 par_7
BI <--- MLC .081 .042 1.917 .055 par_23
BI <--- US .174 .034 5.074 *** par_24
BI <--- PE -.026 .054 -.481 .631 par_26
AUML <--- BI .360 .071 5.064 *** par_8
AUML <--- MLC .166 .049 3.395 *** par_12
AUML <--- US .096 .046 2.085 .037 par_27
IMP_ED_SYS <--- AUML .251 .055 4.599 *** par_11
IMP_ED_SYS <--- US .148 .029 5.017 *** par_25
CMI <--- AUML 1.000
CLR <--- AUML .564 .083 6.793 *** par_1
LDT <--- IMP_ED_SYS 1.000
LDF <--- IMP_ED_SYS 1.461 .154 9.497 *** par_2
IRC <--- IMP_ED_SYS 1.133 .129 8.750 *** par_3
IP <--- IMP_ED_SYS .828 .104 7.955 *** par_4
Figure 2 and Table 9 are used to address the various research questions as highlighted
below.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Discussion 43
Joint influence of acceptance factors and user satisfaction on behavioural intention
As can be seen from Table 9, MLC does not have a significant influence on behavioural
intention to accept m-learning. The only acceptance measures or independent variables that
significantly predict BI are SI (ESC = .167, p<.005) and PI (ESC=.163, p=.005). As already
stated, the influence of MLC on BI becomes significant if the effect of PI is suppressed. The two
independent variables have positive effects on the intention of stakeholders to accept mobile
learning. This means that an increase in one standard deviation of SI will result in an increase
of 16.7% in the value of the standard deviation of BI. Similarly, an increase of one standard
deviation of PI will result in an increase of 16.3% in the value of the standard deviation of BI.
Furthermore, the research revealed that US has a significant influence on BI (ESC =
.174, p<.005). This means that an increase in the value of the standard deviation of US will
cause an increase of 17.4% in the value of standard deviation of BI. US has a stronger effect
on BI as compared to the individual effects of SI and PI. This is an indication that the user
satisfaction is a driving force in the intention of stakeholders to accept mobile learning. The
three predictors account for 18% of the variance in the intention of stakeholders in Colleges of
Education in Nigeria to accept mobile learning using Facebook.
Influences of Mobile Learning Conditions, Behavioural Intention and

User Satisfaction on Actual Use of Mobile Learning
Mobile learning conditions (MLC) significantly predict the actual use of mobile learning
(AUML). This effect is positive (ESC =.166, p<.005). The implication of this result is that an
increase in one standard deviation MLC will result in an increase of 16.6% in the value of the
standard deviation of AUML. This result agrees with the results of Venkaesh et al. (2003) as
well as Venkatesh et al. (2012). The results suggest that the actual use of mobile learning by
stakeholders in Colleges of Education will depend on the mobile learning conditions.
Table 9 further shows that BI (ESC =.360, p<.005) significantly predicts AUML as did
US (ESC = .096, p=.037). The effects of the two variables are positive, meaning that an increase
in one standard deviation of BI will yield an increase in the actual use of mobile learning by
36.6% of the value of its standard deviation. Similarly, an increase in one standard deviation
of US will result in an increase of 9.6% in the value of the standard deviation of AUML by
stakeholders.
The implication is that BI has the greatest effect on the actual use of mobile learning.
Thus, the more stakeholders develop an intention to use mobile learning, the more they use it.
The two variables explain 24% of the variance in the actual use of mobile learning.
The conceptual model shows that user satisfaction is an endogenous or dependable
variable, which has relationships with several factors including the actual use of mobile learning
(AUML), information quality (IQ) and service quality (SVQ). To assess the influence of AUML
on US, the relationship between the two variables was reversed. The results indicate that AUML
does not significantly predict US. This is contrary to the finding of Delone and McLean (2003).
In other words, the research indicates that the actual use of mobile learning was not a significant
determinant of the satisfaction of stakeholders in Colleges of Education.
Rather, two exogenous (independent) variables – Information Quality (ESC = .251,
p<.005) and Service Quality (ESC = .250, p<.005) significantly predict the satisfaction
of stakeholders in mobile learning. The two variables explained 39% of the variance in the
satisfaction of stakeholders.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
44 Influences of Actual Use of Mobile Learning and User

Satisfaction in an Improved Educational System
The study reveals that AUML (ECS = .251, p<.005) has a significant influence on IMP-
ED_SYS, as did US (ESC =.148, p<.005). The influences of the two variables are positive.
This means that an increase in one standard deviation of AUML results in an increase of 25.1%
in the value of the standard deviation of IMP_ED_SYS. Similarly, an improvement of one
standard deviation of US yields an improvement of 14.1% in the value of the standard deviation
of IMP_ED_SYS. This shows that AUML has the strongest influence on IMP_ED_SYS. The
two variables explain 32% of the variance in IMP_ED_SYS. This result may be suggesting that
other factors exist that account for the remaining variance in IMP_ED_SYS.
Feedback from the Improved Educational System on Actual

Use of Mobile Learning and User Satisfaction
The feedback that results from the net benefits of mobile learning (IMP_ED_SYS) leading
to satisfaction and actual use of mobile learning could not be ascertained with the quantitative
data. However, it is expected that stakeholders in Colleges of Education will become satisfied
due to the level of improvement that mobile learning brings to the current practices of teaching
and learning in the institutions. In addition, improvement in the teaching and learning conditions
in the colleges should result in the continuous use of mobile learning. Prior studies have not
been able to convincingly show the need for a social network site as a learning and teaching
medium. This study has provided a deeper insight into the use of Facebook as a medium for
teaching and learning among students in teacher education. The implication of these findings
is that more students can gain access to education if implemented properly. More importantly,
it is envisaged that this experience of using Facebook as a teaching and learning medium, can
influence their teaching approach in schools.
Conclusions
The research revealed that the use of mobile learning using a social networking
application such as Facebook may have strong support in Colleges of Education in Nigeria in
view of the derived benefits. Furthermore, the results indicate that the intention of stakeholders
depend significantly on the influence of important others, personal innovativeness and level
of satisfaction of users with the technology. In addition, the actual use of mobile learning
largely depends on the mobile learning conditions, behavioural intention and user satisfaction.
These in turn result in an improved educational system. This improved educational system is
reflected in less demand for facilities such as classrooms and books, improved reading culture,
and improved performance of students. The contribution of this research has been to confirm
that use of Facebook is an important path to improve not only learning, but improved access
to education in a context of teacher education that is often relegated to the background. It is
therefore necessary to harness the power of mobile learning to increase access and possibly
improve learning.
References
Abu-Al-Aish, A., & Love, S. (2013). Factors influencing students' acceptance of m-learning: An
investigation in higher education. The International Review of Research in Open and Distance
Learning, 14(5), 82-107. https://doi.org/10.19173/irrodl.v14i5.1631

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Adedoja, G., Adelore, O., Egbokhare, F., & Oluleye, A. (2013). Learners’ acceptance of the use of mobile 45
Pphones to deliver tutorials in a distance learning context: A case study at the University of Ibadan.
The African Journal of Information Systems, 5(3), 80-93. https://www.semanticscholar.org/paper/
Learners%E2%80%99-Acceptance-of-the-Use-of-Mobile-Phones-to-Adedoja-Adelore/0ff02ad9
dea7c71ed73d1200eec1296b5cfaeaf2
Adewole, E. G., & Fakorede, S. O. (2013). Strengthening the Nigerian higher education system through
the use of information communication technology. International Journal of Social Science and
Education, 3(4), 1006-1012. http://ijsse.com/sites/default/files/issues/2013/v3i4/papers/Paper-15.
pdf
Ahmed, A. K. (2013). Teacher-centered versus learner-centered teaching style. The Journal of Global
Business Management, 9(1), 22-34. http://www.jgbm.org/page/3%20Ahmed%20Khaled%20
Ahmed.pdf
Ahn, J. (2011). The effect of social network sites on adolescents’ social and academic development:
Current theories and controversies. Journal of the American Society for Information Science and
Technology, 62(8), 1435–1445. https://doi.org/10.1002/asi.21540
Alharbi, S., & Drew, S. (2014). Mobile learning-system usage: Scale development and empirical tests.
International Journal of Advanced Research in Artificial Intelligence, 3(11), 31-47.
Alsereihy, H. A, & Al Youbi, A. A. (2014). Towards applying social networking in higher education:
Case study of Saudi Universities. MAGNT Research Report, 2(4), 217-231. https://www.
semanticscholar.org/paper/Towards-Applying-Social-Networking-in-Higher-Case-Alsurehi-
Youbi/ff71ff4928dc8b0aab7a696ac3aa10e7126d3f50
Asiyai, R I. (2013). Challenges of quality in higher education in Nigeria in the 21st century. International
Journal of Educational Planning & Administration, 3(2), 159-172. https://www.ripublication.
com/ijepa/ijepav3n2_07.pdf
Barrett, P. (2007). Structural equation modelling: Adjudging model fit. Personality and Individual
Differences 42, 815–824.
Bishara, S. (2015). Active and traditional teaching of mathematics in special education. Creative
Education, 6, 2313-2324. http://dx.doi.org/10.4236/ce.2015.622238
Briggs, S. R., & Cheek, J. M. (1986). The role of factor analysis in the development and evaluation of
personality scales. Journal of Personality, 54, 106–148.
Buzzetto-More, N. A. (2012). Social networking in undergraduate education. Interdisciplinary Journal
of Information, Knowledge, and Management, 7, 63-90. http://www.ijikm.org/Volume7/
IJIKMv7p063-090Buzzetto611.pdf
Chaka, J.G. (2018). Towards improving teaching and learning in colleges of education using mobile
learning – The Nigerian perspective. Unpublished thesis - University of KwaZulu-Natal., South
Africa.
Chaka, J. G., & Govender, I. (2017a). Students’ perceptions and readiness towards mobile learning in
colleges of education: A Nigerian perspective. South African Journal of Education, 37(1), 1-12.
https://doi.org/10.15700/saje.v37n1a1282
Chaka, J. G., & Govender, I. (2017b). Staff acceptance of mobile learning: The case of Nigerian colleges
of education. Journal of New Generation Sciences (JNGS), 15(1), 223-242. https://hdl.handle.
net/10520/EJC-c85efa1fd
Chinomona, R., & Cheng, J. M. (2013). Distribution channel relational cohesion exchange model: A
small-to-medium enterprise manufacturers' perspective. Journal of Small Business Management,
51(2), 256-275.
Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development
Psychological Assessment, 7(3), 309-319.
Clark, N., & Ausukuya, C. (2013). An overview of education in Nigeria. World Education News and
Reviews website: http://wenr.wes.org/2013/07/an-overview-of-education-in-nigeria
Creswell, J. W. (2014). Research design: Qualitative, quantitative and mixed methods approaches (3rd
ed.). SAGE Publications.
Delone, W. H., & McLean, E. R. (2003). The DeLone and McLean model of information systems success:
A ten-year update. Journal of Management Information Systems, 19(4), 9–30.
Dunn, L. A. (2013). Teaching in higher education: Can social media enhance the learning experience?
Interdisciplinary Science Education, Technologies and Learning-The University of Glasgow, 1-1.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
46 Field, A. (2009). Discovering statistics using SPSS. SAGE Publications.

Gliem, J. A., & Gliem, R. R. (2003). Calculating, interpreting, and reporting Cronbach’s Alpha reliability
coefficient for likert-type scales. Paper presented at the Midwest Research-to-Practice Conference
in Adult, Continuing, and Community Education, Columbus, Ohio. https://scholarworks.iupui.
edu/handle/1805/344
Gorghiu, G., Iordache, D. D., Pribeanu, C., & Lamanauskas, V. (2018). Educational potential of
online social networks: Gender and cross-country analysis. Problems of Education in the
21st Century, 76 (5), 620-632.
Gülbahar, Y. (2014). Current state of usage of social media for education: Case of Turkey. Journal of
Social Media Studies, 1(1), 53-59.
Guo, B., Aveyard, P., Fielding, A., & Sutton, S. (2008). Testing the convergent and discriminant validity
of the decisional balance scale of the transtheoretical model sing the multi-trait multi-method
approach. Psychology of Addictive Behaviour, 22(2), 288-294.
Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2009). Multivariate data analysis (7th Ed.).
Prentice Hall.
Hoe, S. L. (2008). Issues and procedures in adopting structural equation modeling technique. Journal of
Applied Quantitative Methods, 3(1), 76-83.
Hooper, D., Coughlan, J., & Mullen, M. (2008). Structural equation modelling: Guidelines for determining
model-fit. Electronic Journal of Business Research Methods, 6(1), 53-60.
Ilogho, J. E. (2015). The role of picture books in promoting reading culture among Nigerian children:
Implication for libraries and development of leadership qualities. International Journal of
Academic Library and Information Science and Technology, 3(2), 65-71.
Jaja, J. M. (2013). Higher education in Nigeria: Its gain, its burden. Global Journal of Human and Social
Science Linguistics and Education 13, (14), 20-29.
Karpinski, A. C, & Duberstein, A. (2009). A description of Facebook use and academic performance
among undergraduate and graduate students. Research News. http://researchnews.osu.edu/
archive/facebookusers.htm
Klopfer, E., Osterweil, S., Groff, J., & Haas, J. (2009). Using technology of today in the classroom of
today. The education arcade website: http://creativecommons.org/licenses/by/3.0
Kumar, R. (2011). Research methodology - A step-by-step for beginners (3rd Ed.). SAGE Publications.
Lin, H.-H., Wang, Y.-S., & Li, C.-R. (2016). Assessing mobile learning systems success. International
Journal of Information and Education Technology, 6(7), 576-579. https://doi.org/10.7763/
IJIET.2016.V6.754
MacCallum, R. C., Widaman, K. F., Preacher, K., & Hong, S. (2001). Sample size in factor analysis: The
role of model error. Multivariate Behavioral Research, 36, 611-637.
Mazman, S. G., & Usluel, Y. K. (2010). The usage of social networks in educational context. World
Academy of Science, Engineering and Technology, 3(1), 338-342.
Ogunyinka, E. K., Okeke, T. I., & Adedoyi, R. C. (2015). Teacher education and development in Nigeria:
An analysis of reforms, challenges and prospects. Education Journal, 4(3), 111-122.
Okojie, J. (2013). Quality assurance and the challenges of mandate delivery in Nigerian Universities.
Paper presented at the 18th Convocation Lecture, Lagos State University, Lagos Nigeria.
Osang, F. B., & Ngole, J. (2014). Mobile learning implementation: Readiness of the National Open
University of Nigeria. International Journal of Emerging Technology and Advanced Engineering,
4(11), 25-32.
Otoo, E. (2015). Effects of Boko Haram on Nigeria’s children. http://www.globalfundforchildren.org/
effects-of-boko-haram-on-nigerias-children/
Pallant, J. (2011). SPSS survival manual: A step by step guide to data analysis using SPSS (4th Ed.). Allen
& Unwin.
Pasek, J., More, E., & Hargittai, E. (2009). Facebook and academic performance: Reconciling a media
sensation with data. First Monday, 14(5). http://firstmonday.org/article/view/2498/2181
Schreiber, J. B., Nora, A., Stage, F. K., Barlow, E. A., & King, J. (2006). Reporting structural equation
modelling and confirmatory factor analysis results: A review. The Journal of Educational
Research, 99(6), 323-337.
Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data
Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack
(pp. 115-135). University of Saarland Press.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Selwyn, N. (2012). Social media in higher education. http://www.educationarena.com/pdf/sample/ 47
sample-essay-selwyn.pdf
Serbessa, D. D. (2006). Tension between traditional and modern teaching-learning approaches in
Ethiopian primary schools. Journal of International Cooperation in Education, 9(1), 123-140.
Shonola, A. S., & Mike, S. J. (2014). Mobile learning security concerns from university students’
perspectives. In: Proceedings of 2014 International Conference on Interactive Mobile
Communications Technologies and Learning. Thessaloniki, Greece. Nove 13-14.
Shembilu, A. (2013). Importance of social networking for student participation in education in Tanzania.
Blekinge Institute of Technology, Karlskrona, Sweden.
Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th Ed.). Pearson Education.
Tella, A. (2011). Availability and use of ICT in south-western Nigeria colleges of education. International
Multidisciplinary Journal, Ethiopia, 5(5), 315-331.
Thomas, T. D., Singh, L., & Gaffar, K. (2013). The utility of the UTAUT model in explaining mobile
learning adoption in higher education in Guyana. International Journal of Education and
Development using Information and Communication Technology, 9(3), 71-85.
Traxler, J. (2007). Defining, discussing, and evaluating mobile learning: The moving finger writes and
having writ…. International Review of Research in Open and Distance Learning, 8(2), 1-12.
https://doi.org/10.19173/irrodl.v8i2.346
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information
technology: Toward a unified view. MIS Quarterly, 27(3), 425-478.
Venkatesh, V., Thong, J. Y., & Xu, X. (2012). Consumer acceptance and use of information technology:
Extending the unified theory and use of technology. MIS Quarterly, 36(1), 157-178.
Walker, R., Voce, J., Nicholls, J., Swift, E., Ahmed, J., & Horrigan, S. (2014). 2014 survey of technology
enhanced learning for higher education in the UK. Universities and Colleges Information Systems
Association.
Yapici, I. U., & Havedanli, M. (2014). Educational use of social networks: Facebook case study. European
Journal of Research on Education, 16-21.
Cite as: Chaka, J. G., & Govender, I. (2020). Implementation of mobile learning using a
social network platform: Facebook. Problems of Education in the 21st Century, 78(1), 24-47.
https://doi.org/10.33225/pec/20.78.24
John Chaka PhD, Lecturer, Department Computer Science, Federal College of Education,
Pankshin, Plateau State, Nigeria.
E-mail: chakajohn@yahoo.com
ORCID: http://orcid.org/0000-0003-1483-4959
Irene Govender PhD, Professor, University of KwaZulu-Natal, Westville, South Africa.

(Corresponding author) E-mail: govenderi4@ukzn.ac.za

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
48
THE RELATION BETWEEN SCHOOL
CULTURE SUB-CATEGORIES AND
EXPECTED RESULTS OF
LEARNING PROCESS
Ludvík Eger
University of West Bohemia, Czech Republic
E-mail: leger@kmo.zcu.cz
Michaela Prášilová
Palacký University, Czech Republic
E-mail: michaela.prasilova@upol.cz
Abstract
The aim of the research was to find out what factors of school culture affect the expected results of
teaching and learning processes in basic schools. The research evaluated current and desired school
culture in selected basic schools to identify culture gaps. Research using School Culture Inventory
explored relations among selected sub-categories, namely between the sub-categories focused on shared
objectives, trust in school leadership, and on managerial approach and the important sub-categories
focused on innovation process and expected results of teaching and learning processes. The independent
evaluators were final year students of the Faculties of Education from two universities in the Czech
Republic who assessed schools after one month of their position at the selected schools (the research
sample included 182 basic schools). The findings provided the evidence of current culture gaps and of a
positive association between higher positive evaluations in the sub-category leadership and management
and the sub-category expected results of teaching and learning processes. The results brought by the
research will help school leaders when they plan to shape or change the culture of the schools they
manage.
Keywords: culture gaps, school culture, school culture inventory, shaping school culture.
Introduction
School culture includes everything in a school’s surroundings that is made by human

beings, and includes tangible items as well as intangible concepts (Kulhavy, 1990) and values.
Deal and Peterson (1999) stated that culture is everything within a school: the way people
act, how they dress, what they talk about or avoid talking about, whether or not they seek out
colleagues for help, how teachers feel about their work and their students, and how students
feel about the school. Oldroyd et al. (1996, p. 49) defined organizational culture as ‘the values,
norms and beliefs which characterize organizations and are symbolized through patterns of
interaction, dress, speech and shared by all members or, as sub-cultures, by some groups of
members’.
There is increasing evidence that leadership makes a difference in schools (e.g. Hallinger
& Heck, 1996; Lindahl, 2010; Louis et al., 2010, Osiname, 2018). Scholars have made sustained
contributions to the question of how the behaviors of school leaders contribute to student

Ludvík EGER, Michaela PRÁŠILOVÁ. The relation between school culture sub-categories and expected results of learning process
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
achievement? Research studies typically examined a limited range of leadership behaviors, 49

thus making comparisons across studies is difficult. Louis et al. (2010) have called for further
research to examine the specific leadership behaviors that could be most effective in supporting
teaching and learning process.
The aim of the research was to explore relations among selected sub-categories
(dimensions) of school culture (in the context of Czech basic schools) and to understand
how selected characteristics of management and leadership influence perceived results of the
teaching and learning processes.
Research Problem
In general, the school leader's mission is to lead the school in the right direction and also
to motivate all the principal actors to do so. The school head then, as a manager and leader,
focuses on setting goals that are given by the educational strategy and other documents, and on
accomplishing them together with other people and in cooperation with them.
In the Czech Republic (CR), during the last 30 years, large investments have been
made toward the improvement of Czech schools (ISCED 1-3). The system of education in
the CR underwent major changes after the Act of 2004 (EURYDICE, 2019), which sets out
the principles and aims of education and the structure of the educational system. The process
of curriculum reform in basic schools and secondary schools has been in progress since 2005
and is currently at the stage of discussions about results in this area and comparisons with
international standards. The successes of specific schools vary in terms of quality of teaching
and students’ academic performance. In particular, the Czech School Inspectorate (ČŠI, 2017-
2019) pointed out in its thematically focused inspections and analyses that not all schools
achieve the required results, that there are differences between schools, and that sometimes
certain schools are lagging behind in international comparison to other countries.
The system of school management has also significantly changed. ‘The powers and
responsibilities of school heads have increased the introduction of the concept of legal entity’
(Schratz, et al., 2009). The autonomy and responsibility of Czech head teachers is among the
higher percentile in OECD countries (Schleicher, 2012). On other hand, head teachers serve as
statutory representatives of schools and are responsible not only for the teaching and learning
processes but also for financing, internal legislature, cleaning and maintenance of equipment,
etc. (OECD, 2016). They are also accountable for the development of pedagogical staff and
responsible for communication with parents and cooperation with other stakeholders (Eger
et al., 2017). A highly discussed topic is the extent to which school management affects the
teaching and learning processes and school performance. The research presented focuses on
this topic that holds importance not only for the CR, but also to a wider context (c.f. Bush, 2013;
Leithwood & Jantzi, 2006; Obdržálek & Polák, 2008; Pont et al., 2008).
Research Focus
Over the last decades, many researchers (e.g. Cameron & Quinn, 2011; Deal &
Peterson, 1999; Kotler, 2003; Peters & Waterman, 1982; Schein, 1985) have attempted to
define organizational culture and explain its essence, features, and influence on organizational
development and performance. School reforms cannot be implemented without the successful
implementation of changes to the school culture. After a wide analysis of literature (Shahzad et
al., 2012) was found that organizational culture has a deep impact on a variety of organizational
processes, including employees (teachers) and business (schools) performance. Managers
and leaders are advised to develop strong cultures in their organizations to achieve higher
commitment and improve the overall performance of the organization (Owoyemi & Ekwoaba,
2014).

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
50 Most studies on school culture have been conducted in Western countries. In the CR, the
first model of school culture was created by Světlík (1996). For example, research in this area
from new EU countries admitted in 2004 is represented by Pol et al. (2005), Eger (2010), and
Moree (2013) from the Czech Republic, Obdržálek and Polák (2008), Janovská et al. (2016),
Hašková and Bitterová (2018) from Slovakia, and Elsner (1999), Dzierzgowska (2000) and
Pyżalski (2014) from Poland.
School leaders are seen as the key factor that influences the shaping of the particular
school culture. For example, according to Leitwood and Riehl (2003), leadership is the most
significant of all factors and represents nearly one-quarter of the total effect of all school factors.
Similarly, Deal and Peterson (2009, p. ix) underlined the role of school leaders and argue,
‘successful school cultures have leaders who can cope with the paradoxes of their work and take
advantage of the opportunities of the future.’ Further, Oplatka and Hemsley‐Brown (2007, p.
303) declared, ‘principals have a major role in the changing of the school culture.’ Peterson and
Deal (1998) emphasized that the role of school leaders in the crafting of cultures is pervasive
and creating a school culture is the responsibility of the school leader (Peterson & Deal, 1998;
Turan & Bektas, 2013).
The following research questions were formulated based on the literature mentioned and
previous research results.
Q1: What are the main cultural gaps in basic schools in the two selected regions?
Q2: Will selected sub-categories of the School Culture Inventory have a positive impact
on the important sub-categories focused on innovation process and expected results of teaching
and learning processes?
Research Methodology
Research Design Overview
School culture is a multifaceted concept, composed of different dimensions (e.g. Higgins-

D'Alessandro & Sadh, 1997; Hinde, 2004; Zhu et al., 2009). Maslowski (2005) conducted a
critical review of previous school culture inventories and declared that questionnaires can be
a valuable tool in diagnosing school cultures. For example, School Culture Scales’ Properties
(Dumay, 2009) consisted of the areas of collaboration and innovation, and laid emphasis on the
discipline and academics but the research was focused on the consequences between schools’
cultural and mathematical achievements. Another School Culture Scale was used to measure five
school culture dimensions with regard to goal orientation, leadership, innovation orientation,
participative decision-making, and formal relationships (Zhu et al., 2014). The model proposed
by Bell and Kent (2010) recognized the dynamics and importance of both external and internal
organizational forces in shaping the culture of schools.
Conducted research focused on the dimensions of school culture within Czech schools. As
is mentioned above, the theoretical construct of this research was prepared on the basis of the
literature review (among others, Bush, 2003; Everard et al., 2004; Fidler, 2002; Gruenert, 2000;
Maslowski, 2006; Peterson & Deal, 1998).
This research attempted to explore empirically school culture using the quantitative
research design (Creswell, 2014). The quantitative approach used the School Culture Inventory
to gather data with the aim to find out what factors of school culture affect the expected results
of teaching and learning processes in selected basic schools. The research also analyzed how
selected characteristics of management and leadership influence the perceived results of the
teaching and learning processes.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Sample 51
The research sample included 182 basic schools (ISCED 1 and 2) evaluated in two
regions. In the Pilsen region (Western part of the CR) 98 basic schools and in the Olomouc
region (Eastern part of the CR) 84 basic schools were evaluated. Data for this research came
from students = evaluators (convenience sample) who were attending school management
course (see below) in both public universities, that represent centers for teacher development in
both regions, and who evaluated the selected schools.
Instrument
The school cultures were examined using the School Culture Inventory developed by
Eger and Jakubíková (2001) and revised by Eger (2006). The instrument has been used in self-
evaluation activities in the CR since 2001 in diverse school settings. For example, using this
instrument, Hornáčková et al. (2014) conducted research focused on kindergartens and Eger
(2010) on primary and secondary schools. Currently this tool is used for self-evaluation of
school culture in the project conducted by the National Institute for Further Education (2017),
where the module focused on school culture is the starting point of training about school strategy.
The first inspiration for the development of this tool came from publications by Everard
and Morris (1996); Everard et al. (2004); and a description of Handy´s four culture models by
Bush (2003). A tool for school culture assessment by Fidler (2002) was also taken into account
in the preparation of the second version of this School Culture Inventory. School Culture
Inventory contains 16 items, which were developed to assess school culture in important
domains by creating sub-categories and possible variables of school culture.
The construct of the questionnaire is consistent with similar research, e.g. Gruenert
(2000), followed by Turan and Bektas (2013), or with key school culture traits by Sukkyung
et al. (2017). Some items partly align with research by Leithwood and Jantzi (2006), who used
robust tools in their research aimed at transformational school leadership.
The school culture was evaluated in five selected sub-categories. The first sub-category,
leadership and management, contains four items focused on shared objectives, trust in school
leadership, and managerial approach, that is, task management and control. The second sub-
category, school environment, contains three items focused on organizational structure and
delegation, working conditions, and aesthetic environment and cleanliness. The third sub-
category, communication, contains three items focused on motivation of teachers, information
exchange, and communication with parents and stakeholders. The fourth sub-category,
relationships within the school, contains three items focused on supportive leadership style,
relationships and teacher collaboration, and teacher-student relationships. The fifth sub-category,
focused on innovation process and expected results of teaching and learning processes, contains
three items, two of which are mentioned above. Teacher development, as results of leadership
in staff development, and as a precondition for innovation process and expected performance
in education of each school, is also included in this sub-category. These five sub-categories and
their relationships constitute the framework for our evaluation of school culture.
The School Culture Inventory uses five-point Likert scale for a more detailed description
of each school culture item. Each point in each item had a short, detailed description. For
example, item 1, shared objectives, had a detailed description for point 1 = no awareness
of school objectives, no planning, and its opposite for point 5 = full knowledge of shared
objectives, clear orientation, common planning. Item 16 had a detailed description for point
1 = no good results expected, no interest and, its opposite for point 5 = high expectations of
excellent teaching and learning results.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
52 It was very important that the questionnaire worked to identify the so-called culture
gaps (Kilmann-Saxton, c.f. Peters & Waterman, 1982) between current (existing) cultures and
desired school cultures. Information about culture gaps is very important when school leaders
plan to shape positive culture, including positive climate (c.f. Leithwood et al., 2006; Peterson
& Deal, 1998).
Procedure
Because the research was not aimed at evaluating one school in depth, but at identifying
the current state of organizational culture in selected schools and evaluating the outcomes of
their assessment in selected regions, the evaluators were final year students of the Faculties of
Education from two universities. This indicates that the evaluators were almost novice teachers
who were not directly tied to the evaluated school, but they assessed schools after one month
of their position at the selected school. During this one month they repeatedly met with the
school management, studied the school's pedagogical documentation, collaborated with their
trainer (an experienced teacher) and other teachers, and participated in classroom observations.
However, they also directly taught students, participated in other school activities, perceived
the school climate and communicated with other school staff or parents of students, or at least
observed communication of teachers at school and the level of cooperation in the school, with
parents, etc. Consequently, the evaluators of school culture were partly independent, but at the
same time they ‘lived in the school’ for one month and could reasonably recognize its culture
as participants.
The research was coordinated by two experienced academic experts in a subject focused
on school management in the academic year 2018/2019. The questionnaires were administered
directly at universities. The evaluators were asked to indicate their familiarity with items of the
School Culture Inventory (current state and desired state for each item). It means, one student,
who was a member of school community one month, evaluated this school using his or her
experience described above. With a focus on the desired state of school culture, the task was to
determine what was the desired version by the members of the school community.
Data Analysis

The data analysis was done as follows. First, a descriptive and correlational analysis was
conducted to explore the actual level of school culture in basic schools in the selected regions.
It applied the identification of culture gap (Kilmann-Saxton) between the current (existing)
culture and desired school culture using descriptive statistics. Correlations were examined to
evaluate the relations between selected sub-categories of School Culture Inventory. Second,
the comparison between the sub-category focused on leadership and management and the sub-
category focused on innovation process and expected results of teaching and learning processes
was investigated using the Mann–Whitney U test. According to Kolmogorov-Smirnov test,
the variables were non-normally distributed recommending nonparametric statistical tests
(Jackson, 2016). Data were analyzed using MS Excel and Statistica software.
Research Results
Full-time students of the final year course at the Faculty of Education measured school
cultures using School Culture Inventory, after completing one month of practice at the schools.
The questionnaire consisted of 16 items and the evaluators were asked to indicate a current state
and a desired state for each item.
First, the means and standard deviations for each item were calculated.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Table 1 53
List of items, School Culture Inventory
Items Basic schools (n = 182)

Current culture Desired culture
Mean SD Mean SD
1 Shared objectives 3.4 0.8 4.5 0.6
2 Trust in school leadership 3.8 0.8 4.7 0.5
3 Supportive leadership style 3.4 0.9 4.3 0.7
4 Organizational structure and delegation 3.9 0.7 4.7 0.5
5 Managerial approach – task management 3.5 0.8 4.1 0.6
6 Managerial approach - control 3.6 1.0 4.6 0.6
7 Motivation of teachers 3.2 0.8 4.6 0.5
8 Communication in the school and information exchange 3.8 0.8 4.7 0.5
9 Communication with parents and stakeholders 3.8 0.7 4.6 0.5
10 Innovation of teaching and learning processes 3.3 0.8 4.2 0.5
11 Teacher development 3.7 0.8 4.5 0.6
12 Working conditions at the school 3.6 0.7 4.7 0.6
13 Aesthetic environment and cleanliness 3.7 0.6 4.3 0.6
14 Collaboration among teachers in the school 3.8 1.0 4.7 0.6
15 Teacher-student relationships 3.6 0.8 4.9 0.6
16 Expectation of education results 3.1 0.6 3.9 0.6
Note. SD = standard deviation. Cronbach´s alpha for overall score of School Culture Inventory was .87
for the sample of basic schools from Pilsen, and .80 for the sample of basic schools from Olomouc.
As is mentioned above, school culture is a variable from school to school. However, for
the purpose of this research, to compare current school cultures, the culture gaps are presented
in Figure 1. Identification of culture gaps (Kilmann-Saxton) between current cultures and
desired school cultures (Table 1 and Figure 1) showed where the problems lie in school cultures
and where the opportunities exist for school leaders when they are thinking generally about
school improvement.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
54 Figure 1
Kilmann-Saxton culture gaps, basic schools in the Czech Republic (n = 182)
The main culture gaps were identified in the following items: Motivation of teachers
(1.4), shared objectives (1.1), managerial approach – control, and Innovation of teaching and
learning processes (1.0).
To assess the relations between the five selected sub-categories = variables (leadership
and management, school environment, communication, relationships within the school,
innovation process and expected results), bivariate Pearson correlation analysis was carried
out. The correlation matrix was used to investigate the dependence between variables from our
construct, or in other words, between sub-categories of School Culture Inventory.
Table 2
Bivariate correlations among variables used in analysis, basic schools (n = 182)
Sub-categories of school culture 1 2 3 4 5

1 Leadership and management
2 School environment 0.47
3 Communication, including teacher motivation 0.58 0.50
4 Relationships within the school 0.48 0.45 0.60

5 Innovation process and expected results 0.59 0.43 0.47 0.40
Mean 3.6 3.8 3.6 3.6 3.4
SD 0.57 0.49 0.58 0.63 0.56
Note. p < .05
As can be seen in Table 2, there were moderate positive relationships between the
subcategories leadership and management and communication, including teacher motivation (r
= .58) and between leadership and management, and the very important subcategory innovation
process and expected results (r = .59). A higher degree of coefficient was found between the
subcategory communication, including teacher motivation and the sub-category relationships
within the school (r = .60).

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
From this point of view, supportive leadership and management were positively 55
associated with communication, including teacher motivation and also positive and moderate
relationship was found with the sub-category innovation process and expected results. Outputs
(Tables 2) signal the moderate positive association between the subcategories, communication,
including teacher motivation, and relationships within the school, in current school cultures in
the surveyed schools. This result is also the evidence for the importance of paying attention to
communication within the school.
To compare difference between the sub-category leadership and management and the
sub-category innovation process and expected results of teaching and learning processes, the
Mann–Whitney U test was applied between the two groups of schools according to their results
in the sub-category leadership and management. The schools were split in half according to the
evaluation results in this category and the test was used to verify the null hypothesis.
H10 There is no association between the positive evaluation of school culture in the sub-
category leadership and management and positive expected results in the sub-category focused
on innovation process and expected results of teaching and learning processes.
H1A Schools with a positive evaluation of school culture in the sub-category leadership
and management reach significantly higher positive evaluations in the sub-category focused on
expected results.
Figure 2 below shows the distribution of evaluation results in the sub-category innovation
process and expected results of teaching and learning processes according to the two groups of
basic schools. The first group of schools represents half of the basic schools surveyed (n = 91),
with higher evaluations in the sub-category leadership and management. The second one is the
opposite group.
Figure 2
Two groups of basic schools divided according to leadership and management
Median
25%-75%
Min-Max
Outllier
Extreme outlier
Figure 2 uses boxplot visualization to graphically show the distribution of the results in
the sub-category innovation process and expected results of teaching and learning processes for
two groups of basic schools from the selected sample. Spacing between the different parts of
the presented boxplot indicates the degree of dispersion and skewness in the data and displays
outliers. The median for groups with higher evaluations in the sub-category leadership and
management was 3.67, and there was only one outlier, 2.33. The median for the second group
of basic schools is 3.0. The box that comprises the middle 50 % of results is smaller than the
first group. The second group showed more outliers, both above and under the box. While the
values above were within the first group, the values below point to a very negative evaluation
of the five schools in this sub-category. It could be interpreted as a call for help.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
56 To test whether there was a statistically significant difference between the two groups
of basic schools in terms of their evaluation in the sub-category leadership and management
to the results in the sub-category innovation process and the expected results of teaching and
learning processes, the statistical non-parametric approach was used, the Mann–Whitney U-test.
Following the Mann–Whitney results (Z-Score = -2.02339, p = .043033 < .05), Hypothesis H10
was rejected and accepted the alternative hypothesis H1A, which indicates that there was found
significant difference between the two groups of schools in the innovation process and the
expected results of teaching and learning processes.
Discussion
The results of this empirical research showed that culture gaps in Czech schools were
found primarily in the items motivation of teachers and shared objectives. School leaders should
also pay attention to the items in the second place, namely, managerial approach – control,
innovation of teaching and learning processes, and communication, both internal and external.
This research also explored the relationships between the five main sub-categories using
School Culture Inventory (Table 2). As is mentioned above, not only in the CR, experts in other
countries too are looking for ways to improve schools and their performance. The important
question is, what kind of role does school management and leadership play? The presented
results are in line with the theory proposed by Bush (2013), Leithwood et al. (2006), and
Peterson and Deal (1998) and confirm the important role of school management and leadership
in relation to the expected performance of the school. The research found positive and significant
relationships between the sub-category leadership and management and the important sub-
categories that are focused on the innovation process at schools and the expected results of
teaching and learning processes. This finding is in line with the results of research conducted
by Louis et al. (2010) for variables shared leadership and trust. The sub-category of School
Culture Inventory aimed at performance also contains an item focused on teacher development.
This item is considered as a precondition for innovation process and expected performance in
the education of each school. In practice, this item is in relationship with the activity of school
leaders and their care for the most important resource of the school. The boxplot (Figure 2)
graphically showed the distribution of the results in the sub-category innovation process and
the expected results of teaching and learning processes for two selected groups of basic schools.
A positive evaluation of school management and leadership leads to higher evaluations in the
sub-categories focused on innovation process at schools and expected results of teaching and
learning processes.
The results are also in line with the findings by Osiname (2018) who stated that successful
leaders achieve their goals by collaborating and communicating with all stakeholders to establish
the appropriate vision for their schools and support everyone and create a school community
that is safe for teachers and students.
As expected, moderate positive association was found between the sub-category
communication, including teacher motivation and relationships within the school. This finding
is in line with the results of the research conducted for example by Janovská et al. (2016) in
Slovakia, but does not correlate with the findings by You, Kim, and Lim (2017) in similar
areas (academic climate and colleague support) in Korea. Similarly, Zhu et al. (2014) found
differences between Flemish and Chinese schools regarding school culture dimensions. These
findings showed that it will be important for future research to take into account the cultural
context in particular countries.
Cameron and Quinn (2011) summarized scientific studies that report a positive
relationship between dimensions of organizational culture and organizational effectiveness. It
is obvious, that this topic is also important in education area where on national and on an

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
international level government and other institutions search ways how to improve schools and 57
how to support the educational process (e.g. European Commission, 2018). For those interested
in empirical evidence that supports the evaluation procedures and culture change methodology
explained conducted research how to apply Kilmann-Saxton culture gap in practice. This
research also illuminates the key elements of school culture (cf. Lee & Louis, 2019; Maslowski,
2006; Peterson & Deal, 1998) that have been linked with sustainable school improvement.
The main limitation of the conducted research was that the results are relevant to basic
schools in the Czech Republic. However, the results are important for understanding school
culture and leaders’ role in shaping school culture from an international point of view.
Two methodological limitations of the research warrant caution in the interpretation
of the results. First, the evaluators of school culture were full-time final year students of the
Faculty of Education, and they completed questionnaires after one month of practice at their
designated schools. On one hand, they were partly independent observers, but had experience
at the evaluated schools. On other hand they only had teaching experience as novice teachers. A
second methodological limitation of the research was a measure of the sub-categories innovation
process and the expected results of teaching and learning processes. The performance of the
school better represents data from, for example, TIMSS or similar national comparisons of
learning outcomes.
Conclusions and Implications
This research extends and deepens understanding of culture gaps in schools and shows
the suitability of evaluating not only current, but also desired, school culture. It is hoped that
the conducted research provides new knowledge for improving the professional development
of head teachers based on the findings from the application of School Culture Inventory.
The practical application of the conducted research is in recognizing the current situation
of school cultures, including culture gaps. The findings provide some insights about current
situation of the sample schools, which can be helpful for forming recommendations for school
culture construction. The presented School Culture Inventory is a useful tool for auto-evaluation
for some schools, and the findings should be used when school leaders plan to shape or to
change the culture of the schools they manage.
The results have important implications for recognizing the importance of the role of
school managers and leaders in relation to the expected results of teaching and learning processes.
By confirming that schools with higher positive evaluation in the sub-category leadership and
management reach higher positive evaluations in the sub-category expected results of teaching
and learning processes, the research not only explains the role of school managers and leaders,
but also brings new knowledge for shaping school culture. School culture is a key component
of the school improvement process.
References
Bell, L., & Kent, P. (2010). The Cultural Jigsaw: A case study exploring the ways in which sixth-form
students perceive school culture. Educational Management Administration & Leadership, 38(1),
8-32. https://doi:10.1177/1741143209351663
Bush, T. (2003). Theories of educational leadership and management. SAGE.
Bush, T. (2013). Distributed leadership: The model of choice in the 21st century. Educational Management
Administration & Leadership, 41(5), 543–544. https://doi:10.1177/1741143213489497
Cameron, K. S., & Quinn, R. E. (2011). Diagnosing and changing organizational culture. Jossey-Bass.
Creswell, J. W. (2014). Research design. SAGE.
ČŠI (2017-2019). [Czech School Inspectorate], Tematické zprávy [Reports focused on specific topics].
https://www.csicr.cz/cz/DOKUMENTY/Tematicke-zpravy

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
58 Deal, T. E., & Peterson, K. (1999). Shaping school culture. Jossey-Bass.

Deal, T. E., & Peterson, K. (2009). Shaping school culture: Pitfalls, paradoxes, & promises. Jossey-Bass.
Dumay, X. (2009). Origins and consequences of schools’ organizational culture for student achievement
Educational Administration Quarterly, 45(4), 523-555. https://doi:10.1177/0013161X09335873
Dzierzgowska, I. (2000). Dyrektor w zreformowanej szkole. [Head teacher in school after school reform].
Wydawnictwa CODN.
Eger, L. (2006). Řízení školy při zavádění školního vzdělávacího programu. [Managing school and
implementing new curriculum]. Fraus.
Eger, L. (2010). Application of the corporate culture in the field of school management. E+M, Ekonomie
a Management, 13(3), 26-32.
Eger, L., & Jakubíková, D. (2001). Kultura školy. [School culture] TU Liberec.
Eger, L., Pisoňová, M., & Tomczyk, L. (2017). Development programs for head teachers in four Central
European countries: An international comparison. International Journal of Management in
Education, 11(1), 25-45. https://doi:10.1504/IJMIE.2017.080643
Elsner, D. (1999). Doskonalenie kierowania placówka oświatowa. [Improvement of educational
institution]. Mentor.
European Commission (2018, April). European ideas for better learning: the governance of school
education systems. The final report and thematic outputs of the ET2020 working group schools.
Brussels: Directorate-General Education, Youth, Sport and Culture.
https://www.schooleducationgateway.eu/downloads/Governance/2018-wgs6-Full-Final-Output.pdf
EURYDICE (2019, May 7). Czech Republic overview. Organization and governance. https://eacea.
ec.europa.eu/national-policies/eurydice/content/organisation-and-governance-21_en
Everard, K. B., & Morris, G. (1996). Effective school management. Paul Chapman Publishing.
Everard, K. B., Morris, G., & Wilson, I. (2004). Effective school management. Paul Chapman Publishing.
Fidler, B. (2002). Strategic management for school development. Paul Chapman Publishing.
Gruenert, S. (2000). Shaping a new school culture. Contemporary Education 1(2), 14-17.
Hallinger, P., & Heck, H. R. (1996). Reassessing the principal's role in school effectiveness: A review of
empirical research, 1980-1995. Educational Administration Quarterly, 32(1), 5-44. https://doi:1
0.1177/0013161X96032001002
Hašková, A., & Bitterová, M. (2018). School autonomy and school leadership: Case study of school
operation in Slovakia. Problems of Education in the 21st Century, 76(3), 299-308.
Hinde, R. E. (2004, Decenber). School culture and change: An examination of the effects of school
culture on the process of change. Essays in Education 11. http://citeseerx.ist.psu.edu/viewdoc/
download?doi=10.1.1.404.6474&rep=rep1&type=pdf
Higgins-D'Alessandro, A., & Sadh, D. (1998). The dimensions and measurement of school culture:
Understanding school culture as the basis for school reform. International Journal of Educational
Research, 27(7), 553-569. https://doi:10.1016/S0883-0355(97)00054-2
Hornáčková, V., Princová, K., & Šimková, T. (2014). Analysis and evaluation of school culture in
kindergartens. Procedia - Social and Behavioral Sciences, 112, 916-923. https://doi:10.1016/j.
sbspro.2014.01.1249
Jackson, L. S. (2016). Research methods and statistics. Cengane Learning.
Janovská, A., Orosová, O. & Janovský, J. (2016.) Head teacher´s social support, personality variables and
subjective well-being of Slovak primary teachers. Orbis Scholae, 10(3), 71-87.
Kotler, P. (2003). Marketing management. Prentice Hall.
Kulhavy, E. (1990). Skizzen zum Marketing. [Marketing notes]. Universitätverlag Rudolf Trauner.
Lee, M., & Louis, K. S. (2019). Mapping a strong school culture and linking it to sustainable school
improvement. Teaching and Teacher Education, 81, 84-96. https://doi:10.1016/j.tate.2019.02.001
Leithwood, K., Day, Ch., Sammons, P., Harris, A., & Hopkins, D. (2006). Successful school leadership:
What it is and how it influences pupil learning. DfES Publications.
Leithwood, K., & Jantzi, D. (2006). Transformational school leadership for large-scale reform: Effects on
students, teachers, and their classroom practices. School Effectiveness and School Improvement,
17(2), 201-227. https://doi:10.1080/09243450600565829
Leithwood, K., & Riehl, C. (2003, January). What we know about successful school leadership.
Nottingham: National College for School Leadership. http://olms.cte.jhu.edu/olms2/data/ck/file/
What_we_know_about_SchoolLeadership.pdf

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Lindahl, A. R. (2010). Differences in principals' leadership behavior in high‐ and low‐performing schools. 59
Journal of Leadership Studies, 3(4), 34-45. https://doi:10.1002/jls.20137
Louis, S. K., Dretzke, B., & Wahlstrom, K. (2010). How does leadership affect student achievement?
Results from a national US survey. School Effectiveness and School Improvement, 21(3), 315-336.
Maslowski, R. (2006). A review of inventories for diagnosing school culture. Journal of Educational
Administration, 44(1), 6-35. https://doi:10.1108/09578230610642638
Moree, D. (2013). Teachers and school culture in the Czech Republic before and after 1989. Curriculum
Journal, 24(4), 586-608.
Národní institut pro další vzdělávání. (2017, Decenber). [National Institute for Further Education],
Strategické řízení a plánování ve školách a v územích [Strategic management for planning at
schools and in regions]. https://www.nidv.cz/strategicke-rizeni
Obdržálek, Z., & Polák, J. (2008) Príprava školských manažérov ako kľúčový predpoklad efektívnosti
školy. [Training for school managers as prerequisite for effective school]. UKF v Nitre.
OECD (2016, October). Review of policies to improve the effectiveness of resource use in schools. Czech
Republic. http://www.oecd.org/education/school/CZE-CBR_FINAL-OCT2016.pdf
Oldroyd, D., Elsner, D., & Poster, C. (1996). Educational management today. Paul Chapman Publishing.
Oplatka, I., & Hemsley‐Brown, J. (2007). The incorporation of market orientation in the school culture:
An essential aspect of school marketing. International Journal of Educational Management,
21(4), 292-305. https://doi:10.1108/09513540710749519
Osiname, T. A. (2018). Utilizing the critical inclusive praxis: The voyage of five selected school
principals in building inclusive school cultures. Improving Schools, 21(1), 63-83. https://
doi:10.1177/1365480217717529
Owoyemi, O., & Ekwoaba, J. (2014). Organisational culture: A tool for management to control, motivate
and employees´ performance. American Journal of Business and Management, 3(3), 168-177.
https://doi:10.11634/216796061403514
Peters, T. J., & Waterman, R. H. (1982). In search of excellence: Lessons from America’s best-run
companies. Harper & Row.
Peterson, K., & Deal, T. (1998). How leaders influence the culture of schools. Educational Leadership,
56 (1), 28–30.
Pol, M., Hlušková, L., Novotný, P., & Zounek, J. (2005). Kutura školy. [School culture]. Masarykova
univerzita v Brne.
Pont, B., Nusche, D., & Hopkins, D. (Eds.) (2008). Improving school leadership. OECD Publishing.
Pyżalski, J. (2014). Analiza porôwnawcza systemôw ksztatcenia i doskonalenia kadry kierowniczej szkôt
oraz placôwek Raport syntetyczny. [Competencies of headmasters in EU countries - analysis].
Osrodek Rozwoju Edukacji, Wydzial Zarzqdzania i komunikacji spolecznej Uniwersytet
Jagielloński.
Schein, E. H. (1985). Organizational culture and leadership. John Wiley & Sons.
Schleicher, A. (Ed.) (2012). Preparing teachers and developing school leaders for the 21st century: Lessons
from around the world. OECD Publishing. https://www.oecd.org/site/eduistp2012/49850576.pdf
Schratz, M., Slaviková, L., Křížková, E., Kecliková, E., Szabó, M., Révész, É., Laššák, V., Hašková, A.,
Erčulj, J., Peček, P., & Halász, G. (2009). The role of school leadership in the improvement of
learning. Tempus Public Foundation.
Shahzad, F., Luqman, A, Khan, R. A., & Shabbir, L. (2012). Impact of organizational culture on
organizational performance: An overview. Interdisciplinary Journal of Contemporary Research
in Business, 3(9), 975-985.
Sukkyung, Y., Ann, Y., & Sun, A. L. (2017). Job satisfaction among, secondary teachers in Korea: Effects
of teachers´ sense of efficacy and school culture. Educational Management Administration &
Leadership, 45(2), 284-297. https://doi:10.1177/1741143215587311
Světlík, J. (1996). Marketing školy. [Marketing of school]. EKKA.
Turan, S., & Bektas, F. (2013). The relationship between school culture and leadership practices. Eurasian
Journal of Educational Research, 52, 155-168.
Zhu, Ch., Devos, G., & Tondeur, J. (2014). Examining school culture in Flemish and Chinese primary
schools. Educational Management Administration & Leadership, 42(4), 557-575. https://
doi:10.1177/1741143213502190

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
60
Received: October 31, 2019 Accepted: January 08, 2020
Cite as: Eger, L., & Prášilová, M. (2020). The relation between school culture sub-categories
and expected results of learning process. Problems of Education in the 21st Century, 78(1),
48-60. https://doi.org/10.33225/pec/20.78.48
Ludvík Eger Doc., PaedDr, CSc. Associate Professor, University of West Bohemia, Faculty of
(Corresponding author) Economics, Univerzitní 8, 301 00 Plzeň, Czech Republic.
E-mail: leger@kmo.zcu.cz
Website: https://www.fek.zcu.cz/osobni.php?IDWorker=115
ORCID: 0000-0002-5437-3297
Michaela Prášilová Doc., PhDr., PhD, Associate Professor, Palacký University, Faculty of Education,
Olomouc, Žižkovo nám. 5, 771 40 Olomouc, Czech Republic
E-mail: michaela.prasilova@upol.cz
Website: https://www.upss.cz/products/doc-phdr-michaela-prasilova-ph-d-/

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
COMPETENCY-BASED ASSESSMENT AND 61
THE USE OF PERFORMANCE-BASED

EVALUATION RUBRICS IN HIGHER
EDUCATION: CHALLENGES TOWARDS
THE NEXT DECADE
Katherina Gallardo
Monterrey Institute of Technology and Higher Education, Mexico
E-mail: katherina.gallardo@tec.mx
Abstract
Rubrics are assessment guides for grading and giving feedback to students while demonstrating acquired
knowledge and skills. In the last decades, rubrics have been all part of the most used learning evaluation
tools in higher education. Its use is further well-related to competency-based assessment purposes.
Nevertheless, criticism around design and application has been also implemented in certain reports.
In order to understand rubrics’ evolution, practice, benefits, and trends, a systematic literature review
on rubrics’ design and use has been conducted. Two databases were selected: Scopus and ProQuest
Education. Two phases were determined: The first allowed to identify articles related to rubrics’ design
and application for almost three decades. 584 articles were found. From these, most cited articles served
to give a scope of rubric evolution and trends. The second phase permitted to identify design and use
of performance-based evaluation rubrics from 2009 and 2019. Five terms and Boolean combinations
were used. A total of 259 articles was found. After analyzing abstracts and content, 11 open access
articles on performance-oriented rubric design and application were chosen. As a result, some facts
and reflections on rubric design complexity are discussed towards responding to educational challenges
related to competency-based assessment contemporary demands: integration of human learning domains
going beyond cognition, authenticity, and interdisciplinarity as features that characterize learning design
situations, follow students’ progression, educator’s preparation on assessment literacy for responding
to CBA demands, and the involvement of experts of the work field in determining essential evaluation
indicators as the main topics looking forward to the next decade.
Keywords: competency-based assessment, competency-based education, higher education, learning
taxonomy, rubric design.
Introduction
Competency-based assessment (CBA) has been a relevant educational topic for almost 25
years. It is common knowledge that its complex condition comes from its roots in competency-
based education (CBE) as an integrated paradigm derived from several pedagogical approaches.
Therefore, it is not possible to refer to a single entity, but rather, in practice, there are multiple
ways to define and operationalize it (Farris, Demb, Janke, Kelley, & Scott, 2009; Gervais, 2016;
Meyer-Adams, Potts, Koob, Dorsey, & Rosales, 2011; Rodriguez & Gallardo, 2017).
The International Bureau of Education of UNESCO emphasizes the complex outcomes
of a learning process that implies working within CBE, as knowledge, skills, and attitudes are
all part of the learning process, rather than focusing on traditionally defined subject content.
Thus, learners are expected to acquire and apply knowledge, skills, and attitudes to perform in

Katherina GALLARDO. Competency-based assessment and the use of performance-based evaluation rubrics in higher education:
Challenges towards the next decade
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
62 situations they encounter in everyday life (Escudero Escorza, 2016; Kuh, Jankowski, Ikenberry,
& Kinzie, 2014). This concern has been analyzed several times by educational authorities, in
order to fulfill the needs of governments, educational institutions, and teachers for planning,
implementing, and assessing performance. In fact, they are obliged to render counts to society
about how close students are in reaching the standards of stated competencies. Then, CBA
becomes fundamental for ensuring educational quality and accountability (Duran, Estay-
Niculcar, Alvarez, & Randolph, 2015; Koh, Tan, & Ng, 2012).
CBA is based mainly on performance and authentic assessment. It has been defined
as involving observation and assessment of students’ behavior while making things (Gao
& Grisham-Brown, 2011; Hancock, 2007). Students are asked to demonstrate that they are
capable of transferring knowledge, executing some skills in a certain complex situation in order
to solve problems, and taking decisions, or creating new alternatives within a given context.
This performance is generally observed in a semi-structured context or in real-life situations
where collaborative work is required. Some advantages of applying performance and authentic
assessment are: (a) giving more meaning to complex learning targets; (b) assessing students’
ability to make decisions and solve problems; (c) integrating knowledge, skills, and attitudes
into assessment and feedback; (d) giving the learning processes an important role rather than
only focusing on solutions and products; and (e) being consistent with philosophical and
pedagogical CBE basics.
Most of the time, discussions about implementing CBA focus on how to guarantee
that performance assessment would be done with feasibility and validity, without depending
just on standardized tests (Barber & Phillips, 2000; Duckworth & Yeager, 2016). Instead,
alternative and authentic assessment practices are required (Caliskan & Kasikci, 2010). Thus,
some alternative assessment practices emerge and are used by teachers: project-based learning,
challenge-based learning, portfolio, etc. Nevertheless, poor/fast teacher training and several
misunderstandings while designing and interpreting data through these assessment mechanisms
could be considered weak areas of CBA implementation.
Competency-based Assessment: Challenges in Times of Educational Change
Almost all societies are living in a period of rapid and disruptive educational development.
Having access to a high-quality education that considers human beings’ main concerns and
needs of development besides simple disciplinary knowledge, becomes part of worldwide
priorities to be solved in the future. European, Asian, Australian and Latin-American countries
are modifying their educational scope structurally in order to introduce CBE at different
levels: primary, secondary, vocational and tertiary education (Boukhentache, 2016; Voronov,
Chasovskykh, Popov, Anyanova, & Krainova, 2015).
Even if there has been a high interest in implementing CBE over these years as one
possible way to give answers to the Economy and Society Knowledge demands (Carrillo,
2015; Slaus, 2013), there are aspects that are not totally solved around how to assess students’
performance with accuracy and feasibility, in a world that has explained learning results based
simply on grades and statistics for years. Indeed, identifying and recognizing CBA advantages
does not automatically change traditional learning assessment practices. However, we cannot
deny that the scope is changing progressively.
For instance, the Australian Learning and Teaching Council has made some relevant
assessment decisions (Boud, 2009) related to strengthen CBA toward the year 2020: (a)
assessment is a central feature of teaching and the curriculum; (b) it is an accurate way to
understand how students learn and what students achieve; and (c) it is one of the most significant
influences on students’ experience, so it is mandatory to improve assessment practices due to its
huge impact on learning quality.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Another example of the CBA evolution is taking place in Bangladesh (Roy, 2016). The 63
Ministry of Education of this country is looking forward and moving faster to implement new
ways to improve the quality of basic education by evaluating learning achievement in terms of
terminal competencies, considering different school systems based on a nationally representative
sample, and examining the relationship between learning achievement and selected students’
characteristics, such as socioeconomic status, school-related variables, and receipt of extra
educational inputs. Thus, the government is applying different assessment strategies rather than
just examining and scoring content retention through standardized tests.
Moreover, it is understood that CBA not only has challenges to face and solve in the
implementation phase but also a number of issues to address for gaining credibility (Barber &
Phillips, 2000; Ellis et al., 2015). Among these main challenges, the most relevant are:
• Revising competencies and academic programs to make sure CBA is conducted with
coherence
• Deciding processes, terms, and parameters related to competence accomplishment
along with each program
• Counting on internal and external experts’ support for establishing valuable tasks
and projects
• Establishing collegiate agreements for the performance-based evaluation rubric
• Selecting meaningful standards for assessing and giving feedback.
Thus, it can be affirmed that in the eye of this hurricane of educational changes toward adopting
CBA, big efforts must be made to deliver a high-quality assessment process.
Rubric: More than 20 Years of Research
The operationalization of CBA into applied strategies and the use of different tools for
observing, collecting, and giving feedback about students’ performance have been studied and
enriched over the past few years. Undoubtedly, one of the most-used performance-assessment
tools is rubric. Its main intention is to describe what students are expected to demonstrate
through their performance. This is possible as long as a rubric contains a clear description of
intended and connected actions corresponding to the statement of criteria, and the standards and
levels of performance expected from the related competencies (Jonsson, 2014).
Rubrics for learning assessment purposes have been studied for several years. From the
first discussion about its use within holistic evaluation processes (Grubb, 1981), almost six
hundred articles have been published to discuss new knowledge on the design or use of this
tool in different disciplines (Elsevier, 2019). Thus, rubric has passed through a rapid evolution
process over the past few years (Bharuthram & Patel, 2017).
Rubric design has challenged educators and educational researchers to study, understand
and create different ways to answer one of the most difficult aspects of the learning process:
giving judgments about performance. These judgments can come from different actors involved
in the educational process, even from experts in the field of study who are asked to give their
opinion about students’ performance. Then, rubric is expected to be a simple, clear, accurate,
consistent and flexible tool for collecting diverse points of view related to certain performance
criteria, in a holistic or an analytic way (Mertler, 2001; Panadero & Jonsson, 2013).
Many efforts have been made lately to establish norms to improve rubrics’ design and
use from the technical point of view. Curiously, this norming trend comes from engineering
education initiatives. For instance, Goldberg (2014) discussed some common problems while
designing a rubric: lack of consistency and parallelism, detection of “orphan” and “widow”
words and phrases, redundancy in descriptors, inconsistency in focus of qualifiers, limited
routes to partial credits, unevenness in incremental levels of performance and inconsistencies
across the suite of rubrics. Additionally, Schoepp, Researcher, and Kranov (2018) established

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
64 a norm for designing accurate and valid rubrics focusing on a higher education context, which
consisted of document preparation; rubric review; initial reading and scoring of one learning
outcome; initial sharing/recording of results; initial consensus development and adjusting of
results; initial reading and scoring of remaining learning outcomes; the reading and scoring
of remaining transcripts; sharing/recording results; and the development of consensus and
adjustment of results.
Nevertheless, in the literature, profound discussions about the need for alignment among
educational model features are rarely found, nor are performance-assessment principles and
rubric design parameters, especially in terms of CBA.
Research Problem
This research intention was to have a wide perspective on one of the most-used
performance-assessment tools: the rubric. It has become a powerful way to assess and give
feedback to students in the past few years (Jonsson, 2014; Lasater, 2006; Reddy & Andrade,
2010). Nevertheless, several positive as well as negative ideas have been discussed about its
design and use in different disciplines. It could be relevant to understand practices, weak areas
as well as challenges for the next years in terms of its use. The two researched questions that
guided this research were: What can it be said and understood about rubric evolution in the last
decades? and What are the main challenges about performance-based evaluation rubric design
and application?
General Background
The aim was to understand rubrics’ design and use in tertiary education in general and for
performance assessment purposes in particular while CBE model is getting used and expanded
worldwide. This aim involved a Systematic Literature Review (SLR). This methodology aims
to identify, evaluate and summarize the findings of all relevant individual studies, thereby
making available evidence more accessible. The process includes elements such as (1) start by
a research question; (2) specify the search strategy and criteria for inclusion and exclusion of
studies; (3) identify, describe and synthesize relevant research. The methodology enables the
incorporation of quantitative, qualitative, and mixed methods studies as long as the importance
of both the intellectual quality of the original studies and the context, culture, and values
relevant in evaluation studies are included (Caracelli & Cooksy, 2013).
Two abstract and citation database of peer-reviewed literature databases were selected:
Scopus and ProQuest Education. For the searching process five keywords were chosen according
to the research interest: Rubrics, Competency-based education, Competency-based assessment,
Performance Assessment, and Competence. It was determined that “and” was the right Boolean
command for this search purpose. The sample then was determined by the search process run
on these databases combining main and subsequence keywords.
Instrument and Procedures
The researcher counted on access to make searches in two databases through the
university digital library system (https://biblioteca.tec.mx/basesdedatos). Hereby, the phases
and steps followed for the analysis.
Phase I: (1) Enter to the databases; (2) use of the keyword “Rubric” in the title of the

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
article; (3) select a period of 23 years (1996-2019) for search purposes, as 1996 was identified 65
as the year where the level of scientific production in this topic started to increase.
Phase II: (1) Enter to the databases; (2) use of five selected words for the search
purpose: Rubric as the main word in the title of the article and Competence, Competency-based
education, Competency- based assessment and Performance assessment as complimentary
words in the title, keywords and abstract. The Boolean combination “and” was used for filter
of search: Rubric and Competency-based education in the title, abstract and keywords, Rubric
and Competence in the title, Rubric and Competence-based assessment in the title, abstract and
keywords, Rubric and Competence-based assessment in the title, Rubric and Performance-
assessment in the title, abstract and keywords and Rubric and Performance-assessment in the
title identify; (3) gather systematically all the findings in independent datasheets for making
searches and select specifically the articles corresponding to research purpose.
Data Analysis

The database was organized in an Excel file with a total of 11 independent sheets. In each
sheet, the list of collected data was registered. The headings for storing systematically each
article data were: Authors, Author(s) ID, title, year, source title, volume issue, article number,
page start, page end, page count, cited by, DOI, link, affiliations, abstract, author keywords,
document type, publication stage, access type. These fields were analyzed for ensuring that any
article was duplicated. A counting table was designed in a complimentary independent sheet for
knowing exactly how many articles were found in this first moment (Table 1).
Table 1
SLR strategy and criteria for inclusion and exclusion of studies
Phases of
Database Main word Complimentary words Amount
research
Phase I Scopus & ProQuest Rubric (TIT) D/A 584
Scopus Rubric (TIT) Other four keywords 153
ProQuest Rubric (TIT) Other four keywords 112
Scopus & ProQuest Rubric (TIT) Other four keywords *259
Competency-based
Scopus Rubric (TIT) 9
education (TKA)
Competency-based
ProQuest Rubric (TIT) 7
education (TKA)
Scopus Rubric (TIT) Competence (TIT) 10
ProQuest Rubric (TIT) Competence (TIT) 2
Phase II
Competency-based
assessment (TKA)
Competency-based
assessment (TKA)
Performance assessment
(TKA)
(TKA)
(TIT)
(TIT)
Note: TIT means title, TKA means title, keywords and abstract, D/A means does not apply
* 6 articles were deleted: 3 duplicates, and 3 products not recognized as research articles

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
66 Research Results
Phase I Results
The first analysis of the 584 collected articles related to rubric design and application
from the title of these works, permitted a first glance of their origins: these articles have been
published mainly in these countries: United States (31%), Spain (12%), Australia (10%), Canada
(8%) and Turkey (7%), United Kingdom (6%), Netherlands (5%) Japan (5%). Other countries
have 2% or less. Besides, faculty highly involved in the research of this topic are affiliated to
Universidad Autónoma de Madrid, Purdue University, University Albany State of New York,
Universidad Jaume and Texas A&M University.
From the 584 articles, 29 were chosen by their high number of citations as a criterion
that might take the researcher to read the most popular publications in this subject area along
last years. This permitted to revise carefully the content, orientation, results, and discussions
around rubric application.
In this phase of the research, it was also possible to map the different disciplines involved
in the study of rubric design and application besides social science and education. It was
interesting to find that many articles were published by researchers of other disciplines different
from social studies, such as engineering, health science, business, agriculture, etc.
Besides, it was detected first that most of the articles emphasize the advantages that
rubrics bring to diagnostic, formative, and summative assessment, specifically on certain
features such as utility and transparency while rubric is used for giving feedback. Besides, its
power of making students reflect around self-assessment and self-efficacy practices is being
discussed (O’Brien, Franks, & Stowe, 2008; Smit, Bachmann, Blum, Birri, & Hess, 2017;
Yune, Lee, Im, Kam, & Baek, 2018).
Nevertheless, some misunderstandings and criticism around rubric design process are
also debated. For instance, Tigelaar and Janssen (2012) reported some dilemmas that arose
while designing rubrics for assessing science student teachers’ performance: fully detailed
criteria against open-ended description; the importance given to the theoretical domain against
practical knowledge application; the establishment of novice against expert performance level
as point of departure for performance explanation; the involvement for validation purposes of
only teachers against including some stakeholders, among others. Moreover, Eshun and Osei-
Poku (2013) and Jonsson and Svingby (2007) highlighted some disadvantages and criticisms of
designing and using rubrics: the exclusive and excessive use of rubrics for assessment purposes,
a lack of guidance for making judgments through rubric levels, the absence of a validity and
reliability process, were among the main problems identified. Table 2 contains an overview and
summary of Phase I synthesis of rubric research and evolution in the last decades.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Table 2 67
Rubric evolution based on a Scopus and ProQuest analysis of articles about rubric design
and application over the past 23 years
1996-2000 2001-2005 2006-2010 2011-2015 2016-2019
Number of
scientific
23 32 79 188 262
published
articles
Disciplines Agriculture
Mathematics Earth Science Chemistry
besides Pharmacology Biochemistry
Medicine Economics Computer
Social Physics Business
Nursery Energy science
Science and Veterinary Creative
Neuroscience Engineering Dentistry
Education thinking
Validity
Parallel Self-assessment Higher education
Reliability
emergent Self-efficacy Peer
Formative Quantification
topics related Teaching assessment
assessment analysis Metacognition
to rubric Inter-rater Reasoning skills
Feedback Scoring
design and and intra-rater Writing skills
Design
application reliability Factor analysis
recommendations
Time consuming
Lack of validity and design and use
Redundancy Students lack
Risk of exclusive reliability Intolerance to
Criticism Inconsistency in of participation
use Right number of change
focus qualifiers in design
criteria Converting
scales to scores
Phase II Results
From the 259 articles, 11 open access research articles were chosen for a deeper analysis
according to SLR methodology. Their contents have been considered of high involvement
with performance-based evaluation design and application rubric topic. First, a summary of
main aspects of each article was organized in a table. Then, three questions were asked to
extract relevant information about performance-oriented rubric research: (1) what are the main
interests while studying performance-oriented rubric design and use? (2) what kind of data
are these studies collecting for analysis; (3) what is the proposal for future research? Finally,
some facts, as well as some missing aspects according to competency-based assessment and
performance-assessment demands, are discussed.
The main aspects related to the 11 chosen articles were collected. It was taking into
consideration: Author(s) and year of publication, the academic level where the research was
conducted, the subject area or discipline, focus of the research problem, methodology, main
results and suggested future research. The information is organized in table 3 following a
chronological order.

OF EDUCATION
PROBLEMS
68
Vol. 78, No. 1, 2020
IN THE 21st CENTURY
Table 3
Summary of main facts and findings from 11 open access articles chosen for SLR
Author(s) and Academic Focus of the research

Subject area Methodology Results Future research
year level problem
Quantitative and qualitative:
(1) an ex-post-facto study by
Positive: moderate internal
collecting 2016 results from
consistency or the rubric (0.88)
rubric application for estimating
https://doi.org/10.33225/pec/20.78.61
and moderate reliability at all
Latimer, Bergee, music students’ performance Reinforce the practice
High school – Music Analyze reliability of rubrics levels (by dimension and total).
and Cohen in music festivals along 3 of using rubrics for
large groups performance as well as the level of These results were similar to
(2010) consecutive years. music performance
festival assessment acceptability of use. what other studies in this field
(2) categorizing comments assessment.
have published.
collected from 515 open-ended
Utility level was also positive
questionnaires to estimate
according to rubric adjudicators.
utility and improvement
suggestions.
Quantitative: Crossed
How does the rubric use random-effects modeling to
Positive: The necessity of using
influence the scoring of estimate rater effects among
scoring a rubric for grading Study other
Teachers’ use performance assessment? teachers and effect of teaching
Kan and Bulut performance assessment is types of rubrics
8th grade of performance- Is there any interaction experience.
(2014) proved. for performance
oriented rubric between teachers’ work Three facets were established:
assessment purposes
experience and their grading persons, tasks, and raters.
behaviors? 50 students
17 teachers.
Positive: The results revealed that
Verano-
Analyze the reliability and there was a relatively high
Tacoronte,
validity of a rubric designed to Quantitative: Pearson positive correlation among the
González-
guide and evaluate correlation for estimating score of each evaluator as well as Make more research
Betancor, Higher
Students’ oral oral presentation skills, with global scores giving by two an acceptable level of consensus including the
Bolívar-Cruz, education-
skills support professors. Then, Cohen One of the factors that evaluation of the
Fernández- Undergraduate
in visual media and with a time Kappa score for estimating the contributed to the acceptance standard of utility
Monroy, and
limitation, in a work context in level of agreement. of the rubric was the previous
Galván-Sánchez
team. training of the appraisers through
(2016)
of different actions.

Quantitative mainly. The study Positive: According to the expert
was divided into five moments: review, the
(1) considerations about average score was 3.65 (out of
engineering design phases: 4.0), and all content validity index
problem, solution and and
Develop a performance-based implementation; inter-relater agreement values for
evaluation rubric that can (2) establishment of each item exceeded 0.8. These
Higher assess performance criteria and tasks; results indicate all elements to be
Jin, Song, Shin, Engineering

education and enhance students’ (3) and decisions about scoring highly valid and reliable. Not included
and Shin (2015) design
Undergraduate engineering design skills scales: 1: poor; 2: marginal; 3: According to the students’

in introductory engineering satisfactory; 4: excellent; and perception of the learning
design courses (4) validation review by experts the rubric
using three methods: expert was helpful for student activities
review, a usability test, and a and could be used
field evaluation. as a useful guide for successful
(5) Students ‘perception after engineering design
its use. activities.
Positive: (1) training work

Qualitative. The study was
highlighted lack of consensus.
divided into two phases:
Then
Crespo-Martín, Design a model of evaluation (1) orientation and support Establish a
putting definitions in common
Figueras-Ferrer, headings sessions for professors to get department or
served to get to a rubric with a
Mascarella- Higher ad hoc, and to carry out a pilot to a consensus about expected center to consolidate
Soft skills better structure. (2) getting to an
Vilageliu, education- test to verify its effectiveness, achievement all performance
(teamwork) agreement on the definition of
Nualart-Torroja, undergraduate evaluate the results obtained (2) the assessment
teamwork competence. (3) the
and Valle-Martí and evaluate the results. definition of the competencies, procedures
pilot experiences
(2016) table of headings,
contribute to make deeper
implementation of the rubric
reflection on the competence and
and analysis of the results.
way to develop it.
https://doi.org/10.33225/pec/20.78.61
69
PROBLEMS
Vol. 78, No. 1, 2020

OF EDUCATION
IN THE 21st CENTURY
OF EDUCATION
PROBLEMS
70
Vol. 78, No. 1, 2020
IN THE 21st CENTURY
Positive: Although rubrics

required an initial investment of
time, aligning observation
rubrics with the requirements
of teacher performance
Make an analysis of the
assessments provided teacher Continuously
components of a student
candidates a valuable information examine how faculty
teaching practicum rubric
https://doi.org/10.33225/pec/20.78.61
Qualitative: several information source about expectations of the introduce professional
used in teacher preparation

Higher Teachers’ sources like planning, examinations and teaching standards,
program to determine if
Brown (2017) education - assessment observation criteria, instruction, assessments required for initial performance-based
the performance-based
Undergraduate competencies standards and assessment teacher certification. assessment and
assessment supported the
practices were all analyzed. This evaluation instruments
expectations and requirements
information provided them the in the teacher
of a teacher performance
opportunity to analyze their own preparation program.
assessment.
work and
progress towards to fulfill the
expectations of the
teacher performance
assessment.

Positive: Tracking students’ way
of learning through the interviews
permitted to design a
a detailed way to map learning in
Characterize competence different areas of
change in undergraduate e-textile design than those
students in a semester using Qualitative: pre and post examined before.
e-textiles. The focus was the interviews with 7 students. A Besides, as the rubric was Continue making
Higher Makers rubric and its illustration in coding scheme was developed developed from student research about
Lee and Fields
education - movement and each dimension rather than and organized into an interviews within the situated learning with other
(2017)
Undergraduate electronic textiles describing all three areas analytical rubric to understand context of e-textile student

of learning simultaneously. performance in the three a specific e-textiles course. It groups
These three dimensions dimensions can provide a starting point for
were: Circuitry, crafting and investigations of other
computation. e-textiles learning environments,
such as those beginning to reach
into K12 classrooms
where assessments are being
developed.
Positive: The use of rubric
based on Bloom’s taxonomy and
feedback were useful to follow the
performance track along the four
Quantitative: eight writing
writing courses.
scaffolding groups were
The initial writing
divided into four, from less
skills could vary based on the
Explore if the use of feedback complex to more complex
Higher quality of writing instruction
Carson and Writing skills comments, a graded rubric and writing skills goals to achieve.
education - students receive while in high Not included
Kavish (2018) Sociology studies scaffolding within a discipline Bloom’s taxonomy was used
Undergraduate school.
will increase writing skills to operationalize progression
However, it is reasonable to
along writing skills performance
assume
improvement.
that the scaffolding of rubrics
would be beneficial to all
students, regardless of the initial
writing skill of students when they
https://doi.org/10.33225/pec/20.78.61
begin their higher education.

71
PROBLEMS
Vol. 78, No. 1, 2020

OF EDUCATION
IN THE 21st CENTURY
OF EDUCATION
PROBLEMS
72
Vol. 78, No. 1, 2020
IN THE 21st CENTURY
Quantitative: 97 students
participated in the study. It
was divided into two phases:
(1) Decide the components in
This research had two focus: authentic assessment design
the first one to understand by: a) making requirements;
the pattern of development b) deciding on authentic
Positive:
of authentic assessment assessment that can develop
Validity and reliability of problem
https://doi.org/10.33225/pec/20.78.61
rubric, gather accurate students’ potential in hard
solving and journal analysis skill

information related to students’ skill and soft skill; c) making
Higher rubric
Hindriana and Biological performance and learning indicator in cognitive,
education – Had a big loading factor: above Not included
Setiawati (2018) education achievement. The second one psychomotor, and affective
Undergraduate 0.50. Thus, most of the items
to designing domains; d) deciding strategy
were valid. Reliability stated as
authentic assessment rubric. of development and application
adequate: Cronbach greater than
The research started it by in the class.
or equal to 0.70
deciding the aims, and (2) Implementation on the
components needed to be in ongoing assessment and
the rubric. doing research in time series
to observe the development
of students’ potential along
with the use of authentic
assessment rubric.
Quantitative:
Rubrics’ reliability and
Positive: Significant difference
Test the hypothesis: a reliable consistency were measured
was found between midterm and
assessment method will detect through written assignments
final performance evaluations. No
increased scores from midterm and clinic performance Replicate this study in
Higher Health and correlation was found between
to final evaluations. Then, observations. multiple health schools
Kopp (2018) education – clinical written assessment and final
reliability of an undergraduate 58 first semester students including interrater
Undergraduate instructors performance evaluation.
clinical performance grading Independent sample T test and assessment.
Rubric consistency was high
rubric was the focus of this Pearson correlation were used.
(0.91)
research study. Cronbach’s alpha was used
to measure rubric consistency
outcomes.

Both, quantitative and
qualitative: Exploratory study.
The analysis included the
estimated frequencies of:
(1) Type of tasks evaluated:
Both, positive and negative.
written products, oral
Positive: Criteria homogeneity,
Make an analysis of 59 presentations, use of software, Continue exploring
appropriate progression along
(Velasco- Higher rubrics designed by different problem solution, teamwork, ways as well as
Rubric design levels, language use parallelism.
Martínez & education- professors in three different simulations, audiovisual training professors
analysis Negative: Lack of explanation

Hurtado, 2018) Undergraduate universities in Spain. design, and others specific to evaluate students’
and poor level of specifications

from certain disciplines. performance
about criteria and purposes,
(2) type of competences:
combination of scales types.
instrumental, interpersonal,
systemic, combination of skills
(3) different disciplines: social
science, engineering, health,
science, arts and humanities.
https://doi.org/10.33225/pec/20.78.61
73
PROBLEMS
Vol. 78, No. 1, 2020

OF EDUCATION
IN THE 21st CENTURY
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
74 What are the main interests while studying performance-based evaluation rubric design
and use? It could be affirmed that there are two main research interests: In the first place,
reliability and validity as a way to ensure rubric quality. The goal refers, on the one hand to
consistency and stability of the instrument. On the other hand, to the consensus from experts
to select criteria, describe expected performance levels and agree upon meaning. Indeed, this
interest has been considered relevant since 2001 (Table 1). It is possible to infer from Latimer,
Bergee and Cohen (2010), Jin, Song, Shin and Shin (2015) and Kopp (2018) that performance-
based evaluation rubrics are instruments with high possibilities on supporting reliable and valid
assessment process for observing and judging students’ performance advances.
In the second place, the capabilities and preparation of educators for designing rubrics.
Undoubtedly, rubric design is a complex task that involves decision making starting by the
understanding on student’s profile, ways to promote progression along the educational program,
the context and conditions of the educational scenario, the task purpose, previously defined
competencies and standards, authenticity of tasks, among other variables. Velasco-Martínez &
Tójar Hurtado (2018) concluded after a wide analysis of diverse rubrics for different disciplines
assessing needs that there is a latent risk educator that could have designed rubrics without
any previous training. Thus, it is recommended to establish explicit and clear criteria to ensure
rubrics design quality. Therefore, it seems advisable from the analysis that participation in
workshops of design and use of rubrics in order to train teachers for an adequate evaluation with
rubrics is highly recommended.
What kind of data are these studies collecting for analysis? From the 11 articles
analyzed, almost all of them presented quantitative data analysis. This information was used for
demonstrating through conventional statistics the level of reliability and correlation of rubrics
results in contrast to grading. Cronbach’s alpha and Pearson correlation coefficient were the
most used measures to make judgments about rubrics quality.
Nevertheless, results from studies based on mixed methods or qualitative methodology
gave information about the design process and perceptions of users rather than results of its
application. These studies focused on decision making, consensus, and proposed different ways
of making rubrics useful. For instance, in the study of Brown (2017) as a proposal for improving
performance-based evaluation rubric design started by the examination of certain documents
and procedures that guide educators’ performance. In the case of Lee and Fields’ study (2017)
the main information source for rubric design was students’ perception. Neither theoretical
contents, nor teachers’ expectations but the way students perceived their performance advances
in three different tasks along a semester were taken into consideration.
What is the proposal for future research? Most researchers’ point of view agreed that
the application rubrics in other scenarios as well as the exploration of new proposals for rubric
design are the most common recommendations. Besides, educators training and practice for
designing and improving rubrics should be also considered for future research.
In the opinion of the researcher, it was unexpected to find just a very few ideas for
future research while performance-based evaluation rubrics is a complex vast topic with a
wide spectrum of educational research possibilities. This situation derived to certain aspects of
discussion.
Discussion
Results from this research permitted to confirm that rubrics design is a topic of upward
interest in the educational process around the world. The rise of interest from decade to decade
doubled and tripled the number of studies published in recognized indexed journals. Thus,
SLR permitted to have a holistic view of rubric evolution for almost three decades as well as a
current view around relevant issues in the last decade about its design, application and quality
insurance for performance-assessment purposes.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
One of the main results from the SLR is the confirmation of the importance of the design 75
and use of rubrics as a matter of studies in different disciplines’ educational processes as well as
soft skills assessment interests. Undoubtedly, the popularization of rubrics as a tool for guiding,
observing and measure students’ progress in different knowledge areas and task is increasing.
Besides, it was confirmed through the specific analysis in Phase 2 that higher education
is the education level where most of these studies are taking place. Nevertheless, the advances
in rubric studies related to complementary topics of CBA and performance assessment are
quite weak at this time. Indeed, just a few isolated efforts could be found through SLR that
give a deeper understanding on how to integrate in rubrics design elements of authenticity
and complexity of learning scenarios (Hindriana & Setiawati, 2018), full integration of other
domains going further cognition (Lee & Fields, 2017) or assessment literacy focused on
performance-based evaluation rubrics design (Velasco-Martínez & Hurtado, 2018).
As a matter of fact, there are some educational challenges around CBA and performance
assessment that educators, as well as educational researchers, would need to identify for the
next decade related to the study of CBA and performance-based evaluation rubrics:
(1) Integration of cognitive, psychomotor, social and affective domains. Even if
almost all studies refer to performance-based evaluation rubrics as powerful tools
to observer capabilities and executing some skills, very few of them go beyond
integrating elements distinct from the cognitive domain. This fact could be
considered contradictory as psychomotor, social and affective domains need to be
integrated into students’ performance in complex situations based on competencies
development (Escudero Escorza, 2016; Kuh et al., 2014).
(2) Comprehension of the authentic complex situation where performance assessment
takes place. Authentic complex situations (Gao & Grisham-Brown, 2011; Hancock,
2007) need to be comprehended to clarify the indicators that integrate a performance-
based evaluation rubric. This is even more important when problems or situations
to be solved integrate interdisciplinary activities. As one of the most important
performance assessment goals is giving information on the process of gaining
expertise, then the identification and operationalization of the main variables and
circumstances that configure a complex situation to be solved are critical pieces of
information for learners and educators.
(3) Definition of expected performance reached level according to students’ progress.
From the 11 studies, just Brown (2017) and Lee & Fields (2017) took explicit
consideration of this variable related to one of CBA main purpose. Indeed,
performance-based evaluation rubrics should permit to understand the way students
progress from one time to another. This is possible when some consensus about the
expectations on students’ performance level is previously established.
(4) Assessment literacy in educator’s preparation for CBA demands. From the 11
studies, just one of them (Velasco-Martínez & Hurtado, 2018) focused on the
understanding of how educators, specifically professors, design rubrics. The report
revealed among the most relevant weakness aspects: the lack of explanation, poor
level of specifications about criteria and purposes, and possible misunderstandings
in the selection of scales. As educators are the main responsible for instrumental
assessment practices in the classroom, it would be necessary to identify good
practices while taking decisions for rubric design purpose.
(5) Participation of external experts in rubric design. CBA features of authenticity
involve challenging situations from the work field (Gao & Grisham-Brown, 2011;
Hancock, 2007). Students are exposed intentionally to these situations to gain
expertise and improve their level of performance in certain competencies. Then,
it would be appropriate to take opinions from experts of diverse disciplines as

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
76 employers and seniors while designing a performance-based evaluation rubric as

well as when applying and giving feedback to students.
Conclusions
Rubrics have evolved in the last years as a powerful support tool to make judgments
about students’ learning in several disciplines. The amount of studies related to rubric design,
application and reliability for ensuring high level of quality and utility has increased globally.
The frequency of the studies related to this evaluation tool is concentrated by now on higher
education mainly.
Rubric design features have been changing according to the educational model and
assessment needs. In the case of this study, performance-based evaluation rubrics have been
considered a relevant topic as CBE and CBA have transformed higher formative education
scope in many countries around the world.
In the framework of CBA and performance assessment settings, a SLR took the researcher
to focus on 11 articles related to the design and use of performance-based evaluation rubric.
The analysis reflected that almost the same research topics are also studied, even if CBE scope
would have conducted researchers to incorporate other subjects such as integration of human
learning domains going beyond cognition, authenticity and interdisciplinarity as features that
characterize learning design situations, follow students’ progression along their higher education
preparation, the involvement of experts of the work field in determining essential evaluation
indicators or assessment literacy in educator’s preparation for CBA demands. These aspects
need to be considered in the next decade of performance-based evaluation rubrics studies.
Acknowledgements
The author would like to acknowledge the financial support of Novus Grant - Tecnologico
de Monterrey, Mexico, in the production of this work.
References
Barber, M., & Phillips, V. (2000). Big change questions: Should large-scale assessment be
used for accountability? Journal of Educational Change, 1(3), 277–281. http://dx.doi.
org/10.1023/A:1010064308012
Bharuthram, S., & Patel, M. (2017). Co-constructing a rubric checklist with first year university students:
A self-assessment tool. Journal of Applied Language Studies, 11(4), 35–55. http://dx.doi.
org/10.17011/apples/urn.201708073430
Boud, D. (2009). Assessment 2020: Seven propositions for assessment reform in higher education.
Education and Training, 330–333. https://doi.org/10.1200/JOP.2011.000236
Boukhentache, S. (2016). Operationalization of competency-based approach: From competency-based
education to integration pedagogy. Arab World English Journal, 7(4), 439–452. https://ssrn.com/
abstract=2898657
Brown, C. S. (2017). Aligning a performance-based observation rubric to support a teacher performance
assessment. Journal of Interdisciplinary Studies in Education, 5(2), 11–25. http://search.ebscohost.
com/login.aspx?direct=true&db=ehh&AN=125891944&site=ehost-live
Caliskan, H., & Kasikci, Y. (2010). The application of traditional and alternative assessment and evaluation
tools by teachers in social studies. Procedia - Social and Behavioral Sciences, 2, 4152–4156.
https://doi.org/10.1016/j.sbspro.2010.03.656
Caracelli, J. V., & Cooksy, L. (2013). Incorporating qualitative evidence in systematic reviews: Strategies
and Challenges. Mixed methods and credibility of evidence in evaluation. New Directions for
Evaluation, 138, 97–108. https://doi.org/10.1002/ev.20061

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Carrillo, F. J. (2015). Knowledge-based development as a new economic culture. Journal of Open 77
Innovation: Technology, Market, and Complexity, 1–17. https://doi.org/10.1186/s40852-015-
0017-5
Carson, L., & Kavish, D. (2018). Scaffolding rubrics to improve student writing: Preliminary results of
using rubrics in a sociology program to enhance learning and mechanical writing skills. Societies,
8(2), 34. https://doi.org/10.3390/soc8020034
Crespo-Martín, B., Figueras-Ferrer, E., Mascarella-Vilageliu, M., Nualart-Torroja, A., & Valle-martí,
M. R. J. A. (2016). Proyecto de investigación para la definición e implementación de rúbricas
de evaluación de la competencia [Research project for the definition and implementation of
competency assessment rubrics]. REIRE. Revista d’Innovaciói Recerca En Educació Recerca En
Educació, 9(9 (1)), 62–87. https://doi.org/10.1344/reire2016.9.1915
Duckworth, A. L., & Yeager, D. S. (2016). Measurement matters: Assessing personal qualities other than
cognitive ability for educational purposes. Educational Research, 44(4), 237–251. https://doi.
org/10.3102/0013189X15584327
Duran, R. A., Estay-niculcar, C., Alvarez, H., & Randolph, J. (2015). Exploratory study about
the implementation of the competency-based approach in graduate programs from two
Panamanian universities. Journal of Hispanic Higher Education, 1(18), 1–18. https://doi.
org/10.1177/1538192715612913
Ellis, L., Marston, C., Ph, D., Lightfoot, J., Ph, D., Sexton, J., & Ph, D. (2015). Faculty professional
development in student learning assessment: The Assessment Leadership Institute.
Research & Practice in Assessment, 10, 69-80. Retrieved from https://search.proquest.com/
docview/1829513087?pq-origsite=gscholar
Elsevier. (2019). Analyze search results for Rubric along last decade using Scopus Index.
Escudero Escorza, T. (2016). Evaluation research in the 21st Century: A more and more relevant tool
for the educational and social development. RELIEVE - Revista Electronica de Investigacion y
Evaluacion Educativa, 22(1), 1–20. http://dx.doi.org/10.7203/relieve.22.1.8164
Eshun, E. F., & Osei-Poku, P. (2013). Design students’ perspectives on assessment rubric in studio-based
learning. Journal of University Teaching & Learning Practice, 10(1), 1–13. Retrieved from http://
ro.uow.edu.au/jutlp/vol10/iss1/8
Farris, K. B., Demb, A., Janke, K., Kelley, K., & Scott, S. A. (2009). Assessment to transform competency-
based curricula. American Journal of Pharmaceutical Education, 73(8), 1–14.
Gao, X., & Grisham-Brown, J. (2011). The use of authentic assessment to report accountability data on
young children’s language, literacy and pre-math competency. International Education Studies,
4(2), 41–53. https://doi.org/10.5539/ies.v4n2p41
Gervais, J. (2016). The operational definition of competency-based education. The Journal of Competency-
Based Education, 1(2), 98–106. https://doi.org/10.1002/cbe2.1011
Goldberg, G. L. (2014). Revising an engineering design rubric: A case study illustrating principles and
practices to ensure technical quality of rubrics. Practical Assessment, Research & Evaluation,
19(8). Retrieved from http://pareonline.net/getvn.asp?v=19&n=8
Grubb, M. (1981). Using holistic evaluation (First). Encino, CA.: Glenco Publishing.
Hancock, D. (2007). Effects of performance assessment on the achievement and motivation of. Active
Learning in Higher Education, 8(3), 219–231. https://doi.org/10.1177/1469787407081888
Hindriana, A. F., & Setiawati, I. (2018). The development of authentic assessment rubric for assessing
undergraduate students’ learning and performance. Indonesian Journal of Learning and
Instruction, 1(1), 21-28. https://doi.org/10.25134/ijli.v1i1.1279
Jin, S. H., Song, K. Il, Shin, D. H., & Shin, S. (2015). A Performance-based evaluation rubric for assessing
and enhancing engineering design skills in introductory engineering design courses. International
Journal of Engineering Education, 31(4), 1007–1020. Retrieved from https://www.researchgate.
net/profile/Sung_Hee_Jin/publication/282929432_A_Performance-Based_Evaluation_Rubric_
for_Assessing_and_Enhancing_Engineering_Design_Skills_in_Introductory_Engineering_
Design_Courses/links/5786009d08aec5c2e4e2c301.pdf
Jonsson, A. (2014). Rubrics as a way of providing transparency in assessment. Assessment and Evaluation
in Higher Education, 39(7), 840–852. https://doi.org/10.1080/02602938.2013.875117
Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational
consequences. Educational Research Review, 2, 130–144. https://doi.org/10.1016/j.
edurev.2007.05.002

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
78 Kan, A., & Bulut, O. (2014). Crossed random-effect modeling: examining the effects of teacher experience
and rubric use in performance assessments. Eurasian Journal of Educational Research, 14(57),
1–28. https://doi.org/10.14689/ejer.2014.57.4
Koh, K. H., Tan, C., & Ng, P. T. (2012). Creating thinking schools through authentic assessment: The case
in Singapore. Educational Assessment, Evaluation and Accountability, 24(2), 135–149. https://
doi.org/10.1007/s11092-011-9138-y
Kopp, M. L. (2018). A standardized clinical performance grading rubric: Reliability assessment. Journal
of Nursing Education, 57(9), 544–548. https://doi.org/10.3928/01484834-20180815-06
Kuh, G. D., Jankowski, N., Ikenberry, S. O., & Kinzie, J. L. (2014). Knowing what students know
and can do: the current state of student learning outcomes assessment in U. S. Colleges and
Universities. National Institute for Learning Outcomes Assessment. Retrieved from https://
learningoutcomesassessment.org/documents/2013%20Survey%20Report%20Final.pdf
Lasater, K. (2006). Clinical judgment development: Using simulation to create an assessment rubric.
Journal of Nursing Education, 46(11), 496–504. https://doi.org/10.3928/01484834-20071101-04
Latimer, M. E., Bergee, M. J., & Cohen, M. L. (2010). Reliability and perceived pedagogical utility of a
weighted music performance assessment rubric. Journal of Research in Music Education, 58(2),
168–183. https://doi.org/10.1177/0022429410369836
Lee, V. R., & Fields, D. A. (2017). A rubric for describing competences in the areas of circuitry,
computation, and crafting after a course using e-textiles. International Journal of Information and
Learning Technology, 34(5), 372–384. https://doi.org/10.1108/IJILT-06-2017-0048
Mertler, C. A. (2001). Designing score rubrics for your classroom. Practical Assessment, Research and
Evaluation, 7(25), 1–8. https://scholarworks.umass.edu/pare/vol7/iss1/25
Meyer-Adams, N.; Potts, M. K.; Koob, J.; Dorsey, C. J.; Rosales, A. M. (2011). How to tackle the shift
of the educational assessment from learning outcomes to competencies: one program’s transition.
Journal of Social Work Education, 47(3), 489–508. https://doi.org/10.5175/JSWE.2011.201000017
O’Brien, C. E., Franks, A. M., & Stowe, C. D. (2008). Multiple rubric-based assessments of student
case presentations. American Journal of Pharmaceutical Education, 72(3), 1–7. https://doi.
org/10.5688/aj720358
Panadero, E., & Jonsson, A. (2013). The use of scoring rubrics for formative assessment purposes
revisited: A review. Educational Research Review, 9, 129–144. https://doi.org/10.1016/j.
edurev.2013.01.002
Reddy, Y. M., & Andrade, H. (2010). A review of rubric use in higher education. Assessment & Evaluation
in Higher Education, 35(4), 435–448. https://doi.org/10.1080/02602930902862859
Rodriguez, I., & Gallardo, K. (2017). Redesigning an educational technology course under a
competency-based performance assessment model. Pedagogika, 127(3), 186–204. http://dx.doi.
org/10.15823/p.2017.48
Roy, G. (2016). Competency-based Assessment in Primary Education in Bangladesh- A Review. Retrieved
from https://ssrn.com/abstract=2899109 or http://dx.doi.org/10.2139/ssrn.2899109
Schoepp, K., Danaher, M., & Kranov, A. A. (2018). An effective rubric norming process. Practical
Assessment, Research & Evaluation, 23(11), 1-12. https://scholarworks.umass.edu/pare/vol23/
iss1/11
Slaus, I. (2013). Entering global knowledge society: Role of education. Donald School. Journal
of Ultrasound in Obstetrics and Gynecology, 7, 239–247. https://doi.org/10.5005/jp-
journals-10009-1289
Smit, R., Bachmann, P., Blum, V., Birri, T., & Hess, K. (2017). Effects of a rubric for mathematical
reasoning on teaching and learning in primary school. Instructional Science, 45(5), 603–622.
https://doi.org/10.1007/s11251-017-9416-2
Tigelaar, D., & Janssen, F. (2012). Dilemmas in designing rubrics for the assessment of student teachers
in teacher education: Stories of a design and evaluation process in the context of biology teaching.
Educación Química, 23(2), 179–187. https://doi.org/10.1016/S0187-893X(17)30106-4
Velasco-Martínez, L. C., & Hurtado, J. C. T. (2018). The use of rubrics in higher education and competences
evaluation. Profesorado, 22(3), 183–208. https://doi.org/10.30827/profesorado.v22i3.7998

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Verano-Tacoronte, D., González-Betancor, S. M., Bolívar-Cruz, A., Fernández-Monroy, M., & Galván- 79
Sánchez, I. (2016). Valoración de la competencia de comunicación oral de estudiantes universitarios
a través de una rúbrica fiable y válida [Assessment of the oral communication competence of
university students through a reliable and valid rubric]. Revista Brasileira de Educação, 21(64),
39–60. https://doi.org/10.1590/S1413-24782016216403
Voronov, M., Chasovskykh, V., Popov, V., Anyanova, E., & Krainova, T. (2015). Effectiveness research
of the new learning elements initiated by the change to competency-based education model in
Russia. Procedia - Social and Behavioral Sciences, 214, 758–768. https://doi.org/10.1016/j.
sbspro.2015.11.712
Yune, S. J., Lee, S. Y., Im, S. J., Kam, B. S., & Baek, S. Y. (2018). Holistic rubric vs. analytic rubric for
measuring clinical performance levels in medical students. BMC Medical Education, 18(1), 1–6.
https://doi.org/10.1186/s12909-018-1228-9
Received: October 31, 2019 Accepted: January 18, 2020
Cite as: Gallardo, K. (2020). Competency-based assessment and the use of performance-
based evaluation rubrics in higher education: Challenges towards the next decade. Problems
of Education in the 21st Century, 78(1), 61-79. https://doi.org/10.33225/pec/20.78.61
Katherina Gallardo Dr., Director of Educational Innovation Ph.D. Program, School of Humanities and
Education, Monterrey Institute of Technology and Higher Education, Av. Eugenio
Garza Sada 2501 Sur. Colonia Tecnologico, Mexico.
E-mail: katherina.gallardo@tec.mx
Website: https://itesm.academia.edu/KatherinaGallardo
ORCID: https://orcid.org/0000-0001-8343-9518

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
80
MATHEMATICAL LITERACY PROFICIENCY
DEVELOPMENT BASED ON CONTENT,
CONTEXT, AND PROCESS
Jailani Jailani, Heri Retnawati
Yogyakarta State University, Indonesia
E-mail: jailani@uny.ac.id, heri_retnawati@uny.ac.id
Nidya F. Wulandari
SMPN 4 Pakem, Sleman, Special Region of Yogyakarta, Indonesia
E-mail: nidyaferry@gmail.com
Hasan Djidu
University of 19 November Kolaka, Indonesia
E-mail: hasandjidu@gmail.com
Abstract
The literacy proficiency development is one of concerns in education generally, so is in mathematics

education as well. The growth of literacy proficiency is one of the issues in education, because it is very
important to problem solving skills in students’ real life. This research aimed to describe the growth
of lower secondary school and upper secondary school students’ mathematical literacy proficiency
in Yogyakarta Special Region province, Indonesia. Three mathematical literacy proficiency aspects
were examined in the research, namely content, process and context. It was an exploratory descriptive
research with cross-sectional type research design. The population was 1,001 lower secondary school
and upper-secondary school students ranging between 13 and 16 years old. They were selected using the
combination of stratified and the cluster random sampling technique. A test consisting of 30 items, was
adopted from the existing PISA test items used to collect the data in the research. The main data analysis
was conducted by estimating students’ ability through the item-response theory approach. The results
showed that the mathematical literacy proficiency of the students based on content, context, and process
was still low. In the content and context domain, there was progress from 8th grade to 9th, from 9th grade
to 10th grade. In the process domain, the development of students’ abilities on formulate showed relatively
the same results for 8th, 9th, and 10th were around 500, and in the employ and interpret process domain,
there was a development of abilities from 8th to 9th, and from 9th to 10th grade.
Keywords: mathematics literacy proficiency development, domains based on PISA study.
Introduction
Advances in Information and Communication Technology (ICT) have introduced new

changes and challenges. Education is expected to prepare individuals with skills to deal with
these changes and challenges. Many countries have realized the need to equip their young
generation through education with multiple competencies. One of the students’ competences
that has become an international issue is literacy proficiency. Literacy is one of the main
competencies that learners need to face the challenges of the 21st Century (Drew, 2012). The
traditional definition of literacy is the ability to read and write a short simple statement about
individual’s life (Krasny, 2013, p. 14). Today, literacy does not only evolve into being able
to read, but also about being intellectual, and knowing how to research and solve complex
problems. Therefore, literacy is fundamental for individuals to be able to participate in society

Jailani JAILANI, Heri RETNAWATI, Nidya F. WULANDARI, Hasan DJIDU. Mathematical literacy proficiency development based on
content, context, and process
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
and achieve their goals in work and life (UNESCO, 2015, pp. 136–137). In addition, literacy 81
also has an effect on all cognitive domains (Matute et al., 2012, p. 124).
Literacy proficiency has been assessed in Programme for International Students’
Assessment (PISA) that was initiated by the Organization for Economic Co-operation and
Development (OECD). The objectives of PISA are to assess the students’ knowledge and
skills in the real world and to prepare them with the long-life learning and the community
participation (Stacey, 2011, p. 105). The result from PISA might be used by the government
to, for example, monitor the educational system (Retnawati & Wulandari, 2019; Stacey, 2011).
There are three aspects of literacy proficiency that are measured namely as reading literacy,
mathematical literacy, and scientific literacy proficiency (OECD, 2017).
Mathematical literacy is related to the individual proficiency in formulating, identifying,
understanding and implementing the mathematical foundations in multiple contexts that an
individual needs in daily life (Ojose, 2011, p. 90). It is also needed to verify the solutions
to problems that have been created (Hillman, 2014). Mathematical literacy is important for
students’ competency to read, write, and speak about mathematics (Atsnan, Gazali, & Nareki,
2018; Casey, 2013; Hillman, 2014). The result of many studies showed that mathematical
literacy is affected by some factors, e.g. school-level characteristics (Chowa, Masa, Ramos, &
Ansong, 2015; Kartianom & Ndayizeye, 2017), and teachers’ behaviors (Magen-Nagar, 2016,
pp. 318–319) which are related to the implementation of learning in the classroom. Another
factor arising from the students themselves, includes mathematics interest and self-concept
(Uysal, 2015, p. 1670), grade level, gender (Magen-Nagar, 2016, p. 318), time allocated for
learning mathematics (Savaş, Taş, & Duru, 2010, p. 113), learning facility at home (Türkan,
Üner, & Alcı, 2015, p. 359), as well as the economic, social and cultural status (Kartianom &
Ndayizeye, 2017Stacey, 2011).
The mathematical literacy proficiency involves some aspects of mathematical thinking
including reasoning, modelling, making connections between idea (NCCA, 2012, p. 8),
mathematical concept, mathematical procedure, and mathematical fact. These aspects are
central in explaining and predicting a phenomenon by emphasizing the competencies of
process, content and context (OECD, 2006). The domain of the content to assess includes the
Change and Relationship (CR), Shape and Space (SS), Quantity (QNT), and Uncertainty and
Data (UD). The next domain is the context that is related to Personal (PER), Occupational
(OCCP), Societal (SOC), and Scientific (SC). The process competencies in the mathematical
literacy proficiency are to formulate (FRM), employ (EMP), and interpret (INT).
In order to measure the context competencies, a researcher should implement several
types of test items. According to Shiel, Perkins, Close, and Oldham (2007), the test item designs
for the PISA assessment format are the traditional multiple-choice items, the complex multiple-
choice items, the closed-constructed response items, the short-response items and the open-
constructed response items.” After the students’ responses have been analyzed, the individual
capacity was classified into 7 levels, starting from below Level 1 to Level 6. The descriptions
of students’ capacity are at each level using the PISA classification.
Research Problem
Based on the results of a study conducted by PISA from 2000 to 2015, it was found
that only few students were able to reach level 4 or above. Meanwhile, most of them are still
below level 2. Some parties claimed that the low level of students’ literacy proficiency showed
the failure of education system organized by the government. Teachers are considered still
not successful in training student literacy. But the other consider that the results of the PISA
study cannot be used as a basis to justify the quality of education in a country. The things that
are debated are related to sampling, context, and differences in curriculum in each country.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
82 Retnawati and Wulandari (2019) described literacy proficiency development, but their research
had not explained the development of literacy proficiency based on its domain, namely content,
context, and process.
Although the correlation between the results of the PISA and the quality of education
is still debated, the results of PISA provide an overview of the growth mathematical literacy
proficiency of students in Indonesia in recent years. However, PISA results have not been able
to explain the mathematical literacy proficiency of students based on its domain. In addition,
students’ mathematical abilities which also influence the literacy proficiency of students were
very diverse (Balitbang Kemdikbud RI, 2014, 2015).
By utilizing the development of literacy proficiency in detail, educators can find out in
which part of the students’ development that needs to be improved. Likewise, further research
can be carried out, in order to ensure optimal proficiency development, for example, support
for teaching materials and increasing the ability of teachers to practice literacy. Based on this
problem, a research to describe the growth of mathematical literacy based on its domain was
needed.
Research Focus
The focus of this research was the growth of mathematical literacy proficiency. Based on
the PISA international study, the literacy proficiency could be determined in three domains, that
were content, process and context (OECD, 2006).
Research Aims
The aims of this research were to describe the growth of mathematical literacy proficiency
of students based on the domains of literacy, that are content, process and context domain.
Research Questions
The research questions were:

1. How was the growth of literacy mathematics based on content domain?
2. How was the growth of literacy mathematics based on context domain?
3. How was the growth of literacy mathematics based on process domain?
Research Design
The research was an exploratory descriptive research with cross-sectional type research
design. Within the research, the researcher would like to describe the growth of mathematical
literacy proficiency of the students in the 8th, 9th and 10th grade, especially the trend in
content, context, and process literacy. This research was conducted with the stages of preparing
instrument by adapting items released PISA, proving the validity and estimating reliability,
conduct tests to students, estimating the ability of students in the content, context, and process
of mathematical literacy using the international item parameters, then present the results of the
analysis to describe the trend of development of students’ mathematical literacy proficiency
in content, context, and process domain. The data collecting was conducted in 2015, and data
analysis was conducted in 2016-2017.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Population and Sample 83
The population of the research was about 125,000 lower secondary school and secondary
school students, the students ranging between 13 and 15 in the lower secondary schools and
the students ranging between 15 and 16 years old in the upper secondary schools in Yogyakarta
Special Region Province in Indonesia. A combination of the stratified random sampling technique
and the cluster random sampling technique was used to select the research participants. The
sample comprised the 8th and 9th grade students from the lower secondary school and upper
secondary school degree and the 10th grade students from the upper secondary school degree in
Yogyakarta Special Region. These students came from three different levels of schools, namely
the high-performance, the moderate-performance and the low-performance category. This
categorization was made based on the scores in the Mathematics National Examination. A total
of 1,001 students participated in the research, 464 male and 537 female, 155 students were in
their 8th grade (13-14 years old), 386 students were in their 9th grade (14-15 years old), and 460
students were in their 10th grade (15-16 years old). The sample size determined by formula to
estimate 95% confidence interval for mean of students’ literacy proficiency, using error 6.5 and
standard deviation 100, and got minimum sample size 909.25.
When data collecting was conducted, researchers informed the teachers and students that
the test was conducted only for research. All of identities about students, teachers, and school
were coded. The results of the test were not used for any decision about students.
Instrument
A test consisting of 30 items was used to collect the data in the research. The test items
were adopted from the existing PISA test items, i.e. PISA 2003, 2007 and 2011. These items
were translated into Bahasa Indonesia (Indonesian language) and the contexts were also
adjusted to correspond Indonesian contexts. These test items were developed and validated by
Wulandari (Jailani, Retnawati, Musfiqi, & Wulandari, 2015). The validity of the test instrument
was examined through the content validity that was conducted in order to identify the relevance
and the representativeness of the instrument toward the domain under assessment. It involved
consulting the test instrument to experts (professional judgments) in relation to the domains
of content and context and the domain of process in the PISA test-item model. The domain of
test item content includes the four contents (QNT, CR, SS, and UD). The content validity was
also examined to identify the coverage and the relevance of the test items to the domain of
context (PER, SSC, OCCP, and SSC), and to the domain of process (FRM, EMP, and INT). The
experts also provided feedbacks regarding the material truth, the composition of substances in
each domain, the test item readability and the relevance between the test item context and the
students in Indonesia.
Cronbach’s α was used as a measure of the reliability of the test that took the form of
essay or multiple choice with the dichotomous data. The index of reliability was .707 and the
SEM was equal to 2.81. The Cronbach’s α indicated the internal consistency at this level is
considered reasonably high. Based on the SEM score, the researchers would like to imply that
if the PISA test item model had been administered again then the score that the students would
attain would be from XT – 2.81 until XT + 2.81.
Data Analysis

The students’ abilities were estimated first by implementing the Rasch model in the
unidimensional item response theory. The abilities were used to analyze the growth of literacy
proficiency. The estimation was conducted to utilize the item parameters that had been equalized

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
84 into the international study test item with the concordance model for the linking score. The
concordance was conducted by implementing the Mean and Mean Method. The researcher
subsequently interpreted the inter-year ability literacy proficiency, especially in content,
context, and process literacy.
Table 1
Adjustment of ability parameter into the PISA international parameter
Mean of Difficulty Parameter

Sub- Equation of Ability Scale
Domain Beta
Domain This Conversion
International Research
Research
CR 0.935 -0.7005 1.6355
QNT 0.2599 0.4768 -0.2169

Content
SS –0.3173 -0.1807 -0.1367
UD –0.7122 -1.04188 0.3297
OCCP –0.6751 -1.7395 1.0644
Context PER 0.5334 0.3713 0.16214

SC 1.441 -0.5215 1.9625
SOC –1.3221 -0.3367 -0.9855
EMP –0.86963 -0.92664 0.0570
Process FRM 0.036693 -0.298 0.3347
INT 0.9454 0.4944 0.451
The steps of data analysis were as follows: (1) Estimating the item parameters and the
ability parameters by operating the Rasch model proposed by Masters (2010). It was applied
to both the students’ response from the multiple-choice test items and the dichotomous and
polytomous constructed response test item. CONQUEST program was used in the analysis (Wu,
Adams, & Wilson, 1997) with the calibration concurrent model for the 8th, 9th and 10th grade; (2)
Adjusting the test item parameters to the international test item parameters by means of Mean
and Mean method (Hambleton & Swaminathan, 1985). It was conducted until the researcher
attained the adjustment of the item parameter to the international scale for the 8th, 9th and 10th
grade. The results of the adjustment are presented in Table 1; (3) Implementing the modification
equation from the second step in order to adjust the capacity parameter for each class; (4)
performing a descriptive analysis in order to present the growth of the mathematical literacy
proficiency of the students for the 8th, 9th and 10th grade; and (5) categorizing the participants’
ability in accordance with the data analysis within the PISA model for each classroom by using
the results of participants’ capacity estimation.
The participants’ ability in accordance with the data analysis within the PISA model,
the student ability scale should be transformed to a mean that was equal to 500 and the
standard deviation that was equal to 100, minimum 0 and maximum 1,000. The results of the
transformation were then categorized into 7 levels that consisted of Below Level 1, Level 1,
Level 2, Level 3, Level 4, Level 5 and Level 6 in accordance to the Technical Report (OECD,

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
2016). Based on the results of transformation and the categorization of literacy proficiency, the 85
researcher subsequently monitored the development trend. The growth of the mathematical
literacy proficiency described by referring to the standard and by considering the aspect of
literacy content, context and process.
Research Results
Literacy Based on the Content
The mathematical literacy proficiency of the 10th grade students was better than that of
the 8 and 9th grade students’, particularly for the content of QNT, SS, and UD. The complete
th
result is shown in Figure 1. This finding showed that there was an improvement on the 8th, 9th
and 10th grade students’ mathematical literacy for the content of QNT, SS, and UD. On the
contrary, the 8th grade students had the highest mathematical literacy proficiency in comparison
to the 9th and 10th grade students for the content of CR. This finding showed that there was a
decreasing on the students’ mathematical literacy proficiency for the content of CR.
Figure 1
The mathematical literacy proficiency based on content
The growth of the students’ mathematical literacy in the low school level improved
among the 8th, 9th and 10th grade for the content of QNT, SS, UD. This result is shown in Figure
2. However, the highest score in CR was obtained by the 8th grade students. The growth of
students’ mathematical literacy proficiency in the moderate school tended to be unstable which
implied the increasing and the decreasing proficiency in accordance with the students’ grade.
For example, the Figure 2 shows that the literacy proficiency for the content of QNT increased
from the 8th grade to the 9th grade but decreased from the 9th grade to the 10th grade. Similar
result was obtained for the quantity content. In the meantime, the students’ mathematical
literacy proficiency in the high school level had improved along with the students’ grade for all
of the PISA contents that had been administered.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
86 Figure 2
The mathematical literacy proficiency based on content domain, grade and school level
Overall, there was an improvement on the students’ mathematical literacy in the 9th and
10 grade in accordance with the school level. The higher the school level was, the higher the
th
mathematical literacy proficiency that the students attained. However, for the 8th grade students,
the moderate school level attained the highest score in comparison to the high school level. The
possible reason was that the 8th grade was the most prominent.
Figure 3
The mathematical literacy on the content of QNT
There was an improvement of the students’ mathematical literacy and that the students
achieved Level 1 to Level 6 for the content of QNT. There were 10% of the 9th and 10th
grade students who had been able to achieve the Level 6. However, in general the students’
mathematical literacy for the content of QNT was low because most of the students only
achieved the following category: Below Level 1 – Level 3. Furthermore, there were only 20%
students who achieved Level 4 – Level 6. These results are shown in Figure 3.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Figure 4 87
The level of mathematical literacy proficiency on the content of CR
Figure 4 shows that base on the students’ grade, the mathematical literacy proficiency for
the content of a CR tended to be unstable. However, there were many 8th grade students who
were in the “Level 6” category compared to the 9th and 10th grade students. This result might be
due to the fact that the learning material of CR is taught to students in their 8th grade.
Figure 5
The level of mathematical literacy proficiency on the content of SS
Another content that became the focus of assessment in the mathematical literacy
proficiency was the content of SS. Figure 5 shows that, overall, the mathematical literacy
proficiency of students in their 8th grade, 9th grade, and 10th improved along with the grade level.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
88 However, in general, the students’ mathematical literacy proficiency for the content of SS was
low because most of the students achieved the below level 1–level 3 category. Figure 6 shows
that the growth of mathematical literacy proficiency for the content of UD from the 8th grade
students until the 10th grade students improved.
Figure 6
The mathematical literacy proficiency on the content of UD
Literacy Proficiency Based on the Context
The contexts that had been implemented in the mathematical literacy proficiency were in
accordance with the standards that had been implemented in the PISA and the contexts included
the use of Mathematics in the personal life (PER), social life (SOC), occupation (OCC), and
science (SC). Being adjusted to the level of ability that became the standards of PISA, the
students’ mathematical literacy proficiency was also differentiated for each domain.
Based on the Figure 7, the researchers found that the students’ mathematical literacy
proficiency improved in all contexts. The improvement was in accordance with the students’
grade level. The findings showed that the students’ mathematical literacy proficiency in the
Province of Yogyakarta Special Region improved in accordance to the grade level.
However, in this case the 8th and the 9th grade students achieved the highest score for
the SC context while the 10th grade students achieved the highest score for the OCC context.
Meanwhile, for the PER and SOC context, the students achieved lower score than the other
two contexts (OCC, and SC). The reason was that the PER and SOC context test items were
designed under the process of interpreting, implementing and evaluating the mathematical
results (the third domain of process) and under the indicator of drawing the conclusion on the
mathematical results toward the contextual problems and of evaluating and providing logical
reasons or arguments toward the mathematical results that had been attained. In addition, the
students’ ability in this domain of process was lower than their ability in the other two domains
of process (FRM, and EMP).

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Figure 7 89
The mathematical literacy proficiency based on the Context
Figure 8 shows that the students’ mathematical literacy proficiency in the low, moderate,
and the high school level increased from the 8th grade to the 9th grade and from the 9th to the
10th grade for the OCC and SC context. On the contrary, the students’ mathematical literacy
proficiency for the PER and SOC context in the low and moderate school level decreased from
the 8th to the 9th grade but increased from the 9th to the 10th grade. Then, for all contexts the
students’ mathematical literacy proficiency increased from the 8th grade to the 9th grade and
from the 9th grade to the 10th grade. In general, the 9th and the 10th grade students’ mathematical
literacy proficiency increased for all contexts in the low, moderate and high-level schools. The
reason was that the model level school that had been sampled was the best moderate level
school.
Figure 8
Mathematical literacy proficiency based on the context, grade and the school level

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
90 Subsequently, the growth of the students’ mathematical literacy proficiency based on

each context is shown in Figure 9. The growth of grade 8 to 10 students’ mathematical literacy
proficiency for the OCC context. The OCC context was related to the students’ life in the
school or in the working environment. For the OCC context in the 8th grade, the percentage
showed that the 8th grade students’ ability in the Province of Yogyakarta Special Region was
equally distributed from the “Below Level 1” category until the “Level 5” category for the
OCC context; most of the students were in “Below Level 1” category and there were very few
students or there were almost none of the students who were in “Level 6” category.
Figure 9
The Level of mathematical literacy proficiency on the OCC context
The results shown in Figure 9 suggest that the improvement between the 9th grade and the
10 grade students’ mathematical literacy proficiency was almost similar. The number of the 9th
th
grade and the 10th grade students who were in “Below Level 1” category were very few or near
0% and there were less than 5% of the students who were in “Level 1” category. The number
of the 10th grade students who were in “Level 5” and the “Level 6” category was higher than
that of the 9th grade students. The Figure showed that based on the OCC context the students’
mathematical literacy proficiency improved from the 8th grade to the 9th grade and from the 9th
grade to the 10th grade in accordance with the students’ grade level.
The domain of PER context had direct relationship to the students’ daily activities. In
the daily activities, the students definitely encountered the personal problems that demanded
immediate solutions. The growth of the students’ mathematical literacy proficiency from the 8th
grade to the 10th grade students is displayed in Figure 10.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Figure 10 91
The level of mathematical literacy proficiency on the per context
Figure 10 shows that for the PER context 30% of the 8th, the 9th and the 10th grade students
were in “Below Level 1” category and 10% of the 8th, the 9th and the 10th grade students were
in “Level 1” category. In addition, still based on the above Figure 10 it was apparent that the
students’ mathematical literacy proficiency improved from the 8th grade to the 9th grade and
from the 9th grade to the 10th grade. The improvement was found in the decreasing number of
the 9th and the 10th grade students who were in “Below Level 1” category and the increasing
number of the students who were in “Level 2,” “Level 3,” “Level 4” and “Level 5” category.
However, in overall the students’ mathematical literacy proficiency for the PER context was
still low because more than 70% of the students were still in “Below Level 1,” “Level 1,”
“Level 2” and “Level 3” categories and around 20% of the students were in “Level 4,” “Level
5” and “Level 6” categories.
Figure 11
The level of mathematical literacy proficiency on the soc context

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
92 The SOC context was related to the use of mathematical knowledge in the SOC life
and the wider neighborhood in the daily life. Figure 11 shows that the growth of the students’
mathematical literacy proficiency was based on the SOC context. Figure 11 suggests that for
the SOC context, most of the students in grade 8 to 10 were in “Below Level 1” category. The
percentage above showed that based on the SOC context the students’ mathematical literacy
proficiency improved from the 8th grade to the 9th grade and from the 9th grade to the 10th grade.
The improvement was found in the decreasing number of 9th grade and 10th grade students who
were in “Below Level 1” and “Level 1” category and the increasing number of the students who
were in “Level 2” category and above. However, in general the students’ mathematical literacy
proficiency for the SOC context was still low since most of the students were in “Below Level
1,” “Level 1,” “Level 2” and “Level 3” categories.
The SC context was related to the scientific activities that were more abstract and that
demanded theoretical mastery and understanding in performing the mathematical problem
solution (see Figure 12). For the SC context, Figure 12 shows that the students’ mathematical
literacy proficiency improved from the 8th grade to the 9th grade and from the 9th grade to the
10th grade in accordance with the students’ grade level. In addition, for the SC context more than
50% of the students in the Province of Yogyakarta Special Region achieved the “Level 4,” the
“Level 5” and the “Level 6” category.
Figure 12
The level of mathematical literacy proficiency on the SC context
Literacy Proficiency Based on the Process
The students’ literacy proficiency was also classified into each domain of process. The
mapping of the 8th, 9th and 10th grade students’ mathematical literacy proficiency in the domain
of process is presented in Figure 13.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Figure 13 93
Mathematical literacy proficiency based on grade and process
The domain of process in formulating the mathematical situations (formulate) the

mathematical literacy proficiency of 8th, 9th and 10th grade of the students improved along with their
grade level shown in Figure 13. Similar finding was also found in the domain of employing the
mathematical concepts, facts, procedures and reasoning (EMP), and interpreting, implementing
and evaluating the mathematical results (INT). However, the students’ mathematical literacy
proficiency in the process of employing the mathematical concepts, facts, procedures and
reasoning was lower than that in the process of formulating the mathematical situations.
Similarly, the students’ mathematical literacy proficiency in the process of interpreting,
implementing and evaluating the mathematical results was the lowest compared to the other
two domains of process. These findings implied that the students were likely able to employ the
mathematical concepts, facts, procedures and reasoning appropriately if they could formulate
the mathematical situations. However, there were also some students who might formulate
the mathematical situations, but they were not able to solve the mathematical situations by
employing the mathematical concepts, facts, procedures and reasoning. The students might
also interpret, implement and evaluate the mathematical situations appropriately if they were
able to formulate the mathematical situations and to employ the mathematical concepts, facts,
procedures and reasoning appropriately.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
94 Figure 14
The mathematical literacy proficiency based on process and school level
In addition to the domain of process, the growth of the students’ mathematical literacy
observed by the school level (i.e., low, moderate and high). The growth of the students’
mathematical literacy proficiency for the domain of process in each school level is presented
in Figure 14. The result in Figure 14 shows that the growth of the 8th grade students had
improved from the low-level school to the moderate-level school. However, the growth of the
literacy proficiency had decreased in the high-level school in terms of the domain of process. In
overall, mathematical literacy proficiency of the students had improved in accordance with the
increasing school level and quality.
Figure 15
The mathematical literacy proficiency to formulate
Then, the growth of the students’ mathematical literacy proficiency was examined under
each domain of process. The first domain was to formulate the mathematical situations is shown
in Figure 15. The second domain was to employ the mathematical concepts, facts, procedures
and reasoning. The growth of the students’ mathematical literacy proficiency from the 8th grade
until the 9th grade in the second domain is shown in Figure 16.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Figure 16 95
The mathematical literacy proficiency to employ
These findings showed that there had been improvement of the students’ mathematical
literacy proficiency in accordance with their grade level. The improvement can be seen in the
decreasing percentage of the 9th grade and the 10th grade students who were in “Below Level 1”
category and the increasing percentage of the students who were in “Level 6” category.
Subsequently, the researchers examined the students’ mathematical literacy proficiency
based on the third domain of process, namely, to formulate, to employ, and to interpret the
mathematical results. Figure 17 presents the mathematical literacy proficiency of the students’
in the Province of Yogyakarta Special Region on the third domain of process.
Figure 17
The mathematical literacy proficiency to interpret
The percentage showed that there was improvement on the students’ mathematical
literacy proficiency from the 8th grade to the 9th grade and from the 9th grade to the 10th grade
although there were few students in “Level 5” and “Level 6”. There is a decrease in the number
of grade 9 students and 10th grade students who were in “Below Level 1” category and there
are increasing number of students who were in “Level 4,” “Level 5” and “Level 6” category.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
96 Discussion
The results of this research showed that there was an improvement of mathematical
literacy proficiency of the students from the 8th grade to the 9th grade and to the 10th grade. The
higher the grade was, the higher the students’ achievement in the score of mathematical literacy
proficiency. These results indicate that the achievement level of the students’ mathematical
literacy proficiency was also influenced by the grade. However, the influence of age or year level
was not significant. This result is in accordance with the work of Magen-Nagar (2016, p. 318);
Jabor, Machtmes, Kungu, Buntat, and Nordin (2011), and also Thoren, Heinig, and Brunner
(2016) who found that age and school level influenced the students’ literacy proficiency . The
reason is that the mathematics learning material learned by students in grades 9 and 10 as stated
in the standards of the Indonesian curriculum is more comprehensive than students in grades 8
did. As a result, the higher the students’ grade was the more learning materials that supported
the improvement of mathematical literacy would be.
In relation to the content of mathematical literacy there was a tendency that the literacy
within the content of CR and UD had been higher in comparison to the content of QNT and
the content of SS. Although students’ literacy in numbers, geometry, increased from grade 8 to
10, the students’ ability was still in the low category. It can be seen from the small number of
students in level 4 to 6. In numbers, most of the students in grade 8 to 10 could not achieve level
4. All students in year 8 could not even achieve level 2. This result shows that students in grade
8 found difficulties in formulating information in most of the test items. They could only answer
questions related to geometry and numbers that have been clearly defined. The students’ low
ability in the geometrical content indicated their low ability in spatial skills. This is in line with
previous studies, e.g. Hannafin, Truxaw, Vermillion, and Liu (2008), and Novak and Tassell
(2017) that spatial abilities are directly related to mathematics ability, particularly in geometry.
The student’s literacy in algebra was not significantly different among students in their
year 8 to 10. This is supported by the finding from previous research by Eze, Ezenwafor, and
Obi (2015, p. 99); Josiah and Adejoke (2014, p. 475) that students age is not a significant
correlate of students’ algebra skills. However, student’s literacy in algebra is better than their
literacy in numbers, geometry and uncertainty dan statistics. Most of the students’ scores were
above 669 (level 6). The possible reason is that algebra has been taught to students in year 8
than those in year 9 and 10.
Concerning achievement related to the process of formulating mathematical situations,
of employing the mathematical concepts, facts, procedures and reasoning and of interpreting,
implementing and evaluating the mathematical results, there was an improvement in the
students’ mathematical literacy from the 8th to the 9th grade and from the 9th to the 10th grade
in accordance with the students’ grade level. Meanwhile, in overall the students’ ability in
the process of employing the mathematical concepts, facts, procedures and reasoning was
lower than their ability in the process of formulating the mathematical solutions. Similarly, the
students’ score of mathematical literacy in the process of interpreting, employing and evaluating
the mathematical results was the lowest one in comparison to their scores in the other two
domains of process. These findings showed that the students were likely able to employ the
mathematical concepts, facts, procedures and reasoning appropriately if they could formulate
the mathematical problems. The students’ low ability in the domain of interpreting shows that
they were not able to infer, apply, and evaluate problem solutions. Similarly, Tambychik and
Meerah (2010, p. 150) also found that students found difficulties during making meaningful
connection in the problem. Furthermore, Jupri, Drijvers, and Heuvel-Panhuizen (2014, p. 51);
Hadi, Retnawati, Munadi, Apino, and Wulandari (2018) also found that students experienced
difficulties in solving problems due to their inability in applying the reverse strategy as a step in
verifying solutions to the problems before they come to a conclusion.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
For the achievement related to the context, the students’ score of mathematical literacy in 97
PER and SOC context was lower than their scores in the OCC and the SC context. The reason
was that within the learning process, the mathematics learning materials directed the students to
understand and to master the knowledge as a preparation to pursue higher level education or to
find a job. As a result, PER and SOC context had rarely been implemented in the mathematics
learning materials. This made the students find difficulties in answering questions that used
the contexts. This is in line with the studies by Abdullah, Abidin, and Ali (2015, p. 140), and
Lieven, Fien, & Erik (2015) when they had to answer questions related to contexts that they
have not learned before. Jailani, Sugiman, and Apino (2017) suggests the need to integrate
various contexts in the learning and teaching process.
The students’ achievement score in mathematical literacy was influenced by the school
level. There was correlation between the scores in mathematical literacy and achievement in
national examination. The higher the school performance in national exams, the higher the
achievements of its students of the school in mathematical literacy ability. The good input of
the students, learning process and learning achievement were usually heard by school that had
high achievement. These findings had been in accordance with the findings from Bohlmann and
Pretorius (2008), and also Chowa et al. (2015) that school-level characteristics affect academic
achievement. There were many factors which influenced the students’ mathematical literacy
ability, the score in the content of CR and of UD was higher than that in the content of QNT and
of SS both based on the school grade and the school level. The possible factor was the material
content in every grade. The content of CR and of uncertainty dominated the contents in the 8th
grade and the 9th grade in Indonesian Curriculum.
The research results showed that the students’ mathematical literacy was unsatisfying.
It should get attention from government, teacher, and researcher. The mathematics teaching
quality, including process of planning, implementing and assessing learning outcomes should
support the students’ mathematical literacy proficiency. Although the quality of schools affects
students’ ability (Chowa et al., 2015), but the quality of teaching is the most important key
to improve the students’ achievement in mathematical literacy (Retnawati, Djidu, Kartianom,
Apino, & Anazifa, 2018). The improvement toward the learning quality and the learning
assessment can be pursued through the integration of the literacy into the mathematics learning
process (Hillman, 2014, p. 403), and also to the other subjects. Besides that, mathematics
education programs should be developed to help students be able to make real life connections
(Apino & Retnawati, 2017; Djidu & Retnawati, 2018; Yavuz, İlgün Dibek, & Yalçın, 2017).
Therefore, further studies are necessary to determine the strategies that can be used to train the
students’ mathematical literacy, and also the development of teaching sets to teach mathematical
literacy.
In the content domain, specifically Shape and Space (SS), Quantity (QNT), and
Uncertainty and Data (UD), there was progress from 8th grade to 9th, from 9th grade to 10th grade,
but in Change and Relationship (CR), the ability of students from 8th grade was higher than
students from 9th grade and 10th grade. The percentage of students whose literacy proficiency
was in categories 1-6 in 9th grade and 10th grade tends to be higher than in 8th grade, and for
8th grade was dominant in below level 1. For CR and UD, the average ability of students was
approaching 600. In the context domain, the literacy ability of students in occupational and
scientific contexts showed the average of students’ ability in 8th, 9th, and 10th grade was relatively
similar. In the personal and societal context domains, the ability of students in grades 8th, 9th,
and 10th was relatively the same. For the 8th grade, students’ mathematical literacy abilities
related to occupational and scientific contexts were already above the international average in

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
98 grades 9 and 10 that met 600. In the process domain, the development of students’ abilities on
formulate (FRM) showed relatively the same results for 8th, 9th, and 10th were around 500. In the
employ and interpret process domain, there was a development of abilities from 8th to 9th, and
from 9th to 10th grade.
Based on the results of these studies, some further research can be conducted related to
this research. The development of mathematical literacy abilities based on content, contexts
and processes domains that have not been as expected, needs to be determined. The decreasing
scores of students’ literacy skills from 8th to 9th grade, from 9th to 10th grade also need to be
known as the contributing factors. The results of students’ literacy scores in the domain of
content, context and processes that have not reached optimal scores, efforts to improve the
quality of mathematics learning need to be done. Research related to improve teaching and
learning that train mathematical literacy competencies, especially related to each mathematical
literacy subdomain needs to be done. The assessment model to assess the development of student
literacy skills for each domain also needs to be examined, so that the mathematical literacy
ability of each stage can be measured. Likewise, students’ difficulties in solving problems
related to mathematical literacy skills also need to be described and planned how to cover it.
References
Abdullah, A. H., Abidin, N. L. Z., & Ali, M. (2015). Analysis of students’ errors in solving higher order
thinking skills (HOTS) problems for the topic of fraction. Asian Social Science, 11(21), 133–142.
https://doi.org/10.5539/ass.v11n21p133
Lieven, V., Fien, D., & Erik, D. C. (2015). Mathematics education. In J. D. Wright (Ed.), International
Encyclopedia of the Social & Behavioral Sciences (Vol. 2, pp. 816–821). Oxford, UK: Elsevier.
https://doi.org/10.1016/B978-0-08-097086-8.92048-8
Apino, E., & Retnawati, H. (2017). Developing instructional design to improve mathematical higher
order thinking skills of students. Journal of Physics: Conference Series, 812(1), 1–7. https://doi.
org/10.1088/1742-6596/812/1/012100
Atsnan, M. F., Gazali, R. Y., & Nareki, M. L. (2018). Pengaruh pendekatan problem solving terhadap
kemampuan representasi dan literasi matematis siswa [The effect of the problem-solving approach
on students’ mathematical and literacy abilities]. Jurnal Riset Pendidikan Matematika, 5(2), 135-
146. https://doi.org/10.21831/jrpm.v5i2.20120
Balitbang Kemdikbud RI. (2014). Laporan hasil ujian nasional 2014. [Research and Development
Board, Ministry of Education, Republik Indonesia. (2014). Report of National Examination 2014]
[Software].
Balitbang Kemdikbud RI. (2015). Laporan hasil ujian nasional 2015. [Research and Development
Board, Ministry of Education, Republik Indonesia. (2015). Report of National Examination 2015]
[Software].
Bohlmann, C., & Pretorius, E. (2008). Relationships between mathematics and literacy: exploring some
underlying factors. Pythagoras, 67(June), 42–55. https://doi.org/10.4102/pythagoras.v0i67.73
Casey, G. (2013). Interdisciplinary literacy through social media in the mathematics classroom: An action
research study. Journal of Adolescent & Adult Literacy, 57(1), 60–71. https://doi.org/10.1002/
jaal.216
Chowa, G. A. N., Masa, R. D., Ramos, Y., & Ansong, D. (2015). How do student and school characteristics
influence youth academic achievement in Ghana? a hierarchical linear modeling of Ghana youth
save baseline data. International Journal of Educational Development, 45, 129–140. https://doi.
org/10.1016/j.ijedudev.2015.09.009
Djidu, H., & Retnawati, H. (2018). Cultural values-integrated mathematical learning model to develop
HOTS and character values. In E. Retnowati, A. Ghufron, Marzuki, Kasiyan, A. C. Pierawan, &
Ashadi (Eds.), Character education for 21st century global citizens (pp. 363–370). Routledge.
https://doi.org/10.1201/9781315104188-46

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Drew, S. V. (2012). Open up the ceiling on the common core state standards: preparing students for 99
21st-century literacy-now. Journal of Adolescent & Adult Literacy, 56(4), 321–330. https://doi.
org/10.1002/JAAL.00145
Eze, T. I., Ezenwafor, J. I., & Obi, M. N. (2015). Effects of age and gender on academic achievement of
vocational and technical education (VTE) students of a Nigerian university. Journal of Emerging
Trends in Educational Research and Policy Studies, 6(1), 96–101.
Hadi, S., Retnawati, H., Munadi, S., Apino, E., & Wulandari, N. F. (2018). The difficulties of high school
students in solving higher-order thinking skills problems. Problems of Education in the 21st
Century, 76(4), 520–532.
Hambleton, R. K., & Swaminathan, H. (1985). Item response theory. Kluwer.
Hannafin, R. D., Truxaw, M. P., Vermillion, J. R., & Liu, Y. (2008). Effects of spatial ability and
instructional program on geometry achievement. The Journal of Educational Research, 101(3),
148–157. https://doi.org/10.3200/JOER.101.3.148-157
Hillman, A. M. (2014). A Literature review on disciplinary literacy. Journal of Adolescent & Adult
Literacy, 57(5), 397–406. https://doi.org/10.1002/jaal.256
Jabor, M. K., Machtmes, K., Kungu, K., Buntat, Y., & Nordin, M. S. (2011). The influence of age and
gender on the students’ achievement in mathematics. 2011 International Conference on Social
Science and Humanity, 5, 304–308.
Jailani, J., Retnawati, H., Musfiqi, S., & Wulandari, N. F. (2015). Pengembangan perangkat pembelajaran
matematika berbasis HOTS dan pendidikan karakter [The development of mathematics teaching
material based on Higher Order Thinking Skills (HOTS) and character education]. Laporan
Penelitian Universitas Negeri Yogyakarta [Research Report of Yogyakarta State University],
Indonesia.
Jailani, J., Sugiman, S., & Apino, E. (2017). Implementing the problem-based learning in order to improve
the students’ HOTS and characters. Jurnal Riset Pendidikan Matematika [Journal of
Mathematics Education Research], 4(2), 247-259. http://dx.doi.org/10.21831/jrpm.v4i2.17674
Josiah, O., & Adejoke, E. O. (2014). Effect of gender, age and mathematics anxiety on college students’
achievement in algebra. American Journal of Educational Research, 2(7), 474–476. https://doi.
org/10.12691/education-2-7-7
Jupri, A., Drijvers, P., & Heuvel-Panhuizen, M. Van den. (2014). Student difficulties in solving equations
from an operational and a structural perspective. International Electronic Journal of Mathematics
Education, 9(1), 39–55.
Kartianom, K., & Ndayizeye, O. (2017). What’s wrong with the Asian and African students’ mathematics
learning achievement? The multilevel PISA 2015 data analysis for Indonesia, Japan,
and Algeria. Jurnal Riset Pendidikan Matematika [Journal of Mathematics Education],, 4(2),
200-210. http://dx.doi.org/10.21831/jrpm.v4i2.16931
Krasny, K. A. (2013). Gender and literacy: A handbook for educators and parents. Santa Barbara, CA:
ABC-CLIO, LLC All.
Magen-Nagar, N. (2016). The effects of learning strategies on mathematical literacy: A comparison
between lower and higher achieving countries. International Journal of Research in Education
and Science, 2(2), 306–321.
Masters, G. N. (2010). The partial credit model. In W. J. van der Linden (Ed.), Handbook of item response
theory models (pp. 109–122). Routlegde.
Matute, E., Montiel, T., Pinto, N., Rosselli, M., Ardila, A., & Zarabozo, D. (2012). Comparing cognitive
performance in illiterate and literate children. International Review of Education, 58(1), 109–127.
https://doi.org/10.1007/s11159-012-9273-9
NCCA. (2012). Project maths: Responding to current debate. National Council for Curriculum and
Assessment. http://www.ncca.ie/en/Publications/Other_Publications/Project-Maths-Responding-
to-Current-Debate.pdf
Novak, E., & Tassell, J. L. (2017). Studying preservice teacher math anxiety and mathematics performance
in geometry, word, and non-word problem solving. Learning and Individual Differences, 54, 20–
29. https://doi.org/10.1016/j.lindif.2017.01.005
OECD. (2006). Assessing scientific, reading and mathematical literacy. OECD Publishing. https://doi.
org/10.1787/9789264026407-en

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
100 OECD. (2016). PISA 2015 results (Volume I): Excellence and equity in education. OECD Publishing.
https://doi.org/10.1787/9789264266490-en
OECD. (2017). PISA 2015 assessment and analytical framework: Science, reading, mathematics,
financial literacy and collaborative problem solving. OECD Publishing. https://doi.
org/10.1787/9789264281820-en
Ojose, B. (2011). Mathematics literacy: Are we able to put the mathematics we learn into everyday use?
Journal of Mathematics Education, 4(1), 89–100.
Retnawati, H., Djidu, H., Kartianom, K., Apino, E., & Anazifa, R. D. (2018). Teachers’ knowledge about
higher-order thinking skills and its learning strategy. Problem of Education in the 21st Century,
76(2), 215–230. http://oaji.net/articles/2017/457-1524597598.pdf
Retnawati, H., & Wulandari, N. F. (2019). The development of students’ mathematical literacy
proficiency. Problems of Education in the 21st Century, 77(4), 502–514. https://doi.org/10.33225/
pec/19.77.502
Savaş, E., Taş, S., & Duru, A. (2010). Factors affecting students’ achievement in mathematics. Inonu
University Journal of the Faculty of Education, 11(1), 113–132.
Shiel, G., Perkins, R., Close, S., & Oldham, E. (2007). PISA mathematics: a teacher’s guide. Dublin:
Department of Education and Science.
Stacey, K. (2011). The PISA view of mathematical literacy in Indonesia. Indonesian Mathematical Society
Journal on Mathematics Education, 2(2), 95–126. https://doi.org/10.22342/jme.2.2.746.95-126
Tambychik, T., & Meerah, T. S. M. (2010). Students’ difficulties in mathematics problem-solving: What
do they say? Procedia - Social and Behavioral Sciences, 8(5), 142–151. https://doi.org/10.1016/j.
sbspro.2010.12.020
Thoren, K., Heinig, E., & Brunner, M. (2016). Relative age effects in mathematics and reading:
investigating the generalizability across students, time and classes. Frontiers in Psychology, 7,
1-12. Article 679. https://doi.org/10.3389/fpsyg.2016.00679
Türkan, A., Üner, S. S., & Alcı, B. (2015). 2012 PISA matematik testi puanlarının bazı Değişkenler
Açısından İncelenmesi [An analysis of 2012 PISA mathematics test scores in terms of some
variables]. Ege Eğitim Dergisi, 16(2), 358–372.
UNESCO. (2015). Education for all 2000-2015: achievements and challenges. EFA Global Monitoring
Report (2nd ed.). UNESCO Publishing.
Uysal, Ş. (2015). Factors affecting the mathematics achievement of Turkish students in PISA 2012.
Educational Research and Reviews, 10(12), 1670–1678. https://doi.org/10.5897/ERR2014.2067
Wu, M. L., Adams, R. J., & Wilson, M. R. (1997). ConQuest: Multi-aspect test software. Australian
Council for Educational Research.
Yavuz, H. Ç., İlgün Dibek, M., & Yalçın, S. (2017). Türk ve Vietnamlı öğrencilerin PISA 2012 matematik
okuryazarlığı ile dürtü ve güdülenme özellikleri arasındaki ilişkiler [Relationship between drive
and motivation features and PISA 2012 mathematics literacy of Turkish and Vietnamese students].
İlköğretim Online, 16(1), 178–196. https://doi.org/10.17051/io.2017.45107

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
101
Cite as: Jailani, J., Retnawati, H., Wulandari, N. F., & Djidu, H. (2020). Mathematical literacy
proficiency development based on content, context, and process. Problems of Education in
the 21st Century, 78(1), 80-101. https://doi.org/10.33225/pec/20.78.80
Jailani Jailani Associate Professor, Lecturer & Researcher, Mathematics Education Department,
(Corresponding author) Universitas Negeri Yogyakarta, Jl. Colombo No. 1, Karangmalang, Yogyakarta, 55281,
Indonesia.
E-mail: jailani@uny.ac.id
Website: https://scholar.google.co.id/citations?hl=id&user=qw0wE8wAAAAJ
Heri Retnawati Professor, Lecturer & Researcher, Mathematics Education Department, Universitas
Negeri Yogyakarta, Jl. Colombo No. 1, Karangmalang, Yogyakarta, 55281, Indonesia.
E-mail: heri_retnawati@uny.ac.id
Website: http://staffnew.uny.ac.id/staff/132255129 ; https://scholar.google.com/
citations?user=7CzPTYIAAAAJ&hl=en ORCID: http://orcid.org/0000-0002-1792-5873
Nidya F. Wulandari Master, Alumnae, Mathematics Education Department, Universitas Negeri Yogyakarta
Master, Teacher, SMPN 4 Pakem, Sleman, Special Region of Yogyakarta, Indonesia.
E-mail: nidyaferry@gmail.com
Website: https://scholar.google.co.id/citations?user=TCH-r5AAAAAJ&hl=en
Hasan Djidu Assistant Professor, Lecturer & Researcher, Mathematics Education Department,
Universitas Sembilanbelas November Kolaka, Jl. Pemuda, Kolaka, South-east Sulawesi,
Indonesia.
E-mail: hasandjidu@gmail.com
Website: https://scholar.google.co.id/citations?user=PSAwkTYAAAAJ&hl=id

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
102
EFFECT OF LEARNING CLIMATE,
THINKING PATTERN, AND CURIOSITY ON
ACADEMIC PERFORMANCE IN HIGHER
EDUCATION
Ali Maksum, Fifukha D. Khory
State University of Surabaya, Indonesia
E-mail: alimaksum@unesa.ac.id, fifukhakhory@unesa.ac.id
Abstract
This research aimed to examine the theoretical model that explains the relations among learning climate,
thinking patterns, and curiosity on academic performance. This research was a non-experimental
research with 1,000 respondents from State University of Surabaya, Indonesia. They came from 20
departments, consisting of 324 men and 676 women with a mean age of 19.81 years. Data were collected
using a questionnaire, including inventory of learning climate, thinking patterns, and curiosity. Data were
analyzed using Structural Equation Modeling. The results showed that the model was compatible with the
data. The examination also showed the effect of the learning climate on thinking patterns, the influence
of thinking patterns on curiosity, the influence of the learning climate on curiosity, and the influence of
curiosity on academic performance. It is concluded that learning climate, thinking pattern, and curiosity
play an important role in academic performance. Thus, the quality of students can be improved if curiosity
is cultivated and reflective thinking patterns are formed. This requires serious efforts, both in policy and
practice, to build a learning climate for the growth of students’ thinking and curiosity.
Keywords: academic performance, epistemic curiosity, learning climate, structural equation modelling,
thinking pattern.
Introduction
The quality of education in several developing countries, including Indonesia, is still

a very serious problem. All countries will certainly try to improve the quality of education,
because only with education the quality of a country’s human resources can be improved. With
a population of 265 million spread across a number of islands with diverse socioeconomic
status, it is certainly not easy for Indonesia to make policies that are appropriate for all levels
of society. Although Indonesia has adopted a policy of education costs of at least 20% of the
national budget, the results are still far from expectations. The achievements of Indonesian
students are relatively low compared to a number of other countries (OECD, 2019; 2013). In
competitions that test critical thinking skills and ability to overcome problems such as Program
for International Student Assessment (PISA) and Trends in International Mathematics and
Science Study (TIMSS), the thinking ability of Indonesian children is very low (OECD, 2013;
PIRLS, 2015). In PISA, for example, of the 65 countries that were assessed, Indonesia ranked
64th, while Malaysia ranked 52nd, Thailand ranked 50th, Singapore ranked 2nd, and China ranked
1st (OECD, 2013). In the TIMSS score, for mathematics, Indonesia is ranked 45 out of 50
countries surveyed. While in the field of science, Indonesia’s ranking is in 45 out of 48 countries
(PIRLS, 2015). The results of an evaluation conducted by the Ministry of Education in 2016
through the Indonesian Student Competency Assessment program also showed relatively
similar conditions (Nugroho, 2019). Students who entered the category lacking in mathematics

Ali MAKSUM, Fifukha D. KHORY. Effect of learning climate, thinking pattern, and curiosity on academic performance in higher
education
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
by 77.13%, the category lacking in science by 73.61%, and the category lacking in the ability 103
to read by 46.83%.
The low quality of human resources will in turn have an impact on the competitiveness of
a country. Based on the report of the World Economic Forum 2019, Indonesia’s competitiveness
index weakened, from 45th in 2018 to 50th in 2019 (World Economic Forum, 2019). From the
report it was stated that the contribution of the downgrade was mainly contributed by the low
ability of innovation, part of the resource aspect humans including education. The question
then, what is the matter with education in Indonesia? Why did some of the policies carried out
have not produced optimal results? How to improve academic performance of students? The
main problem is still the same from time to time, namely the quality of education. Education in
Indonesia has not been able to develop critical and creative thinking which is a prerequisite for
the competitiveness of a nation. The data shows that in a number of countries that have a high
competitiveness index, the quality of education is also good, such as Singapore, Hong Kong,
and South Korea. Good quality education correlates with the level of thinking of students. In
simple reasoning, the higher the quality of education, the higher the level of thinking.
Discourse about thinking patterns is not merely a matter of cognitive style, but has a
neural basis in brain structure (Evans, 2008; Kahneman, 2011; Peters, 2012; Rosenzweig,
2015). Reflective thinking patterns, which reflect depth in thinking, rest on the neocortex area,
including the anterior cingulate cortex, prefrontal cortex, and hippocampus. The intuitive
mindset which tends to be emotional relies on the limbic system area, including the amygdala,
hypothalamus, and basal ganglia. Research conducted by Maksum and Khory (2018) on
State University of Surabaya students showed that the majority (80%) of students think using
emotions and only a few (20%) of them use reasoning. If the individual is accustomed to
putting forward an intuitive thinking model, then in the long run his thinking cannot function
optimally. Critical thinking habits become undeveloped. Finally, individuals lose reference to
understanding problems clearly, including solving problems in life. This situation is not only
unfavorable but also counterproductive to progress. From the perspective of neuroscience,
the brain’s default mechanism is the limbic system (Chopra & Tanzi, 2012; Peters, 2012) and
the brain will instruct the body to release hormones such as dopamine and endorphin to gain
comfort (Amen, 2010; Pfaff & Joels, 2017). That is why, most people like the comfort zone and
don’t like difficulties.
Research results showed that thinking patterns are closely related to curiosity, namely a
trait that drives people to ask exploratory questions and find creative ways to solve problems
(Hagtvedt, Dossinger, Harrison, and Huang, 2019; Hardy, Ness, & Mecca, 2017; Ligneul,
Mermillod, & Morisseau, 2018). The desire to know has more to do with reflective thinking.
Individuals who have reflective thinking patterns try to find information, collect data, and
evaluate the information they have. Such characteristics are positively correlated with curiosity.
Thinking patterns and curiosity are also related to academic performance (Wulandari, Widayati,
& Suryobroto, 2016; Zhang, 2002). Individuals who have high curiosity have high creativity and
innovation. This is what has happened in developed countries like South Korea, Singapore and
Hong Kong. Students are stimulated by their desire to find something. The teacher’s task is to
create a learning environment conducive to growing curiosity (Hagtvedt, Dossinger, Harrison,
& Huang, 2019; Hardy, Ness, & Mecca, 2017; Leslie, 2014). Therefore, active learning models
such as case studies, discussions, group projects, individual projects, and peer reviews must be
the main pattern in learning (Cumming & Blatherwick, 2017).
Reflective thinking and curiosity are the main determinants in developing an advanced
and intelligent human civilization (Brockman, 2013; Maksum, 2015; Schwab, 2017). In the
history of civilization, many new discoveries were built on the foundation of these two things.
How Charles Darwin discovered the theory of evolution, he spent many years travelling sea
and land to study, record, and analyze the various species that exist on earth. About the same

education
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
104 thing happened to Albert Einstein when he discovered the theory of relativity and the law of
photo electricity which became an important step in the development of quantum theory. James
Maxwell as well in developing the theory of electromagnetic radiation was carried out with
repeated trials with full perseverance and hard work until a mathematical equation was found
that connected electricity with magnetism. Even in the present context, such as the Silicon Valley
phenomenon, which is the location of the discovery and development of new technologies that
change the way of life of world citizens, it is also built through a tradition of curiosity and an
extraordinary research enthusiasm (Fisher, 2018). In the region, technology-based companies
such as Apple Computer, Google, Hewlett-Packard, and Intel are growing, and are supported
by Stanford University, which stimulates research-based ideas and innovations. In short, the
great findings that changed civilization were born not through intuitive thinking patterns that
prioritize emotions, but through reflective thinking patterns that prioritize the mind supported
by curiosity that does not know the final word.
The pattern of thinking and curiosity is not a talent that is brought from birth but is the
result of the learning process through the environment. Because of that the environment is that
allows the mindset and curiosity to grow to be important (Burns, 2010; Joe, Hiver, & Al-Hoorie,
2017). Referring to the thought of Bronfenbrenner, the microscopic environment such as family
and education has a strategic role in the formation of mindset and curiosity (Bronfenbrenner,
2005). Related to the learning environment, there are some things that need attention. First,
does the learning process provide optimal opportunities for students to explore their thoughts,
ideas and performance? The approach that is often referred to as student centered needs to color
the lecture process (Cumming & Blatherwick, 2017; Hagtvedt, Dossinger, Harrison, & Huang,
2019). Active student participation in learning becomes an important indicator in assessing
the success of this approach. Thus, students feel have the freedom and autonomy to develop
themselves. Learning resources are provided on a variety basis and the role of lecturers is more
as a facilitator and inspiratory (Cummings & Blatherwick, 2017; Zhang, 2002). Second, the
attitude of lecturers should be democratic and responsive. Lecturers must stimulate students’
minds to read, make observations, question, analyze, evaluate, and ultimately create something
(Lamnina & Chase, 2019; Moreira, Ferreira, Cardoso, Gomes, & Collazos, 2018). Third, the
functional relations between lecturers and students are relatively warm. That is, if students face
problems in learning, the lecturer is at least willing to listen and understand and if possible,
provide alternative solutions.
Research Problem
From the explanation above it appears that macroscopically the learning climate, patterns
of thinking and curiosity become crucial problems in building civilizations and microscopically
become fundamental problems in learning. The aim of this research was to find a theoretical
model that explains the structural relations of the academic climate, thinking patterns, curiosity,
and academic performance. The theoretical model explained how academic performance is
formed through a tradition of strong thinking and curiosity and a conducive learning climate.
The results of this research were useful for lecturers, including policy makers in education,
to construct thinking patterns and provide appropriate treatment to students. Mistakes of
thinking have an impact on errors in behavior, including achievement in academics. Thus, the
results of this research were very useful for building academic civilization in a constructive
and progressive manner. Specifically, this research attempts to answer the following problem
formulation: Did the theoretical model that explained the structural relationship among the
learning climate, thinking patterns, curiosity, and academic performance fit the data?

education
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Research Methodology 105
General Background
This research specifically developed a theoretical model, structural relations among

the learning climate, thinking patterns, curiosity, and academic performance. To achieve the
objectives of this research, systematic and accountable steps are needed. This research used a
non-experimental method that no treatment is given to the research subjects (Cresswel, 2013;
Kerlinger, 1986). The theoretical model was formulated based on literature review, especially
the results of research in the last ten years. In this research there were independent variables,
moderator variables, and dependent variables. As the independent variable was the learning
climate, the moderator variable was thinking patterns and curiosity, and as the dependent
variable was academic performance.
Sample
The participants of this research were 1000 students of State University of Surabaya taken
by proportional random sampling. They consisted of 324 men and 676 women with a mean age
of 19.8 years and SD of 3.7 years. They come from 20 departments that were incorporated
in 8 faculties including postgraduate. Most respondents came from the engineering faculty,
which amounted to 223 (22.3%), consisting of 127 men and 96 women. The smallest number
of respondents came from postgraduate, which is 44 (4.4%), consisting of master and doctoral
students, including 22 men and 22 women.
Table 1
Number of respondents by faculty and gender
Gender
Total
Male Female %
Language & Art 52 143 195 19.5
Economy 21 94 115 11.5
Sport Science 29 20 49 4.9
Education Science 22 115 137 13.7
Faculty
Social Science & Law 30 115 145 14.5
Science & Math 21 71 92 9.2
Engineering 127 96 223 22.3
Postgraduate 22 22 44 4.4
Total 324 676 1000 100
Instruments and Procedures
Measurement of research variables was carried out with a set of questionnaires

consisting of three inventories. The thinking pattern variable was measured using the Intuitive-
Reflective Scale (Maksum & Khory, 2018). The instrument consists of 20 statements that are
bipolar, positive and negative. From the item validity test found a correlation of .44 to .69
and Cronbach’s alpha reliability of .72. Learning climate variable was measured by Learning
Climate Questionnaire (Williams & Deci, 1996). The instrument consists of 24 items. From the
validity test obtained a correlation coefficient of .54 - .80. Reliability test with Cronbach’s alpha
showed a coefficient of .93. The variable of curiosity was measured using Need for Cognition

education
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
106 (Cacioppo & Petty, 1982). The instrument consists of 18 bipolar statements, positive and
negative. From the item validity test the correlation coefficient was obtained from .41 to .70.
Reliability test using Cronbach’s alpha showed a coefficient of .74. Academic performance was
measured using the grade point average [IPK]. IPK was calculated as the ratio of the score gotten
in every subject matter weighted with the total number of class credit she/he took. The IPK
scale from 0 to 4. It is assessed at the end of each semester. Data collection involved a number
of students who had received research methods courses. Before conducting data collection,
team members were given an explanation related to the research instrument, how to fill it out,
and provide an explanation to the respondents. In the data collection process, respondents were
asked to fill in a set of questionnaires consisting of the 3 inventories. Each respondent filled out
the questionnaire for ± 30 minutes. Data collection was carried out in each faculty which was
used as a sample. After the data has been collected, verification is carried out to ensure that the
data entry of the respondents is in accordance with the provisions.
Data Analysis
Data were analyzed using Structural Equation Modeling (SEM), which is a confirmatory
multivariate statistical technique to examine the structural relations of a number of variables
simultaneously (Byrne, 2010; Hair, Anderson, Tatham, & Black, 1998). The final result of
SEM is a theoretical model, which is a summary of theories that illustrate the interrelation
between variables that are normally expressed in mathematical formulations. A model is said
to be good if it is able to explain the actual phenomenon with a small error. In SEM, there are
two interrelated stages. First, examination of the model by seeing whether there are significant
differences between the model and data. Second, if there is a match between the model and the
data (the difference is not significant), then the analysis can proceed with testing the structural
relations in the model.
Table 2
Model testing criteria
Goodness of Fit Index Criteria
Chi-square Small score
p-value p ≥ .05
Root mean square error of approximation (RMSEA) RMSEA ≤ .08
Goodness of fit index (GFI) GFI ≥ .90
Adjusted goodness of fit index (AGFI) AGFI ≥ .90
t-value t > 1.96 (p < .05)
To test the suitability of the theoretical model with the data, the goodness of fit test was
used, the testing criteria are shown in table 1. If the null hypothesis is accepted, which means
there is no difference between the model and the data, the proposed theoretical model is fit to
explain the data. If an appropriate model has been obtained, then each hypothesis can be tested
which shows the impact of a variable on other variables. The testing criteria are based on Chi-
square, GFI, AGFI, and RMSEA.

education
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Research Results 107
Before arriving at the testing of models and hypotheses, an overview of the descriptive
statistics of the main variables was presented. Table 3 showed that the mean of thinking patterns
and curiosity were relatively moderate, respectively 3.76 and 3.8. On a scale of 1-6, the score
was only a few points above the mean. The data proved that students’ thinking patterns were
generally moderate, between intuitive and reflective. The same thing happened to the curiosity
variable. The relatively high mean was in the learning climate and academic performance
variables.
Table 3
Mean and standard deviation of main variables
Variables Mean SD Score Scale
Learning Climate 4.59 .61 1-6
Thinking Pattern 3.76 .42 1-6
Curiosity 3.8 .40 1-6
Academic Performance 3.5 .52 0-4
The next step was to test the relations between variables constructed in the form of
theoretical models as visualized in Figure 1. The model connects four main variables, namely
the learning climate, thinking patterns, curiosity, and academic performance. In the context
of these relations, the learning climate was an exogenous variable, while thinking patterns,
curiosity, and academic performance were endogenous. In the model also explained how the
pattern of thinking affects the variables of curiosity and academic performance. Furthermore,
the variable of curiosity affects academic performance. In the relation model, a variable can
exchange functions as independent variables and at other times function as moderator variables
and dependent variables.
Figure 1
Theoretical model of relations among variables
The theoretical model was tested using SEM with IBM Amos 23. The test results as
shown in Figure 2 showed that the model was not fit because it did not meet the test criteria,
namely: chi-square with p > .05, GFI ≥ .90, AGFI ≥ .90, RMSEA ≤ .08, and t-value > 1.96. The
test results showed p = .0001, GFI = .848, AGFI = .746, RMSEA = .237, and t-value < 1.96. The
test also showed that regression coefficient of the learning climate on academic performance

education
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
108 was .016 with p > .05 so it was not significant. The regression coefficient of thinking pattern
variables on academic performance was .007 with p > .05 so it was not significant. Because
the direct relations between the learning climate and thinking pattern on academic performance
was not significant, the relation between the two was removed from the model.
Figure 2
First test of structural relations model among variables
Considering the results of testing the model do not meet the criteria, it was necessary
to change relations among variables, including the possibility of removing variables that were
considered not to contribute to the model. From the process obtained Model 2 as shown in
Figure 3. From the second examination obtained a chi-square value = .387 with p = .824, GFI =
1, AGFI = .999, RMSEA = .000, and t-value > 1.96. This means that the model was compatible
with the data. The examination also showed the regression coefficient of the learning climate to
thinking patterns of .39 with p < .05, the regression coefficient of thinking patterns to curiosity
of .33 with p < .05, the regression coefficient of the learning climate to curiosity of .05 with p >
.05, and the coefficient regression of curiosity on academic performance was .20 with p < .05.
That was, all relations between variables proved to be significant, except for the effect of the
learning climate on curiosity.
Figure 3
Second test of structural relations model among variables
Discussion
The final result of this research was a theoretical model that explains the structural
relations of the learning climate, thinking patterns, curiosity, and academic performance. This
research has found that the learning climate influences the thinking patterns of students. The
interaction between lecturers and students that prioritizes freedom of thought and strengthening
competencies has a positive impact on students’ reflective thinking. Constructive lecturer-student
relations will create feelings of comfort, openness, and trust in each other. This finding is in line

education
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
with research (Back, Polk, Keys, & McMahon, 2016; Joe et al., 2017; Thapa, Cohen, Guffey, & 109
Higgins-D’Alessandro, 2013) which states that the learning climate is positively correlated with
student commitment and involvement in learning. The way students think, including motivation
and achievement orientation can be changed by improving the learning climate. Referring to the
thought of Baumrind (1971) and Maccoby and Martin (1983), parenting or teacher parenting is
needed which gives achievement demands to children and at the same time is also responsive
to their needs. Thus, the child is motivated to achieve certain goals or targets and at the same
time get attention and affection.
This research also found the influence of thinking patterns on student curiosity.
Reflective thinking patterns, which require serious and evaluative thinking efforts, have a
positive impact on students’ needs for information, raise challenges, and are creative in solving
problems (Brockman, 2013; Hagtvedt, Dossinger, Harrison, & Huang, 2019). Thinking skills
and curiosity are important factors in building a creative and innovative culture that impacts the
nation’s competitiveness. Referring to the results of previous studies (Cumming & Blatherwick,
2017; Dwyer, Hogan, & Stewart, 2014; Koenig, 2011) that there are two things that now and in
the future will become needs, namely critical thinking and problem solving. Critical thinking
is the ability to analyze and evaluate something based on relevant logic, information, and data.
While problem solving is the ability to find the right solution to a problem. Both of these are
important parts of 21st century skills, including the industrial revolution 4.0 (Schwab, 2017). A
country that dreams of competitiveness needs to pay serious attention to both of these. In the
2019 world economic report, Indonesia’s competitiveness index dropped five ranks among 140
countries, from the ranking of the previous 45 years to rank 50. Of the 12 pillars that became
indicators, the ability to innovate ranks lowest. From this report, we can see that countries with
high competitiveness are those with high levels of thinking and curiosity, such as Singapore, the
United States, Hong Kong, and South Korea.
From a neuroscience perspective (Lomanowska, Boivin, Hertzman, & Fleming, 2017;
Peters, 2012), the findings of this research reinforce the notion that reflective thinking based on
neocortex needs to be continually cultivated in order to produce creative and innovative ideas
and works. Although it is recognized that humans generally experience biases in thinking, not
using the ratio optimally, and tend to discourage short-term interests. As a result, many actions
taken, as a result of decision making, are counterproductive and less effective in achieving goals
(Kahneman, 2011). People who think logically and deeply, many use the frontal area, especially
the dorsolateral prefrontal cortex and anterior cingulate cortex in the thought process. People
who respond emotionally to problems without thinking long, prejudice without data, and
draw conclusions speculatively, use many limbic areas in their thought processes. The results
of research conducted by Maksum and Khory (2018) of 383 State University of Surabaya
students showed that 80% of students used intuitive thinking. If the individual is accustomed
to prioritizing intuitive thinking in making decisions, then in the long run will disfunction his/
her common sense. Critical thinking habits become undeveloped. Finally, individuals lose
reference to understanding problems clearly, including solving problems in life.
Another finding of this research is the positive impact of curiosity on academic
performance. This is in line with the research results of Lammina and Chase (2019) and Oudeyer,
Gottlieb, and Lopes (2016) which states that curiosity influences academic performance,
including creativity. In the research it was also mentioned that uncertainty conditions actually
increase students’ curiosity, even though on the other hand it causes anxiety. Learning that is
too much to give instructions, it turns out it is less encouraging to students to seek and find their
own knowledge. Along with today’s post truth era (McIntyre, 2018), where perceptions are
more dominant in influencing people’s judgment than facts (Nichols, 2017), reflective thinking
and curiosity become a necessity.

education
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
110 The most important variable is how to strengthen students’ curiosity. Because curiosity
has a significant effect on academic performance as the findings of this research and is a
precondition for discovering something new. Many great scientists, such as Charles Darwin
and Albert Einstein, produced theories that had a profound impact on human life because of an
extraordinary curiosity. For them, nothing is more important than the desire to know. There are
three main needs that are the same between primates and humans, namely food, sex, and shelter
(Leslie, 2014; Peters, 2012). But there is one type of need that distinguishes between them,
namely the desire to know. Curiosity is only unique to humans (Leslie, 2014).
Although in the context of this research the thinking pattern and curiosity are moderating
variables, but its role is so strategic in building a culture of innovation and productivity. In
the academic world, interest in something new is important. Leslie (2014) called it diversified
curiosity, which is an open mind to be interested in something new and look for answers to
unsolved problems. Restless desire for the new and the next. In the academic world this is
referred to as epistemic curiosity or intellectual curiosity (Hardy, Ness, & Mecca, 2017), which
is the desire to continue to explore thought. The low level of thinking of Indonesian students
shown in the PISA and TIMSS assessments, including the low competitiveness index and the
global innovation index, proves that the failure of the education system in Indonesia is precisely
at this point, namely the inability to foster curiosity in students (Maksum, 2015; 2011). Perhaps
energy is focused on how to make students as teachers, economists or engineers, not to make
them as curious learners. Prepared them to enter the workforce, rather than inspire them. In the
end, we will only get uninspired students and mediocre professionals.
This research has succeeded in building a theoretical model among the learning climate,
thinking patterns, curiosity, and academic performance. Simply stated, this research concludes
that academic performance was influenced by curiosity, curiosity is influenced by reflective
thinking patterns, and reflective thinking patterns are influenced by the learning climate. From
the results of testing the model using SEM it is explained that the learning climate has a direct
impact on reflective thinking and an indirect effect on curiosity. Reflective thinking patterns
have a direct impact on curiosity and an indirect effect on academic performance. Curiosity has
a direct impact on academic performance. The findings of this research provide a framework
about how to explain academic performance from students’ cognitive aspects, namely curiosity
and thinking patterns, as well as from environmental aspects in term of learning climate. Of
course, there are other aspects outside the variables above that affect academic performance.
Therefore, this research can encourage other researcher studying academic performance to
generate and validate new question as well. The results of this research can provide guidance
to policy makers and education practitioners to improve student learning outcomes. Academic
performance can be stimulated, especially related to creativity and innovation, by improving
thinking patterns and fostering curiosity. Therefore, learning that explores higher order thinking
must be an inseparable part of lectures. Reflective thinking can be fostered by creating a
conducive learning climate, where lecturers provide opportunities for students to think freely,
build competencies, and be responsive. Reflective thinking and curiosity are two things that
are closely related, and both are crucial factors in the academic world. Therefore, these two
things must continue to be developed for students in order to build an innovative and productive
culture.

education
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
References 111
Amen, D. G. (2010). Change your brain change your body. New York: Three Rivers Press.
Back, L. T., Polk, E., Keys, C. B., & McMahon, S. D. (2016). Classroom management, school staff
relations, school climate, and academic achievement: testing a model with urban high schools.
Learning Environments Research, 19 (3), 397-410. https://doi.org/10.1007/s10984-016-9213-x
Baumrind, D. (1971). Current patterns of parental authority. Developmental Psychology, 4(2), 1–103.
Brockman, J. (2013). Thinking: The science of decision making, problem solving, and prediction.
Lawrence Erlbaum Associates Publishers.
Bronfenbrenner, U. (2005). Making human beings human: Bioecological perspectives on human
development. Sage Publication.
Burns, J. M. (2010). Leadership. Harper & Row Publisher.
Byrne, B. M. (2010). Structural equation modeling with AMOS: Basic concepts, applications, and
programming. Routledge.
Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of Personality and Social
Psychology, 42(1), 116–131.
Chopra, D., & Tanzi, R. E. (2012). Super brain: Unleashing the explosive power of your mind to maximize
health, happiness, and spiritual well-being. Three Rivers Press.
Cresswel, J. (2013). Qualitative, quantitative, and mixed methods approaches (4th Ed.). Sage Publications.
Cumming, J. B., & Blatherwick, M. L. (2017). Creative dimensions of teaching and learning in the 21st
century. Sense Publisher.
Dwyer, C. P., Hogan, M. J., & Stewart, I. (2014). An integrated critical thinking framework for the 21st
century. Thinking Skills and Creativity, 12, 43-52. https://doi.org/10.1016/j.tsc.2013.12.004
Evans, J. S. B. T. (2008). Dual-processing accounts of reasoning, judgment, and social cognition. The Annual
Review of Psychology, 59, 255–278. https://doi.org/10.1146/annurev.psych.59.103006.093629
Fisher, A. (2018). Valley genius: The uncensored history of sillicon valley. Hachette Book Group.
Hagtvedt, L. P., Dossinger, K., Harrison, S. H., & Huang, L. (2019). Curiosity made the cat more
creative: Specific curiosity as a driver of creativity. Organizational Behavior and Human Decision
Processes, 150, 1–13.
Hair, J.H., Anderson, R.E., Tatham, R.L., & Black, W. C. (1998). Multivariate data analysis (5th Ed.).
Prentice Hall.
Hardy, J. H., Ness, A. M., & Mecca, J. (2017). Outside the box: Epistemic curiosity as predictor of
creative problem solving and creative performance. Personality and Individual Differences, 104,
230–237.
Joe, H. K., Hiver, P., & Al-Hoorie, A. H. (2017). Classroom social climate, self-determined motivation,
willingness to communicate, and achievement: A study of structural relationships in instructed
second language settings. Learning and Individual Differences, 53, 133-144. https://doi.
org/10.1016/j.lindif.2016.11.005
Kahneman, D. (2011). Thinking fast and slow. Penguin Group.
Kerlinger, F. N. (1986). Foundations of behavioral research (3rd Ed.). Harcourt Brace College Publishers.
Koenig, J. A. (2011). Assessing 21st century skills. Washington, DC.
Lamnina, M., & Chase, C. C. (2019). Developing a thirst for knowledge: How uncertainty in the classroom
influences curiosity, affect, learning, and transfer. Contemporary Educational Psychology, 59,
101785.
Leslie, I. (2014). Curious: The desire to know and why your future depends on it. Great Britain.
Ligneul, R., Mermillod, M., & Morisseau, T. (2018). From relief to surprise: Dual control of epistemic
curiosity in the human brain. NeuroImage, 181, 490–500.
Lomanowska, A. M., Boivin, M., Hertzman, C., & Fleming, A. S. (2017). Parenting begets parenting:
A neurobiological perspective on early adversity and the transmission of parenting styles across
generations. Neuroscience, 342, 120-139. https://doi.org/10.1016/j.neuroscience.2015.09.029
Maccoby, E., & Martin, J. (1983). Socialization in the context of the family: Parent-child interaction. In
M. P.H. (Ed.), Handbook of Child Psychology. Wiley.
Maksum, A., & Khory, F. D. (2018). Pengembangan dan validasi instrumen untuk mengidentifikasi
pola berpikir [Development and validation of instrument to identify thinking patterns] (Research
Report). Surabaya: Unesa University Press.

education
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
112 Maksum, A. (2011). Membangun mental prestatif: Tugas utama pendidikan ke depan [Building mental for
achievement: The main task of education in the future]. In S. Syah & Martadi (Ed.), Rekonstruksi
Pendidikan [Reconstruction of Education]. Unesa University Press.
Maksum, A. (2015, April). Kurikulum dan pembelajaran di perguruan tinggi: Menuju pendidikan
yang memberdayakan [Curricullum and learning in higher education: Towards an empowering
education]. Paper presented in The National Seminar on Education and Learning Research at
STKIP PGRI Jombang, Jombang, Indonesia.
McIntyre, L. C. (2018). Post truth. MIT Press.
Moreira, F., Ferreira, M.J., Cardoso, A., Gomes, A.S. & Collazos, C. (2018, July). Learning ecosystem
for higher education disruption: A new approach proposal based on blended active learning
techniques. Paper presented in The 10th International Conference on Education and New Learning
Technologies, Palma, Spain.
Nichols, T. (2017). The Death of Expertise: The Campaign Against Established Knowledge and Why It
Matters. New York: Oxford University Press. https://doi.org/10.5765/jkacap.180021
Nugroho, Y. (2019, May 3). Menyoal pendidikan di Indonesia [Questioning education in Indonesia].
Kompas, 7.
OECD. (2013). PISA 2012 Results in Focus: What 15 year olds know and what they can do with what they
know (Research Report). http://hdl.voced.edu.au/10707/280925
OECD. (2019). PISA 2018: Insights and interpretations (Research Report). https://www.oecd.org/pisa/
PISA%202018%20Insights%20and%20Interpretations%20FINAL%20PDF.pdf
Oudeyer, P. Y., Gottlieb, J., & Lopes, M. (2016). Intrinsic motivation, curiosity, and learning: Theory
and applications in educational technologies. Progress in Brain Research, 229. https://doi.
org/10.1016/bs.pbr.2016.05.005
Peters, S. (2012). The chimp paradox: The mind management. Ebury Publishing.
Pfaff, D. W., & Joels, M. (2017). Hormones, brain, and behavior (3rd Ed.). Elsevier.
PIRLS (2015). TIMSS 2015 International Results (Research Report). Boston College
Rosenzweig, P. (2015). Left brain, right stuff. Profile Books.
Schwab, K. (2017). The fourth industrial revolution. World Economic Forum.
Thapa, A., Cohen, J., Guffey, S., & Higgins-D’Alessandro, A. (2013). A Review of School Climate Research.
Review of Educational Research, 83 (3), 357–385. https://doi.org/10.3102/0034654313483907
Williams, G. C., & Deci, E. L. (1996). Internalization of biopsychosocial values by medical students: A
test of self-determination theory. Journal of Personality and Social Psychology, 70(4), 767–779.
World Economic Forum. (2019). Global competitivenss report 2019 (Insight Report). www.weforum.
org/gcr
Wulandari, N. H., Widayati, K. A., & Suryobroto, B. (2016). Cognitive style and creative quality: Influence
on academic achievement of university students in Indonesia. HAYATI Journal of Biosciences,
23(3), 121–124. https://doi.org/10.1016/j.hjb.2016.09.001
Zhang, L. F. (2002). Thinking styles: Their relationship with modes of thinking and academic performance.
Educational Psychology, 22, 331–348.
Received: November 11, 2019 Accepted: January 29, 2020

education
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
113
Cite as: Maksum, A., & Khory, F. D. (2020). Effect of learning climate, thinking pattern, and
curiosity on academic performance in higher education. Problems of Education in the 21st
Century, 78(1), 102-113. https://doi.org/10.33225/pec/20.78.102
Ali Maksum Professor of psychology at School of Postgraduate Education, State University

(Corresponding author) of Surabaya, Kampus Lidah Wetan, Surabaya, Indonesia
E-mail: alimaksum@unesa.ac.id
Website: http://www.unesa.ac.id
ORCID ID: 0000-0002-2247-4595
Fifukha D. Khory Master of education, faculty member of physical education department, State
University of Surabaya, Kampus Lidah Wetan, Surabaya, Indonesia
E-mail: fifukhakhory@unesa.ac.id

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
114
A BIBLIOMETRIC ANALYSIS OF THE
USE OF OPEN SOURCE SOFTWARE IN
EDUCATIONAL CONTEXTS
Orfa Nidia Patiño Toro, Yesenia Acevedo Correa
University institution ESCOLME, Colombia
E-mail: cies4@escolme.edu.co, yacevedoc@ucn.edu.co
Alejandro Valencia-Arias, Martha Benjumea-Arias

Metropolitan Institute of Technology, Colombia
E-mail: jhoanyvalencia@itm.edu.co, marthabenjumea@itm.edu.co
Abstract
Open source software has now become a significant alternative in meeting different needs in business,
government and academic environments, such as needs related to economics, management, learning and
innovation, among others. The purpose of this research was to examine the research trends and evolution
of the field of open source software adoption between 2001 and 2019. The methodology used involved
bibliometric analysis of 289 documents obtained through a Scopus extracted search equation, generating
indicators of quantity and quality and analyzing the emerging themes in said field of knowledge. The
findings of the research include the existence of a strong trend towards research and dissemination
regarding open source software, particularly in countries such as the United States, whose institutions
and authors demonstrate high levels of productivity and dissemination. There was also evidence of an
interest in reducing barriers and encouraging the adoption and implementation of the software in other
sectors where its use is still lagging behind. The main conclusion of the research is that the research of
open source software adoption focuses on the following topics: innovation, Linux, FLOSS, engineering
requirements, risk management, open innovation, the public sector, social network analysis and total cost
of ownership.
Keywords: bibliometric analysis, information technology, open source software, research trends,
technology adoption.
Introduction
The adoption of open source software (OSS) is recognized as an alternative for accessing
platforms for content creation, management and learning, free of charge and in a range of
contexts. As a result, software of this type can be “freely used, copied, studied, modified
and redistributed” by any individual (González, 2006). This has led to the creation of a free
software movement that works through partnerships to develop programming projects. These
communities are characterized by having members who are highly trained and motivated to
meet the proposed objectives (Clavero, Formentí, & Prieto, 2008).
Prior Research
Open Source Software (OSS) has been extended in recent years, focusing strong attention
and becoming one of the main driving forces of the business ecosystem (Silic & Back, 2016;
Velázquez-Juárez, Valencia-Pérez, & Peña-Aguilar, 2016); however, companies and individuals

Orfa Nidia PATIÑO TORO, Yesenia ACEVEDO CORREA, Alejandro VALENCIA-ARIAS, Martha BENJUMEA-ARIAS. A bibliometric
analysis of the use of open source software in educational contexts
PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
counterbalance difficulties and challenges to achieve an appropriate adoption of OSS; since not 115
only challenges derived from the technological components must be solved but also from the
changes in the organizational culture and transformation of the mentality in the decision makers
of IT (Méndez, 2016; Londoño-Patiño & Acevedo-Álvarez, 2018).
The use of OSS arises as a strategic action for both public and private institutions, and
through which tools from the information and communication technologies (ICTs) can be
employed critically and creatively in order to address specific needs at low cost (Valverde,
2005). Research has been conducted that has focused its efforts on determining variables or
advantages that drive organizations and users towards the adoption of Open Source Software,
which, despite its strengths, is still sparsely used compared to licensed software (Kagiri, He, &
Henglin, 2013).
In this context, the progress of ICT has influenced settings that have enabled content
transformation processes to be stimulated, generating new ways of educating and new areas for
employability related to the creation of new technological and virtual environments (Serrano &
Narváez, 2010). It is here that the use intention of OSS begins to be consolidated, not only in
order to bring about technical breakthroughs, but also for a social purpose, given the existing
ethical and political need to favor this community (Alonso & D’Antoio, 2015).
Among the latest trends in the ICT community is an appreciation for the integration of
open source software (Taha, Abbood, Razzaq, & Al-Bahri, 2018), which has gained popularity
as a highly valuable tool in modern organizational contexts around the world (Kramer, 2014).
In that sense, it should be noted that demand for OSS has increased considerably, strengthening
its adoption and having a significant impact on the software development industry as a result
of its functionality and the trust generated by the multiplicity of options and communities that
provide online support (Koloniaris, Kousiouris, & Nikolaidou, 2018).
There has also been evidence of a upsurge in the adoption of OSS by both governmental
and private entities, with the medium acting as an alternative to traditional proprietary software
(Goode, 2014; Sarrab & Rehman, 2014). In addition, it is important to mention that developing
countries show a significant delay in the adoption of OSS especially by end users. In these
contexts, most users do not buy proprietary software, but benefit from it through the reuse of
licenses, so the tendency to piracy is a common practice (Kamau and Sanders, 2013).
Research Problem
Despite the clear progress made by OSS, limitations to adoption by some sectors still
exist. It is therefore important to raise awareness of the contribution made by incorporating
such software (Moreno-Agudelo & Valencia-Arias, 2017; Taha et al., 2018), given the still
prevalent belief in the centralized innovation model of private software, where only those who
possess the source code are authorized to improve it (González-Barahona, 2008).
Linked to this, despite the increasing worldwide popularity of the adoption of Open
Source Software (Mijinyawa & Abdulwahab, 2014), in the literature there is a gap in research
related to this topic (Lakka, Stamati, & Michalakelis, 2014), so the adoption and use of OSS are
not well understood yet (Mahapatra, Manzar, & Bhadauria, 2015). Due to this need for more
research on the subject, there are many organizations and people who are reluctant or skeptical
to adopt OSS, so it is considered relevant to carry out new research in order to provide supportive
frameworks to formulate new adoption strategies from OSS (Mijinyawa & Abdulwahab, 2014).
The growth dynamic of research by various authors is immediately apparent in the
documents reviewed and the relevant factors over time in the process of users adopting and
using these tools, as are an orientation towards specific elements and the search to improve
acceptance. This is despite the barriers and potential resistance that persist in some settings.
According to these circumstances, the research seeks to answer the question What are the
research trends and developments in the acceptance and use of open source software?

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
116 For this reason, this research aimed to examine the research trends and evolution of the
acceptance and use of open source software. Although the open software initiative (OSI) was
created in 1988 by E. Raymond and B. Perens, only until 2001 was the first research registered
in Scopus directly related to the adoption of free software (Wang, & Wang, 2001). Therefore,
this bibliometric analysis has taken as a period of analysis the documents reported from 2001
to June 2019.
General Background
It should be noted that this research provides a quantitative approach that enables the
data collected to be calculated (Vega-Malagón et al., 2014), while also offering a descriptive
scope in order to record and present in detail the data found regarding the evolution of the
variables observed over a specific timeframe, which is an important foundation in the generation
of scientific knowledge (Sánchez, Blas, & Tujague, 2011). The research also sets out the
primary objective of reviewing the current situation and trends demonstrated by experts in their
research into the adoption process of open source software (OSS), taking into account the most
noteworthy characteristics of the research processes carried out between 2001 and 2019.
Tools and Procedures
The research methodology was based on consulting literature references through

bibliometric analysis, supported by information drawn from the Scopus database, which was
selected taking into account the ease of navigation, access and analysis of a large number of
citations and summaries of specialized scientific texts (peer reviewed), recognized for the quality
of the sources; and especially, for being a repository and valuable tool in the performance of
bibliometric analysis and validation of scientific productions (de Granda-Orive, Alonso-Arroyo,
& Roig-Vázquez, 2011; Martín-Ravelo & Carbonell-de la Fe, 2015). This analysis was used as
a basis to describe bibliometric indicators related to quantity, quality and structure, which are
significant aspects in achieving the results of the research.
Quantitative techniques based on statistical calculations were used for this bibliometric
review, including mathematical and statistical components that are normally used to measure
academic output in different knowledge areas.
The above is significant, given the interest in obtaining the results that are characterized
by scientific accuracy and relevance, which contributes to the presentation of indicators through
methodologies systematized and generalized by the researchers, promoting the verification of
similarities or differences between the indicators used (Bordons & Zulueta, 1999; Camps, 2008).
These techniques contributed to a review of numerous scientific publications that demonstrate
revealing derivations as well as the progress made on the subject, and enabled contrasts to be
drawn between the variables mentioned in the research, such as publication type and aspects of
the authors, institutions and the countries where the publications were produced.
A variety of terms and synonyms associated with the topic in question were considered
in the creation of the search equation, including open source, software, OSS, adoption and
acceptance. Additionally, other aspects vital to obtaining advanced and specific information
(such as title, key words, summary and logical operators) were considered. The Scopus database
was selected due to it being the most extensive source of accessible, reliable information of high
academic validity and quality (with indexed journals containing summaries, research results
and citations from peer-reviewed publications) and a global benchmark for research (Villa,
Ruiz, Valencia, & Picón, 2018). The following structure was defined as the search equation

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
for the procedure: ((TITLE ((“Open source” W/1 software) OR oss) AND TITLE (adoption 117
OR acceptance))) OR ((KEY ((“Open source” W/1 software) OR oss) AND KEY (adoption OR
acceptance))). According to the database reports, as of June 2019 the equation showed a total
of 289 records as a result.
Data Analysis
The most significant derivations provided data related to publications, such as quantity
indicators highlighting the main journals and the productivity levels of authors, institutions and
countries. Indicators linked to quality were also presented, measuring the impact created by the
number of citations achieved by each author and the journal’s number of publications. Finally,
the structure indicators were presented.
The above was carried out in order to identify the topics related to the processes of open
source software adoption, weighing up those publications related to the subject under research
and those themes with a greater chance of development, based particularly on the publications
and fostered by the links described above as the combination of terms included in the search
(open source, software, OSS, adoption, acceptance).
Research Results
About the indicators mentioned in the methodology section, the results generated for the
construction of the bibliometric analysis are shown below.
Quantity and Quality Indicators – Annual Journal Productivity
In terms of the number of publications (annual productivity), Figure 1 demonstrates the

existence of a high rate of growth in the interest in studies involving processes of open source
software adoption in recent years. Despite this, the period that showed the biggest increase in
article publication was between 2004 and 2018, with the most productive year being 2013 with
30 publications.
Figure 1
Number of publications per year

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
118 Figure 2 displays the results for the 10 most important journals according to the number
of publications concerning the topics. ‘IFIP Advances in Information and Communication
Technology’ has occupied first place in the ranking with 20 publications, followed some way
behind by ‘Lecture Notes in Computer Science, including Subseries Lecture Notes in Artificial
Intelligence and Lecture Notes in Bioinformatics’ with 9. In third place was ‘IFIP International
Federation for Information Processing’ with 8 publications, ahead of the ‘International Journal
of Open Source Software and Processes’ and ‘ACM International Conference Proceeding
Series’ with 7 and 6 publications respectively. ‘Ceur Workshop Proceedings’ and the ‘Journal
of Systems and Software’ reported 4 results each, and the ranking is completed by ‘Decision
Support Systems’, ‘Information Systems Research’ and the ‘Journal of Database Management’,
with each having three publications to their credit. This information is of great value in the
process of measuring and confirming potential gaps or particularities concerning the publications
of the various journals.
Figure 2
Number of publications and citations per journal
In Figure 2, the journals with the highest citation rates per publication related to the topic
analyzed can be seen. ‘Information and Management’ was the most cited journal with a total of
103 citations, followed by ‘Computers, Environment and Urban Systems’ with 98 citations per
publication. ‘Information and Software Technology’, the ‘2005 International Symposium on
Empirical Software Engineering (ISESE)’, ‘Communications of the ACM’ and ‘Proceedings-
ICSE 2007 Workshops: Fourth International Workshop on Mining Software Repositories (MSR
2007)’ were next in the ranking with 59, 56, 54 and 48 mentions respectively. The list was
completed by ‘Proceedings of the 15th European Conference on Information Systems (ECIS
2007)’ the ‘Review of Industrial Organization’, ‘IEEE Software’ and the ‘European Journal of
Information Systems, which generated 44, 37, 36.5 and 35 citations in their respective order of
appearance.
Figure 3 shows the results of the analysis of the most productive authors. It shows that
in first place was Kris Ven with 13 publications. Jan Verelst was second in the list with 11,
followed by Xavier Franch, Lidia López and Dolors Costal, with an output of 10, 9 and 8
articles respectively. Next, Yan Li, Angelo Susi and Chuan Hoo Tan all produced 7 publications
each and, finally, 6 articles each were attributed to María Carmela Annosi and Maha Shaikh.
Furthermore, the indicators related to the impact generated by the authors can be seen
in Figure 3, which shows the researchers who boast the highest number of mentions of their
work on the topic at hand. However, it is important to mention that impact factors can give
rise to differences of opinion among experts, as they do not precisely validate the quality

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
of a publication with regard to the information provided. Nevertheless, impact factors do 119
demonstrate the impact of an article in the scientific context, given that the differentiation of the
variables that affect and influence indicators of this type is binding.
Authors Productivity
Figure 3
Publications and citations of the top ten authors
In line with the above, the indicators articulated in this research prove to be relevant
when acknowledging the impact forged by the various authors, coupled with the possibility of
identifying the benchmarks for measuring the data collected. The most citations per author were
received by Andrew Hunter and Stefan Steiniger with 98 each, followed by Samuel Ajila, Di
Wu, Conradi Reidar and Øyvind Hauge with 66 mentions each. Elad Harison and Heli Koski
were next with 60 citations, ahead of Chris Exton and Eugene Glynn with 56 citations and Anol
Bhattacherjee with 54 citations (see Figure 3).
Regarding or about the above classification, no correlation is revealed between the most
productive author and the suggested quality of the publications, according to the number of
articles and the number of citations received.
Additionally, in accordance with the productivity and the type of publications concerning
the adoption of open source software, it is worth mentioning that 55% of the publications arose
from conferences and 40% came from articles, making them relevant tools for the exchange
of knowledge between experts. The progress and relevance of the subject was also evident,
representing a determining factor in generating interest and discussion in the scientific
community and promoting the increased importance and validity of the subject in academic
contexts.
Productivity of Institutions and Countries
About the productivity of institutions, Figure 4 shows that the publications were produced
by 160 different institutions, of which 94 universities (58.75% of the total) produced 80.12% of
the articles, meaning that this specific situation did not conform to the Pareto principle. Thus,
in this case the universities were divided into quartiles according to their level of importance,
reflecting the fact that 25.6% of the publications originate from 7.5% of meaning that this

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
120 specific situation did not conform to the Pareto principle in the institutions. Similarly, the
statistics indicated that 24.38% of the universities produced 50% of the output, while 50.63%
were responsible for 75.30% of the work published. Additionally, 78.13% of the universities
were found to provide 2 or fewer publications, which correspond to 52.4%, demonstrating a
wide spread of knowledge on the topic investigated.
Similarly, it is highlighted that 22.89% of the articles were attributed to universities
positioned in the top 10. The Polytechnical University of Catalonia published the most with 14
articles, University of Antwerp got second place with 13. These were followed by the ESSEC
Business School, Free University of Bozen-Bolzano, National University of Singapore and
Fondazione Bruno Kessler, which produced 7 publications each. The list was completed by the
University of Warwick with 6 articles and HEC Montréal, North Carolina State University and
Youngstown State University with 5 publications each.
Figure 4
Productivity of Institutions and Countries
With regard to the countries that had done the most progress in the generation and
dissemination of scientific knowledge in the field, a total of 62 countries was found, of which
30.65% produced 75% of all the publications. Figure 4 shows that the most influential country
is the United States with 68 articles, followed by Spain with 29, Italy with 24, the United
Kingdom with 20 and Belgium with 16. Australia and France appeared with 14 publications
each, and the ranking was completed by Canada and Germany with 13 publications each.
Similarly, 57.47% of the publications were attributed to the 10 most prevalent countries, while
40 countries (equivalent to 64.52% of the sample) produced 2 or fewer articles per year.
Discussion
The aim of the research was to examine the research trends and evolution of the field of
adoption of open source software between 2001 and 2019. The results of the inquiry provided
important details of the strong trend in the development of research and publications around
the adoption of free software, which highlights countries such as the United States, as their
institutions and researchers who exhibited higher levels of productivity and product circulation
around the topic, this was due to the number of researchers and publications from this country.
Also, there was an interest in reducing gaps and promoting the adoption and use of this tool in
sectors where it has not yet achieved great reception.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Keyword analysis is used to identify the subjects of specific areas of knowledge that 121
are generating significant impact on processes of scientific dissemination by driving trends in
research activities. In this sense, the standardized keyword dynamics is presented below, this
was carried out through a search for the key terms used by researchers in their studies, the
agreement of similar meanings, and the relationship of the keywords obtained in the research.
The keywords are compiled in order to confirm which appear most frequently
in the research. An examination of these terminologies is a definitive tool to describe the
themes outlined by the researchers in the publications and represents an important source of
information.
In this regard, the subjects that feature most frequently in the period studied include
innovation, which had the highest number of results, demonstrating the importance the topic
has acquired on the global scientific stage. Along the same line, innovation is related to the
integration of open source software by organizations, through the adoption and use of a variety
of platforms (online service platforms, open source hardware platforms, free and reduced-cost
3D design tools and collective collaboration tools) whose integration is motivated by factors
associated with high reliability, low cost and the possibility of receiving external assistance.
This drives ecosystems of innovation by bringing together a range of actors and platforms
in order to establish effective collaborative relationships (Bloemen, Amrit, Kuhlmann, &
Ordóñez-Matamoros, 2014; Kwak, Kim, & Park, 2018; Ven & Verelst, 2006).
The keyword that has the second-highest number of mentions in the research is survey.
This research reveals a growing interest in and use of this information gathering technique,
as well as different factors associated with the implications of adopting open source software
for companies. These implications include the need to have and invest in competent human
talent in addition to the benefits to organizational learning processes. (Li, Tan, & Yang, 2013).
Additionally, numerous studies are carried out in order to determine the perception of the
commercial value and the actual contribution of the organizational consolidation of OSS
(Marsan, Paré, & Beaudry, 2012), and innovative strategies regarding the commercialization
and economic aspects of OSS provision are analyzed (Harison & Koski, 2010).
About the term Linux (recognized as an open-source operating system) a range of
inquiries are associated with barriers to the adoption of OSS and potential solutions to the
situation. This is because, despite high growth in use by users in the market, challenges also
exist that are caused by the practices employed by distributors of proprietary software, who
use dissuasive strategies (uncertainty, fear and doubt) to reduce the potential adoption of
open source software (Nagy, Yassin, & Bhattacherjee, 2010). The researchers also indicate
that Linux is a tool that has been institutionalized by organizations at the different phases of
their development and processes, considering the characteristic elements of each company as
predictors of its constant adoption (Xiao, 2006).
In terms of free and open source software, also known as FLOSS, different studies
are used to demonstrate the process of introduction of this technology in different settings
and institutions, the impediments to its adoption and the position taken by the different
stakeholders involved. According to experts, there is a clear difference between OSS adoption
in public sector entities and private institutions. The same is true of the usage gaps between
large companies and small and medium-sized enterprises (SMEs), revealing barriers related
to security guarantees (the trustworthiness and legality of transactions), intellectual property
rights and challenges to managerial, technological, productive and competitive capacities
(Machado, & Van Leeuwen, 2006; Tosi, Lavazza, Morasca, & Chiappa, 2015).
In addition to the above, by becoming a strategic asset, it is also necessary to consider
the importance of the dynamics of participants in the incorporation of FLOSS, as well as the
management risks of the tool given its introduction in companies (Kenett, Franch, Susi, &
Galanis, 2014). As such, an interest arises in reflecting on the developmental trajectory of
the adoption of open source software in the various sectors and organizations (Squire, 2017).

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
122 Furthermore, analysis of the research reveals that in compliance with requirements
engineering, barriers in the usage techniques and development of OSS among communities can
be seen (ecosystem users). In this sense, experimentation takes place in the design of strategies,
models and new techniques that will enhance experiences and the perception of usability (ease
of use) (Costal, López, & Franch, 2015; Llerena, Rodríguez, Castro, & Acuña, 2017).
In addition to the importance acquired by the use of OSS in business activities, the
research also expresses the concerns over the numerous risks in management (risk management),
due to significant flaws in terms of the quality, timeframe and cost of delivery. One of the main
shortcomings is the inadequate management of these risks when implementing solutions based
on open source software. In turn, these risks are sought to be reduced through an understanding
of the behavior and dynamics shown by the communities responsible for offering the software
components (Franch et al., 2015).
The need to appropriately manage the different risks when seamlessly incorporating
this tool in processes is therefore evaluated, ensuring that the activity is effectively managed
and has continuity over time. This is achieved through platforms such as RISCOSS, based
on data modeling and easy to configure, promoting the personalization of numerous types of
settings. It adopts two differentiated forms of work due to its ability to measure the impact of
decisions before they are made, and constantly examines the variables associated with risk
and potential deviations from commercial targets, which can be constantly supervised and fed
back to decision makers (Franch et al., 2015). Understanding, managing and mitigating the
numerous risks associated with the adoption of OSS is therefore crucial in protecting against
adverse impacts that have great potential to affect companies (Ranch et al., 2013).
Furthermore, open innovation has been a key element of the processes of OSS
development, adoption and implementation since its emergence in 2003. In these processes, it
is necessary to consider changes to work roles, individual predisposition and the adoption of
the development of commercial OSS. At the same time, personal incentives should be evaluated
depending on performance and the distinctive features of each institution (Alexy & Henkel,
2007; Munir & Runeson, 2015).
Main Topics in the Research Agenda
Figure 5 shows the research fields that display the highest growth or have tended to
decrease on the global stage, in terms of the adoption of open source software. This information
is supported by the dynamic over time of the keywords, thus facilitating the identification of
the principal trends for the most prevalent themes in the studies, such as those that present
innovative topics that generate interest in the field of knowledge under research.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Figure 5 123
Growing topics
Taking the chosen search as a reference, in Figure 5, the topics developed with greatest
interest by the authors, or those that show highest growth, are those related to requirements
engineering, surveys, FLOSS and open innovation. These results focused on the identification
of the defining aspects of the evolution of open source software adoption as a vital tool in
organizational performance. Figure 5 also reveals the importance of meeting quality, reliability
and security standards in the development of open source software, enabling it to be implemented
effectively by users. Furthermore, experts have also promoted the carrying out of different
investigations that are based on consulting different sectors regarding their perceptions of the
usefulness and effectiveness of using OSS. Similarly, an interest in integrating open innovation
strategies is promoted as a source of growth for communities and progress in the incorporation
of OSS.
The fields that show a downward trend include the keywords “Total cost of ownership”
and “innovation”, and the authors of this article believe that the decreased use of these words
could fundamentally be due to the topic having evolved over time towards other perspectives or
fields of knowledge. As such, the researchers analyzed the focus of their studies on other areas.
The research concludes that examination into the evolution of open source software is
increasing significantly in various settings across the world, where it is developed, distributed
and incorporated in order to improve process efficiency and competitiveness in the industries and
institutions that use it. At present, the research is being carried out that focuses on establishing
frames of reference for OSS implementation in the government and private sector that will
facilitate automation, the integration of security models and growth in innovation ecosystems;
in academic contexts, this research aims to improve learning strategies. This research also
explores the need to identify those factors that affect the adoption of OSS and to rethink the
processes of acceptance of new technologies.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
124 With regard to the growing interest of some sectors in adopting open source software,
this is seen to be influenced by the rise of information technologies and the ease of access to
devices, which emerge as a determining factor in the decisions of users. The use of surveys in
the studies is also apparent, as a means of obtaining significant information on factors related to
the acceptance, use and innovation of open source software by different organizations.
Additionally, the information regarding the progress of open source software that was
collected through the research is noteworthy, given that it provides a statistical basis of the
relevance of OSS and its importance in business and academic activities. Analysis is also
performed of the importance of improving management of compliance with technical quality
specifications and differentiation in the times and costs involved in delivering the product to
users. The research also analyzes the protection of intellectual property rights in a way that will
enhance experiences (accessibility and usability) and drive the adoption of OSS to a greater
extent.
Furthermore, the literature review indicates that open innovation is vital in communities
involved in the development of these tools, as an alternative in the market in response to the
barriers to access and the possibility of absorbing the results of efficient collaboration and
integration in all environments, as a strategic focus in organizational forms, technological
growth and increasing the capacity for business innovation, particularly in so-called developing
countries, in order to expand opportunities and attain competitiveness.
In the results generated from the bibliometric analysis, the evolution of the subject has
passed through various perspectives, from the evaluation and classification of the organizational
adoption of OSS to the measurement of the factors affecting adoption. Examples of the latter
include the level of trust and acceptance, in addition to the notable use of Linux; management
capacities; the effect on learning processes; and the potential benefits of the development and
subsequent commercialization and implementation of open innovation in the routine activities
and processes of different sectors of society. This indicates a strong increasing trend as a result
of the initiatives from the public and private sectors (for employees) that extend access to new
technologies to most of the population, and the interest of users in obtaining free tools that
facilitate their learning and adequate performance in all contexts.
Finally, this research manages to identify the need to direct efforts towards future studies
that will promote the possibility of envisaging the potential of incorporating open source software
in small and medium-sized enterprises in developing countries, and of seeking to devise new
strategies for the implementation of OSS in organizational and educational contexts, finding
its substantiation in the processes of knowledge and information management that permeate
organizational cultures to increase acceptance in the adoption of software of this type.
References
Alexy, O., & Henkel, J. (2007). Promoting the penguin: Who is advocating open source software in
commercial settings? Academic of Management Proceeding, 2007(1), 1–9. https://doi.org/10.5465/
ambpp.2007.26530011
Alonso, A. & D’Antonio, S. (2015). El software libre y el open knowledge como comunidades de
conocimiento paradigmáticas. [Free software and Open Knowledge as paradigmatic knowledge
communities]. Utopía y Praxis Latinoamericana, 20 (Abril-Junio). http://www.redalyc.org/
articulo.oa?id=27942241007
Bloemen, R., Amrit, C., Kuhlmann, S., & Ordóñez-Matamoros, G. (2014). Innovation diffusion in open
source software: Preliminary analysis of dependency changes in the Gentoo portage package
database. In 11th International Working Conference on Mining Software Repositories, MSR 2014
(pp. 316–319). Hyderabad, India. https://doi.org/10.1145/2597073.2597079
Bordons, M., & Zulueta, M. A. (1999). Evaluación de la actividad científica a través de indicadores
bibliométricos. [Evaluation of scientific activity through bibliometric indicators]. Revista
Española de Cardiologia, 52(10), 790–800. https://doi.org/10.1016/S0300-8932(99)75008-6

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Costal, D., López, L., & Franch, X. (2015). Using roles for OSS adoption strategy models. In 8th 125
International i*Workshop, iStar 2015 - In conjunction with the 23rd International Requirements
Engineering Conference, RE 2015 (pp. 19–24). Ottawa; Canada. https://www.scopus.com/record/
display.uri?eid=2-s2.0-84943373547&origin=inward&txGid=a6428450a3166e568e47fd99f96fb
ef0
de Granda-Orive, J. I., Alonso-Arroyo, A., & Roig-Vázquez, F. (2011). Which database should we use for
our literature analysis? Web of Science versus SCOPUS. Archivos de Bronconeumología, 47(4),
2013. https://doi.org/10.1016/j.arbres.2010.10.007
Dhir, S., & Dhir, S. (2017). Adoption of open-source software versus proprietary software: An exploratory
study. Strategic Change, 26(4), 363–371. https://doi.org/10.1002/jsc.2137
Franch, X., Kenett, R., Mancinelli, F., Susi, A., Ameller, D., Annosi, M. C., … Oriol, M., Siena, A. (2015).
The RISCOSS platform for risk management in open source software adoption. In 11th IFIP WG
2.13 International Conference on Open Source Systems, OSS 2015 (pp. 124–133). Florence; Italy.
https://doi.org/10.1007/978-3-319-17837-0_12
Franch, X., Kenett, R. S., Susi, A., Galanis, N., Glott, R., & Mancinelli, F. (2015). Community Data for
OSS adoption risk management. In the Art and Science of Analyzing Software Data (pp. 377–
409). Elsevier. https://doi.org/10.1016/B978-0-12-411519-4.00014-8
Glynn, E., Fitzgerald, B., & Exton, C. (2005). Commercial adoption of open source software: An empirical
study. In 2005 International Symposium on Empirical Software Engineering, ISESE 2005 (pp.
225–234). Queensland; Australia. https://doi.org/10.1109/ISESE.2005.1541831
González, J. C. (2006). B-Learning utilizando software libre, una alternativa viable en Educación Superior.
(B-Learning using free software, a viable alternative in Higher Education). Revista Complutense
de Educación, 17(1), 121-133. https://revistas.ucm.es/index.php/RCED/article/view/16745
González-Barahona, J. M. (2008). Las ventajas de la apertura. [The advantages of opening]. El profesional
de la información, 17(1), 5-7. https://doi.org/10.3145/epi.2008.ene.01
Goode, S. (2014). Exploring organizational information sharing in adopters and non-adopters of open
source software: Evidence from six case studies. Knowledge and Process Management, 21(1),
78–89. https://doi.org/10.1002/kpm.1430
Harison, E., & Koski, H. (2010). Applying open innovation in business strategies: Evidence from Finnish
software firms. Research Policy, 39(3), 351–359. https://doi.org/10.1016/j.respol.2010.01.008
Kagiri, T. M., He, Y., & Henglin, S. (2013). Factors limiting OSS adoption. Oulun, Finland: University
of OULU. Working Paper N° 01.
Kamau, J. W., & Sanders, I. D. (2013). An Empirical Investigation into the Effect of Usability on Adoption
of Desktop Open Source Software by University Students in Kenya. Computer and Information
Science, 6(3), 108-125.
Kenett, R. S., Franch, X., Susi, A., & Galanis, N. (2014). Adoption of Free Libre Open Source
Software (FLOSS): A risk management perspective. In 38th Annual IEEE Computer Software
and Applications Conference, COMPSAC 2014 (pp. 171–180). Vasteras; Sweden. https://doi.
org/10.1109/COMPSAC.2014.25
Koloniaris, S., Kousiouris, G., & Nikolaidou, M. (2018). Possibilities of use of free and open source
software in the greek local authorities. IFIP Advances in Information and Communication
Technology, 525, 128–143. https://doi.org/10.1007/978-3-319-92375-8_11
Kramer, F. (2014). On the advantages, perils and pitfalls of using cloud computing and open source
software in small and medium-sized businesses. The case of a german entrepreneurial company.
In 20th Americas Conference on Information Systems, AMCIS 2014. Savannah, GA; United
States: Association for Information Systems. https://www.scopus.com/record/display.uri?eid=2-
s2.0-84905987557&origin=inward&txGid=1719568f0f089c60c8e5c867144881ab
Kwak, K., Kim, W., & Park, K. (2018). Complementary multi-platforms in the growing innovation
ecosystem: Evidence from 3D printing technology. Technological Forecasting and Social Change,
136, 192–207. https://doi.org/10.1016/j.techfore.2017.06.022
Lakka, S., Stamati, T., & Michalakelis, C. (2014, October). Cross country comparison on the factors
determining OSS diffusion. In Proceedings of the 18th Panhellenic Conference on Informatics
(pp. 1-6). ACM.
Li, Y., Tan, C.-H., & Yang, X. (2013). OSS adoption: Organizational investment in internal human capital.
Journal of Computer Information Systems, 54(1), 42–52. https://doi.org/10.1080/08874417.2013
.11645670

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
126 Llerena, L., Rodríguez, N., Castro, J. W., & Acuña, S. T. (2017). Adoption of the focus groups technique in
the open source software development process. In Lecture Notes in Computer Science (including
subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 10271,
(pp. 325–340). https://doi.org/10.1007/978-3-319-58071-5_25
Londoño-Patiño, J. A., & Acevedo-Álvarez, C. A. (2018). El aprendizaje organizacional (AO) y el
desempeño empresarial bajo el enfoque de las capacidades dinámicas de aprendizaje. Revista
CEA, 4(7), 103-118. https://doi.org/10.22430/24223182.762
Machado, M. J., & Van Leeuwen, M. (2006). Barriers to FLOSS in SMEs: The lack of knowledge and
skills. In 1a Conferencia Iberica de Sistemas e Tecnologias de Informacao, CISTI 2006 - 1st
Iberian Conference on Information Systems and Technologies, CISTI 2006, 1, (pp. 143–155).
Ofir; Portugal. https://www.scopus.com/record/display.uri?eid=2-s2.0-84923867270&origin=in
ward&txGid=3623c1ddbbf1060c11c66732be883685
Mahapatra, R. K., Manzar, R., & Bhadauria, V. S. (2015). Adoption and use of open source infrastructure
software by large corporations: The case of MySQL. Journal of Database Management, 26(4),
1–17. https://doi.org/10.4018/JDM.2015100101
Marsan, J., Paré, G., & Beaudry, A. (2012). Adoption of open source software in organizations: A socio-
cognitive perspective. Journal of Strategic Information Systems, 21(4), 257–273. https://doi.
org/10.1016/j.jsis.2012.05.004
Martín-Ravelo, A., & Carbonell- de la Fe, S. (2015). The scientific production on Artificial Intelligence:
first-quartile journals indexed by Scopus Sciverse. Revista Cubana de Ciencias Informáticas,
9(4), 72–88. http://scielo.sld.cu/pdf/rcci/v9n4/rcci06415.pdf
Méndez, L. (2016). Towards a Reference Framework for Open Source Software Adoption. Proceedings
of CAiSE 2016 Doctoral Consortium. Ljubljana, Slovenia.
Mijinyawa, M. K., & Abdulwahab, L. (2014). An extended framework for evaluation of open source
software adoption in small businesses. Research Journal of Information Technology, 6(4), 248-
269.
Moreno-Agudelo, J. A., & Valencia-Arias, J. A. (2017). Factores implicados en la adopción de software
libre en las Pyme de Medellín. Revista CEA, 3(6), 55-75. https://doi.org/10.22430/24223182.673
Munir, H., & Runeson, P. (2015). Software testing in open innovation: An exploratory case study of
the acceptance test harness for Jenkins. In International Conference on Software and Systems
Process, ICSSP 2015 (pp. 187–191). Tallinn; Estonia. https://doi.org/10.1145/2785592.2795365
Nagy, D., Yassin, A. M., & Bhattacherjee, A. (2010). Organizational adoption of open source
software: Barriers and remedies. Communications of the ACM, 53(1), 148–151. https://doi.
org/10.1145/1666420.1666457
Ranch, X., Susi, A., Annosi, M. C., Ayala, C., Glott, R., Gross, D., … Siena, A. (2013). Managing risk in
open source software adoption. In 8th International Joint conference on Software Technologies,
ICSOFT 2013 (pp. 258–264). Reykjavik; Iceland. https://www.scopus.com/record/display.
uri?eid=2-s2.0-84887052413&origin=inward&txGid=4105149499807129379cffcb952eec42
Sánchez, M. J., Blas, H., & Tujague, M. P. (2011). El análisis descriptivo como recurso necesario
en Ciencias Sociales y Humanas. [Descriptive analysis as a necessary resource in Social and
Human Sciences]. Fundamentos en Humanidades, XI (22), 103–116. https://www.redalyc.org/
html/184/18419812007/
Sarrab, M., & Rehman, O. M. H. (2014). Empirical study of open source software selection for adoption,
based on software quality characteristics. Advances in Engineering Software, 69, 1–11. https://doi.
org/10.1016/j.advengsoft.2013.12.001
Serrano, J. E., & Narváez, P. S. (2010). Uso de software libre para el desarrollo de contenidos Educativos.
[Use of Free Software for the Development of Educational Contents]. Formación Universitaria,
3(6), 41-50. https://doi.org/10.4067/S0718-50062010000600006
Silic, M., & Back, A. (2016). The influence of risk factors in decision-making process for open source
software adoption. International Journal of Information Technology & Decision Making, 15(01),
151-185.
Squire, M. (2017). The lives and deaths of open source code forges. In 13th International
Symposium on Open Collaboration, OpenSym 2017, (15), 1-8. Galway; Ireland. https://doi.
org/10.1145/3125433.3125468

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
Taha, A. M., Abbood, A. A., Razzaq, A. A., & Al-Bahri, A. S. (2018). Identifying the affecting factors for 127
adoption of open source software in it community. Journal of Engineering and Applied Sciences,
13(14), 5771–5780. https://doi.org/10.3923/jeasci.2018.5771.5780
Tosi, D., Lavazza, L., Morasca, S., & Chiappa, M. (2015). Surveying the adoption of FLOSS by public
administration local organizations. In 11th IFIP WG 2.13 International Conference on Open
Source Systems, OSS 2015 (pp. 114–123). Florence; Italy. https://doi.org/10.1007/978-3-319-
17837-0_11
Valverde, J. (2005). Software libre, alternativa tecnológica para la educación. [Free software, technological
alternative for education]. Revista Electrónica “Actualidades Investigativas en Educación”, 5
(julio-diciembre). https://www.redalyc.org/html/447/44750220/
Vega-Malagón, G., Ávila-Morales, J., Vega-Malagón, A. J., Camacho-Calderón, N., Becerril-Santos, A.,
& Leo-Amador, G. E. (2014). Paradigmas en la investigación. Enfoque cuantitativo y cualitativo.
[Paradigms in research. Quantitative and qualitative approach]. European Scientific Journal,
10(15), 523–528. http://www.eujournal.org/index.php/esj/article/viewFile/3477/3240
Velázquez-Juárez, J. A., Valencia-Pérez, L. R., & Peña-Aguilar, J. M. (2016). El papel del modelo
de la triple hélice como sistema de innovación para aumentar la rentabilidad en una Pyme
comercializadora. Revista CEA, 2(3), 101-112. https://doi.org/10.22430/24223182.268
Ven, K., & Verelst, J. (2006). The organizational adoption of open source server software by Belgian
organizations. IFIP International Federation for Information Processing, 203, 111–122. https://
doi.org/10.1007/0-387-34226-5_11
Villa, E., Ruiz, L., Valencia, A., & Picón, E. (2018). Electronic Commerce: Factors Involved in its
adoption from a bibliometric analysis. Journal of Theoretical and Applied Electronic Commerce
Research, 13(1), 39–70. http://dx.doi.org/10.4067/S0718-18762018000100104
Wang, H., & Wang, C. (2001). Open source software adoption: A status report. IEEE Software, 18(2),
90-95.
Xiao, B. (2006). Organization adoption of linux: An institutional perspective. In 12th Americas Conference
on Information Systems, AMCIS 2006 (pp. 1053–1056). Acapulco, Mexico.
Zulueta, M. A., Cabrero, A., & Bordons, M. (1999). Identificación y estudio de grupos de investigación
a través de indicadores bibliométricos. [Identification and study of research groups through
bibliometric indicators]. Revista Española de Documentación Científica, 22(3), 333–347. https://
doi.org/10.3989/redc.1999.v22.i3.341%09
Received: November 14, 2019 Accepted: February 06, 2020

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
128
Cite as: Patiño Toro, O. N., Acevedo Correa, Y., Valencia-Arias, A., & Benjumea-Arias, M.
(2020). A bibliometric analysis of the use of open source software in educational contexts.
Problems of Education in the 21st Century, 78(1), 114-128. https://doi.org/10.33225/
pec/20.78.114
Orfa Nidia Patiño Toro Master in Management of Technological Innovation, Cooperation and Regional
(Corresponding author) Development, Professor, [Institución Universitaria Escolme], University Institution
Escolme, Calle 50 #40-39, Medellín – Colombia, 050013.
E-mail: cies4@escolme.edu.co
Yesenia Acevedo Correa Master (c) in Sociology, Professor, Fundación Universitaria Católica del Norte,
Calle 33 # 74 E – 51, Medellín – Colombia, 050013.
E-mail: yacevedoc@ucn.edu.co
Alejandro Valencia-Arias PhD. in Engineering. MSc. Systems Engineering, Professor, [Instituto Tecnológico
Metropolitano] Metropolitan Technological Institute, calle 54 A #30- 01, Barrio
Boston, Medellín – Colombia, 050013.
E-mail: jhoanyvalencia@itm.edu.co
Martha Luz Benjumea-Arias MSc. Management Engineering, Professor, [Instituto Tecnológico Metropolitano]
Metropolitan Technological Institute, calle 54 A #30- 01, Barrio Boston, Medellín –
Colombia, 050013.
E-mail: marthabenjumea@itm.edu.co

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
129
GUIDE FOR AUTHORS
ISSN 1822-7864 /print/ ISSN 2538-7111 /Online/
General Information
The publication language is English. All authors must take care of the language revision on their
own. The language must be clear and accurate. The work should be written in an impersonal style.
The editor reserves the right to send the manuscript to be reviewed. If English is a second language
for the author, please consider having the manuscript proof read and edited before submitting.
Format of Manuscripts
Title
The main research idea/problem should be reflected in the title. The title should show the nature
of the research/study. It is recommended to avoid the title of the question form. The recommended
length for a title is no more than 12 words (APA, 2.01, p. 23).
The title of the paper 14-point, bold with capital letters, align left; titles in the text (chapters) 12-point
bold, not numbered; sub-titles (subchapters) 12-point, italic, not numbered. There must be 1 empty
line before and after a title or a subtitle. The text chapters must be separated by 1 empty row. The
title should be concise and informative. Avoid abbreviations.
Author names and affiliations
Full names/surnames should be provided. Please indicate affiliations of the author(s). All the e-mails
should be indicated below.
Abstract
The first page of the manuscript must begin with the title of the paper and an abstract which should
be about 150 - 250 words. For the papers reporting original research, state in brief: the primary/
main aim (the research questions addressed or any hypothesis tested); the research design; the
methods and procedures employed; the number of participants; the main outcomes and results; the
conclusions drawn from these data and results, including their implications for further research or
application/practice. An abstract represents briefly a content of a text. Do not cite references in the
abstract. The abstract should grip the reader’s attention.
Keywords
Please provide 3 to 5 keywords in alphabetical order. Note that a keyword does not have to be made
of only one word. At least one of these should indicate the topic area and one should indicate the
methodology of the research.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
130 The structure of the manuscript (recommended)

The body of the text of the manuscript must generally have the following parts:
• abstract + keywords;
• introduction (it can be divided into some subchapters if needed);
• methodology of research (it is recommended to divide /rubricate);
• results of research;
• discussion;
• conclusions and/or implications;
• acknowledgements (if any);
• references (in APA style, 7th Ed);
• appendix (if any).
P.S. The structure can be different if the paper is only theoretical qualitative research.
The length of the manuscript
The manuscript should be not shorter than 6 pages (including references, tables and figures).
Manuscripts should be typed on A4, in Times New Roman 12-point font size, single-spaced, 2.5 cm
with all the margins, word-document (Word 6.0 or later) format.
Acronyms
Acronyms should be defined the first time they appear.
Tables and figures
Tables and figures should be valuable, relevant, and visually attractive. Tables (made in Word or
another software) and figures must be referred to in the text and numbered in the order of their
appearance. Each table and figure should have a complete, descriptive title; and each table column
an appropriate heading. The texts in tables and figures should be 11-point (in some cases 10) and
their width should be 12 cm at maximum. The figures should be in format .jpg (unless done in Word
or Excel); resolution 1200 dpi. Figures, tables (black and white colour, without ground-colour) and
captions should be inserted within the manuscript at their appropriate locations. All the graphics
(figures) must be editable. The font size should be Times New Roman for all figures and tables.
Figures should be carefully explained in the text and cited in numerical order.
It is the author’s responsibility to obtain permission to reproduce illustrations, tables, etc. from other
publications.
Discussion
Focus the discussion to the two important questions: What is already known about your topic? What
your research adds? It is recommended that the aim and major findings of the research should be
reminded. And then the similarities and differences of the findings with previous research studies
should be exemplified in detail.
Conclusions
This part is not a summary. Bring out the significance of your research. Show how you’ve brought
closure to the research problem, and point out remaining gaps in knowledge by suggesting issues for
further research. The main research outcome should be clearly seen.

PROBLEMS
OF EDUCATION
IN THE 21st CENTURY
Vol. 78, No. 1, 2020
References 131
The title „References“ must be used. APA style for writing references in the text and in the reference
list must be used. References in the text should be presented in parentheses (Knox, 1988; Martin,
1995). If necessary, the page can be indicated: (Martin, 1995, p. 48). The list of references should
be presented after the text.
The author should make sure that there is a strict one-to-one correspondence between the names and
years in the text and those on the list. All the references should be listed in alphabeti¬cal order by
author’s name.
For the sake of accuracy, references should always be as updated as possible, also in the interest of
the readers and researchers on the topic the paper is dealing with.
Notes
Note 1: Manuscripts in which references are not in the APAstyle will be returned without review.
Note 2: References to online sources should include the type of medium (such as “serial online”
or “monograph online”), the date of that specific reference (if applicable), the uniform resource
locator (URL), and the date that the source was accessed. A source accessed online should al-ways
be referenced accordingly, even if it is also published in printed form.
Note 3: All papers must meet the criteria of originality and scientific quality. Obviously, they must
also comply with style and format requirements. The paper will not be subject to further review, if
the manuscript is NOT WITHIN THE SCOPE and/or there is POOR USAGE OF LANGUAGE (all
manuscripts must be written in clear and grammatical English).
Note 4: Submitted papers will be assessed based on their novelty, technical quality, potential impact,
and clarity of writing.
Note 5: All papers are checked by CrossCheck system.
Editorial Board
Updated: July 2018
Website: http://www.scientiasocialis.lt/pec/

Problems of Education in the 21st Century,
Vol. 78, No. 1, 2020
Compiler Vincentas Lamanauskas

Designer Jurgina Jankauskienė
Paste-up artist Loreta Šimutytė-Balčiūnienė
English language proofreader Ilona Ratkevičienė
20 February 2020. Publishing in Quires 8,25. Edition 200.
Publisher Scientia Socialis Ltd. in cooperation with SMC „Scientia Educologica“

Donelaicio Street 29, Siauliai, Lithuania
E-mail: gu@gu.puslapiai.lt
Phone: +370 687 95668

Printing „Šiaulių spaustuvė“ Ltd.

9A P. Lukšio Street
LT-76207 Šiauliai, Lithuania
Phone: +370 41 500 333.
Fax: +370 41 500 336
E-mail: info@dailu.lt

Problems of Education in The 21st Century, Vol. 78, No. 1, 2020

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Problems of Education in The 21st Century, Vol. 78, No. 1, 2020

Uploaded by

Copyright:

Available Formats

Problems of Education in the 21st Century, Vol. 78, No.

Address: Scientia Socialis, Ltd.

A FEW REFLECTIONS ON CONTEMPORARY SCIENCE EDUCATION AND

STUDENTS’ ATTITUDES TO SELECTED SUBJECTS IN SECONDARY

IMPLEMENTATION OF MOBILE LEARNING USING A SOCIAL NETWORK

THE RELATION BETWEEN SCHOOL CULTURE SUB-CATEGORIES AND

COMPETENCY-BASED ASSESSMENT AND THE USE OF PERFORMANCE-BASED

MATHEMATICAL LITERACY PROFICIENCY DEVELOPMENT BASED ON CONTENT,

EFFECT OF LEARNING CLIMATE, THINKING PATTERN, AND CURIOSITY

A BIBLIOMETRIC ANALYSIS OF THE USE OF OPEN SOURCE SOFTWARE

INSTRUCTIONS FOR AUTHORS ............................................................................. 129

ISSN 1822-7864 (Print) ISSN 2538-7111 (Online)

https://doi.org/10.33225/pec/20.78.04 ISSN 1822-7864 (Print) ISSN 2538-7111 (Online)

ISSN 1822-7864 (Print) ISSN 2538-7111 (Online) https://doi.org/10.33225/pec/20.78.04

https://doi.org/10.33225/pec/20.78.04 ISSN 1822-7864 (Print) ISSN 2538-7111 (Online)

ISSN 1822-7864 (Print) ISSN 2538-7111 (Online) https://doi.org/10.33225/pec/20.78.04

Received: December 22, 2019 Accepted: January 30, 2020

Katarzyna Potyrała Dr hab., Professor, Head of Department of Didactics and Methodology of

https://doi.org/10.33225/pec/20.78.04 ISSN 1822-7864 (Print) ISSN 2538-7111 (Online)

STUDENTS’ ATTITUDES TO SELECTED 9

ISSN 1822-7864 (Print) ISSN 2538-7111 (Online) https://doi.org/10.33225/pec/20.78.09

10 Considering practical implications of the theoretical construct, motivation is observable

https://doi.org/10.33225/pec/20.78.09 ISSN 1822-7864 (Print) ISSN 2538-7111 (Online)

ISSN 1822-7864 (Print) ISSN 2538-7111 (Online) https://doi.org/10.33225/pec/20.78.09

https://doi.org/10.33225/pec/20.78.09 ISSN 1822-7864 (Print) ISSN 2538-7111 (Online)

Group Absolute frequency Relative frequency (%)

Instrument and Procedures

ISSN 1822-7864 (Print) ISSN 2538-7111 (Online) https://doi.org/10.33225/pec/20.78.09

https://doi.org/10.33225/pec/20.78.09 ISSN 1822-7864 (Print) ISSN 2538-7111 (Online)

Correlation of the Importance, Popularity and Attractiveness in Case of Each Subject

Importance Popularity Difficulty Attractiveness

Importance .001 .529 .720 -.015 .001 .478

Popularity .001 .529 .0356 -.088 .001 .670

Difficulty .720 -.015 .0356 -.088 .509 -.028

Attractiveness .001 .478 .001 .670 .509 -.028

In the case of accounting, the research revealed a positive correlation of reviewed

Importance Popularity Difficulty Attractiveness

ISSN 1822-7864 (Print) ISSN 2538-7111 (Online) https://doi.org/10.33225/pec/20.78.09

16 As regards mathematics, the research proved – at the significance level of 5% – major

Importance Popularity Difficulty Attractiveness

Importance .001 .638 .0003 .150 .001 .655

Popularity .001 .638 .040 .086 .001 .735

Difficulty .0003 .150 .040 .086 .004 .118

Attractiveness .001 .655 .001 .735 .004 .118

Importance Popularity Difficulty Attractiveness

Importance .001 .494 .001 .133 .001 .471

Popularity .001 .494 .009 .109 .001 .670

Difficulty .001 .133 .009 .109 .001 .064

Attractiveness .001 .471 .001 .670 .001 .064

https://doi.org/10.33225/pec/20.78.09 ISSN 1822-7864 (Print) ISSN 2538-7111 (Online)

ISSN 1822-7864 (Print) ISSN 2538-7111 (Online) https://doi.org/10.33225/pec/20.78.09

Similarities and Differences of the Selected Subjects with Respect to the

Firstly, a review of significant differences between subjects based on each analyzed

https://doi.org/10.33225/pec/20.78.09 ISSN 1822-7864 (Print) ISSN 2538-7111 (Online)

Rank Importance Popularity Difficulty Attractiveness

1. Economics Economics Economics Economics

2. Accounting Accounting Accounting Accounting

3. Mother tongue Mother tongue Mother tongue Mother tongue

4. Mathematics Mathematics Mathematics Mathematics

ISSN 1822-7864 (Print) ISSN 2538-7111 (Online) https://doi.org/10.33225/pec/20.78.09

https://doi.org/10.33225/pec/20.78.09 ISSN 1822-7864 (Print) ISSN 2538-7111 (Online)

ISSN 1822-7864 (Print) ISSN 2538-7111 (Online) https://doi.org/10.33225/pec/20.78.09

https://doi.org/10.33225/pec/20.78.09 ISSN 1822-7864 (Print) ISSN 2538-7111 (Online)