Learning Environ Res (2018) 21:285–300

https://doi.org/10.1007/s10984-017-9255-8

ORIGINAL PAPER

Evaluation of engineering and technology activities

in primary schools in terms of learning environment,
attitudes and understanding

Rekha B. Koul1 • Barry J. Fraser1 • Nicoleta Maynard1 •

Moses Tade1

Received: 5 July 2017 / Accepted: 31 October 2017 / Published online: 7 November 2017
 Springer Science+Business Media B.V., part of Springer Nature 2017

Abstract Because the importance of science, technology, engineering and mathematics

(STEM) education continues to be recognised around the world, we developed and vali-
dated an instrument to assess the learning environment and student attitudes in STEM
classrooms, with a specific focus on engineering and technology (E&T) activities in pri-
mary schools. When a four-scale instrument assessing classroom cooperation and
involvement and student enjoyment and career interest was administered to 1095 grade 4–7
students in 36 classes in 10 schools, data analyses supported its factorial validity and
reliability. When the new questionnaire and understanding scales were used to evaluate
E&T activities, statistically-significant pretest–posttest changes in career interest and
understanding (with large effect sizes ranging from 0.70 to 0.81 standard deviations)
supported the efficacy of the instructional activities.

Keywords Attitudes
Learning environment
Primary education
STEM

education

Introduction

The World Economic Forum’s (2017) report entitled Realizing Human Potential in the
Fourth Industrial Revolution recognises the failure of world’s educational systems to
prepare children for the workplaces of the future, which are predicted to be focused on
engineering and technology (E&T). The current international focus on STEM education
includes a push to integrate the separate science, technology, engineering and mathematics

& Rekha B. Koul

R.Koul@curtin.edu.au
Barry J. Fraser
B.Fraser@curtin.edu.au
1
Curtin University, GPO Box U 1987, Perth, WA 6845, Australia

123
286 Learning Environ Res (2018) 21:285–300

(STEM) disciplines at the school level. In the project described in this article, our research
team developed and evaluated instructional activities that integrated E&T.
The connections between the science, technology, engineering and mathematics disci-
plines, as well as the distinct features of each discipline, have been recognised at all
educational levels starting with early childhood education (Early Childhood STEM
Working Group 2017). Each discipline warrants subject-specific attention to ensure that
children build their foundational subject-matter knowledge systematically and experience
the creativity, beauty and unique features of these disciplines. Although there are numerous
rich and valuable approaches to integrating STEM disciplines, one of the best approaches
typically involves one discipline being in the foreground (i.e. the focus of the activity),
while one or more of the other disciplines provide further background in STEM literacy,
and the integration of four interrelated and complementary components (Bybee 2013):
• Acquiring STEM knowledge and using it to identify issues, acquire new knowledge
and apply the knowledge to the issues
• Understanding the characteristic features of STEM disciplines as forms of human
endeavour that include the processes of inquiry, design and analysis
• Recognising how STEM disciplines shape our material, intellectual and cultural world
• Engaging in STEM-related issues and with the ideas of science, technology,
engineering and mathematics as concerned, effective and constructive citizens.
Recent changes to global economies have made the teaching of STEM subjects
increasingly important (Kennedy and Odell 2014). Despite massive advances in science,
few citizens worldwide are technologically literate (Lachapelle and Cunningham 2007). As
the world moves towards a knowledge-based economy, STEM literacies and skills are
required for both employment and meeting societal demands (Bundy 1999; Gee 2012;
Murcia 2009; Naisbitt and Aburdene 1990).
Graduates in a wide range of careers report that their STEM knowledge and skills are
helpful in both their work and personal lives (Harris 2012). Furthermore, because the
problem of declining enrolments in school students’ preferences for STEM subjects is a
national concern, young people’s interest in STEM needs to be stimulated and maintained
throughout schooling so that they choose to continue with studies in these fields at the
university level, thereby ultimately contributing to redressing the skills shortage in STEM
(Australian National Engineering Taskforce 2010).
In Australia, Goodrum and Rennie (2007) recognised a decade ago that high-quality
teaching of science in primary schools is a national priority if citizens are to be scientif-
ically literate, contribute to social and economic well-being, and achieve their own
potential. Despite this priority, Australian school students’ performance in science and
mathematics has not improved over the past decade in international or national testing
programs such as the Trends in International Mathematics and Science Study (TIMSS),
Programme for International Student Assessment (PISA) and National Assessment Pro-
gram–Literacy and Numeracy (NAPLAN).
For TIMMS in 2015, Australian mathematics students were outperformed by students
from 21 counties, including Northern Ireland, Ireland, England and the United States, as
well as participating East Asian countries such as Singapore, Hong Kong, Korea, Chinese
Taipei and Japan. Australian students’ scores have been constant since 2007, while scores
for countries such as England and Singapore have been constantly improving. Overall there
has been no improvement in science or mathematics achievement for Australian students in
this age group between TIMMS 1995 and TIMMS 2015 (ACER 2016a).

123
Learning Environ Res (2018) 21:285–300 287

For PISA, Australian students recorded a significant decline of 17 points in scientific

literacy and 10 points in mathematical literacy between 2012 and 2015 (ACER 2016b).
For NAPLAN, numeracy achievement at the national level for grade 3, 7 and 9 students
remained unchanged from 2008 to 2016, but there was an increase of 17 score points in
mean year 5 numeracy achievement during the same time period. In general, NAPLAN
data suggest that improvements in numeracy were smaller than for reading (Goss et al.
2016).
Clearly, primary-school students in Australia and worldwide need extra support in order
to improve their understanding of STEM, which is necessary for participation in an
increasingly scientific and technological world (Williams 2001) and for enhancing eco-
nomic prosperity by having a highly-educated workforce (Office of the Chief Scientist
2014; Riegle-Crumb et al. 2012). Many countries have significantly invested in STEM
educational initiatives largely because of concerns about potential future shortfalls in
STEM-qualified professionals (van Langen and Dekkers 2005). In Australia, robust per-
formance in STEM is considered a key imperative because the educational sector is the
fourth largest export industry in the country.
The Office of the Chief Scientist (2014) emphasises the important role of education in
preparing a skilled and dynamic STEM workforce through (1) strong STEM teaching at all
levels, supported by high-quality and relevant teacher training and subject-specific pro-
fessional development and (2) increased numbers of subject-qualified STEM teachers in
Australian schools (p. 20). Clearly, opening young minds to the wonders of the natural
world, stimulating curiosity and creative thinking, and starting a journey towards scientific
and technological literacy require a strong and effective primary-school STEM program
that promotes positive learning environments and student interest and understanding in
STEM.
STEM education has the potential to advance knowledge, attitudes and skills through
the innovative solution of real-life complex contextual problems, drawing evidence-based
conclusions, using current technologies and tools, and developing STEM literacy (Ken-
nedy and Odell 2014; Zollman 2012). STEM literacy should not be viewed as a set of skills
developed by students, but as a socio-cultural practice based on the negotiation of meaning
construction (Norton 2007). STEM literacy is a multiple literacy that is based on the ability
to recognise, apply and assimilate concepts through the four disciplines and use them to
understand complex real-life problems (Milliken and Adams 2010). STEM education can
further learning and integrate scientific literacy, technological literacy, engineering literacy
and mathematical literacy into a new literacy that is non-discipline specific, but harnesses
the cognitive structures of the four disciplines (Zollman 2012).

Defining STEM literacy

Scientific literacy has long been recognised as a compound ability incorporating many
attributes (Laugksch 2000). Students who are scientifically literate can ‘‘use scientific
knowledge and processes not only to understand the natural world, but also to participate in
decisions that affect it’’ (OECD, in Schibeci 2011, p. 10). This definition was also used as
the basis for the Australian National Assessment Program–Science Literacy (ACARA
2010).
Technological literacy is the ability to understand the designed world and its tools and
infrastructure. It involves the practical skills needed to use appropriate tools, as well as the
ability to understand, use and evaluate technology and technological principles (Schibeci

123
288 Learning Environ Res (2018) 21:285–300

2011), Technologically-literate people can demonstrate creativity, solve problems and

understand how technology is shaped by and, in turn, shapes society (Zollman 2012).
Engineering literacy is understanding the development of technologies through the
design process, incorporating the ability to apply scientific and mathematical principles to
practical ends in designing, manufacturing and operating effective structures (Zollman
2012). Focusing on the Design and Technologies learning area, the Engineering component
of many STEM programs aims to ensure that students can respond to challenges in
designing efficient solutions for specific needs (ACARA 2011).
Mathematical literacy involves recognising and understanding of the role of mathe-
matics in the world (Schibeci 2011). It is the capacity to use mathematics to make rational
judgements, communicate creatively about problem situations, comprehend mathematical
representations and develop understandings of mathematics (Zollman 2012). This capacity
is represented in the curriculum as the three mathematical proficiencies of Understanding,
Fluency, and Problem Solving and Reasoning (ACARA 2011).
STEM literacy is an indeterminate concept that is the sum of the above individual
literacies, but it is also much greater than the sum of the individual parts. STEM literacy
includes content knowledge of the four distinct disciplines, their skills and procedural,
cognitive conceptual and factual knowledges and, even more significantly, their epistemic
structures (Furtak et al. 2012) and the metacognitive capacities required to integrate and
consolidate this knowledge (Zollman 2012). Multiple literacies such as STEM literacy
promote active engagement and learning, as well as the use of interdisciplinary knowledges
to think holistically and link knowledge areas (Masny and Cole 2007). Through STEM and
STEM literacy, students develop the capability to critically apply STEM concepts,
therefore preparing them for secondary and post-secondary education pathways and
leading them to more financially-rewarding careers (Milliken and Adams 2010).

Learning environments

Learning environment research encompasses study of the social, physical, psychological

and pedagogical contexts in which learning occurs and affects student achievement and
attitudes (Fraser 2012, 2014). This field of educational research builds our understanding of
pre-primary, primary, high-school, college and university, and lifelong learning environ-
ments irrespective of subject area.
Our study drew upon and contributed to the field of learning environments (Fraser
2012, 2014). A hallmark of the field of learning environments is the existence of a variety
of economical, robust and extensively-validated questionnaires that measure different
psychosocial dimensions of the classroom from the perspective of the students who make
up a classroom rather than from the perspective of trained observers or teachers (e.g.
Aldridge and Fraser 2008; Fraser, Giddings and McRobbie 1995; Goh and Fraser 2000;
Taylor, Fraser and Fisher 1997; Teh and Fraser 1995). The widely-used What Is Happening
In this Class? (WIHIC), which has been used by dozens of researchers, hundreds of
teachers, and tens of thousands of students (Aldridge, Fraser and Huang 1999; Fraser
2012), was drawn upon in our study to provide two learning environment scales for our
research: Cooperation and Involvement.
Research reviews (e.g. Fraser 2012, 2014) reveal that classroom environment ques-
tionnaires have been used internationally in a wide range of research applications. For
example, many researchers have reported consistent associations between students’ cog-
nitive and affective outcomes and their learning environment perceptions (Fraser and Butts
1982; Cohn and Fraser 2016). Other researchers have investigated the usefulness of

123
Learning Environ Res (2018) 21:285–300 289

learning environment ideas in school psychology (Burden and Fraser 1993), differences
between students’ preferred and actual environments (Byrne et al. 1986; Fraser and Fisher
1983) and studies of exemplary teachers (Fraser and Tobin 1987). As well, teachers have
used assessments of classroom environment as feedback in action research aimed at
improving their classrooms (Aldridge et al. 2004; Fraser 1981a).
One of the most promising applications of classroom environment assessments is their
use as process criteria of effectiveness in evaluating educational programs. In this study,
we used learning environment criteria in evaluating the effectiveness of an implemented
E&T program. For example, learning environment criteria have been used in evaluating the
effectiveness of using anthropometric activities (Lightburn and Fraser 2007), inquiry-based
laboratory activities (Wolf and Fraser 2008), student response systems (Cohn and Fraser
2016), an innovative mathematics program (Spinner and Fraser 2005), field-study science
classrooms (Zaragoza and Fraser 2017), reality pedagogy (Sirrakos and Fraser 2017) and
mathematical games (Afari et al. 2013).

Affective outcomes

The very high importance of affective outcomes in STEM education continues currently
around the world (Khine 2015; Kind, Jones and Barmby 2007; Tytler and Osborne 2012).
A questionnaire used extensively over the past 40 years to measure students’ attitudinal
outcomes is the Test of Science Related Attitudes (TOSRA) (Fraser 1981a, b). Not only has
TOSRA been used successfully with science classes (e.g. Fraser et al. 2010; Wong and
Fraser 1996), but also it has been found to be valid and useful after adaptation for other
school subjects such as mathematics (Ogbuehi and Fraser 2007), geography (Walker 2006)
and English (Liu and Fraser 2013). TOSRA comprises seven scales, with ten items in each
scale. Items from TOSRA were adapted for use in this study to measure two important
aspects of students’ affective outcomes: Enjoyment of E&T Lessons and Career Interest in
E&T.

Objectives

The overarching aim of our E&T teaching materials was to promote interest and under-
standing among elementary school students in science, technology, engineering and
mathematics (STEM) as potential career choices through creating engaging learning
environments. The purposes of the study reported in this paper were to:
• validate a questionnaire for assessing students’ perceptions of E&T classroom learning
environments and their attitudes to E&T
• evaluate the effectiveness of implemented E&T lessons in terms of (1) the quality of
the classroom learning environment, (2) students’ attitudes to E&T and (3) students’
understanding of E&T
• investigate the differential effectiveness of E&T materials for male and female students
in terms of (1) the quality of the classroom learning environment, (2) students’ attitudes
to E&T and (3) students’ understanding of E&T.

123
290 Learning Environ Res (2018) 21:285–300

Methods

We followed the common approach in learning environments research (Fraser 2012, 2014)
of evaluating educational innovations in terms of pretest–posttest changes during
instruction in salient variables (classroom environment and students’ attitudes and
understanding). Ethics approval for our study was granted by the Human Research Ethics
Committee of Curtin University.
Quantitative information was collected through questionnaires with three components.
First, we selected and modified two learning environment scales from the widely-used and
validated WIHIC (What Is Happening In this Class?, Aldridge et al. 1999), namely,
Cooperation and Involvement. For the sake of economy for the young age range of the
students involved, only five items from each of the original eight-item scales were selected.
Keeping in mind that data collection involved the assessment of three components
(learning environment, attitudes and understanding) and that we wanted to administer all
scales to young children in one class period, we decided to choose only the two most
appropriate WIHIC scales. These two scales of Cooperation and Involvement were con-
sidered to be the most suitable by the research team for measuring the effectiveness of
E&T activities because they involve group activities.
Second, we selected and modified two student attitude scales from the widely-used and
extensively-validated TOSRA (Test of Science Related Attitudes, Fraser 1981b), namely,
Enjoyment of E&T Lessons and Career Interest in E&T. Only five items from each original
10-item TOSRA scale were used for economy. All four learning environment and attitude
scales were modified for young children to have only three response alternatives (Disagree,
Not Sure and Agree). All items in the Cooperation, Involvement, Enjoyment and Career
Interest scales are listed in Fig. 1.
Third, because of the importance of promoting understanding (Treagust, Duit and Fraser
1996), we designed a questionnaire to probe students’ understanding of engineering and
technology by asking them ‘‘What does an engineer do?’’ and ‘‘What is technology?’’ and
using eight pictures as prompts for engineering items and six pictures as prompts for
technology items.
After the administration of the questionnaires as a pretest, a lesson defining engineering
and the importance of science, engineering and technology was delivered by two of the
authors. In addition, at least one engineering topic (two to three lessons), chosen by the
class teachers, was taught in these classes. Researchers used lesson plans from ‘Tryengi-
neering’ (www.tryengineering.org) as a guide and modified them to fit local needs.
Teachers were given the lesson plans, materials and any other support needed for these
lessons. These lessons were observed by at least one researcher.
As described by Koul et al. (2016), E&T lessons were designed to provide an appro-
priate and engaging curriculum and to be closely integrated with the primary science topic
that was being taught in each of the participating schools. Each unit fitted within a small
timeframe of 2–3 lessons, maximised cross-disciplinary connections and, above all, was
carefully designed to address the objectives of the study. Three lessons described by Koul
et al. (2016) are summarised below.
Give me a Brake focuses on brakes, force and friction, using bicycle rim brakes to
demonstrate basic braking mechanisms to stop, slow or prevent motion. The topic was
introduced with a short Powerpoint presentation that introduced science (friction), tech-
nology (bicycle brakes), engineering (design) and mathematics (measurement) ideas into
the activity. This 45-min activity aimed to teach students how simple rim brake systems

123
Learning Environ Res (2018) 21:285–300 291

Cooperation
I cooperate with other students when doing science work.
I share my books and resources with other students when doing work in science.
I work with other students on projects in this class.
I learn from other students in this class.
Students work with me to achieve class goals.

Involvement
I discuss ideas in class.
The teacher asks me questions
My ideas are used during classroom discussions.
I ask the teacher questions.
Other students in the class discuss problems with me.

Enjoyment of E & T Lessons

Science lessons are fun.
School should have more science lessons each week.
Science is one of the most interesting school subjects.
I enjoy going to science lessons.
I look forward to science lessons.

Career Interest in E & T

When I leave school, I would like to work with engineers.
Working as an engineer would be an interesting way to earn a living.
I would like to teach engineering when I leave school.
A job as an engineer would be interesting.
I would like to be an engineer when I leave school.

Fig. 1 Listing of items in four learning environment and attitude scales. Response alternatives are disagree
(scored 1), Not Sure (scored 2) and agree (scored 3)

work for bicycles. Teams of three students each explored how different materials react
when used in a braking system to stop the motion and then they evaluated the design and
materials used in standard bicycle brakes. Students developed and presented a design for
improved safety using words and sketches given in information sheets. Students were
encouraged to make amendments to improve the design. The teams then presented their
ideas to the class.
Oil Spill Solutions focuses on the use of various techniques to provide speedy solutions
for oil spills or other threats to natural water resources. Effort was made to integrate all the
four streams of STEM by bringing in science (environmental impact), technology (testing
the system), engineering (design of system) and mathematics (measurement) aspects.
Students worked in teams of three to analyse an ‘oil spill’ in the classroom and then to
design, build and test a system that first contained and then removed the oil from the water.
Students selected items from everyday use to build oil containment and clean-up systems,
evaluate the effectiveness of their solution and those of other teams, and present their
findings to the class.
Solar Structures focuses on how the sun’s energy can be used to heat and cool buildings.
Teams of students were asked to design and build a one-bedroom model house, using
passive solar heating techniques to warm up the house as much as possible. The house
needed to be big enough to read a thermometer inside the middle of the model with the
door closed and the thermometer to be read through the window. The passive solar houses
were constructed from everyday materials and tested to determine how well they regulate
temperature. In contrast to the first two activities, Solar Structures took a longer time
(approximately 2 h) to complete. Concepts of science (passive solar design convection,

123
292 Learning Environ Res (2018) 21:285–300

radiation and conduction), technology (regulating the temperature of the building with
electronic devices such as thermostats or blinds), engineering (design in the form of cross
ventilation, vegetation, colours used, direction of windows) and mathematics (amount of
heat generated, length and breadth of windows, angle of the sun from the location,
recording the temperature) were introduced.
After instruction, our questionnaires were re-administered as posttests in order to
identify any changes in classroom environment or students’ attitudes and understanding in
E&T during the intervention.
Our sample comprised 1095 grade 4–7 students (570 male and 525 female) in 36 classes
in 10 schools (see Table 1). Classes consisted of only grade 4, only grade 6 or a composite of
grades 4/5, grades 5/6 or grades 6/7. Approximately 10% of participating students reported
that one of their parents works or is qualified as an engineer. The study was carried out in
suburban schools in Perth, Western Australia with a mix of private and public schools. All
schools were co-educational. No information about students’ cultural backgrounds or lan-
guages spoken at home were collected, because that was beyond the scope of this study.

Results

Validity of learning environment and attitude scales

Quantitative data collected from the 36 classes (N = 1095) were analysed in various ways
to investigate the validity and reliability of the 20-item 4-scale questionnaire assessing
learning environment and student attitudes. Principal axis factor analysis with varimax
rotation and Kaiser normalisation was used to check the structure of our questionnaire
items for assessing two learning environment scales (Cooperation, Involvement) and two
attitude scales (Enjoyment, Career Interest). Our criteria for the retention of any item were
that it must have a factor loading of at least 0.4 on its own scale and at least 0.4 on each of
the other three scales. Factor analyses were conducted separately for pretest and posttest
data (see Table 2).
Table 2 shows that every one of the 20 items was retained because it had a factor
loading of at least 0.40 on its own scale and less than 0.40 on each of the other scales for

Table 1 Sample size according

Demographic N %
to gender, grade level and whe-
ther a parent is an engineer
Gender
Males 570 52.1
Females 525 47.9
Grade level
4 385 35.1
4/5 105 9.6
5/6 245 22.4
6 290 26.5
6/7 70 6.4
Whether parent is an engineer
Yes 96 8.8
No 999 91.2

123
Learning Environ Res (2018) 21:285–300 293

Table 2 Factor analysis results and internal consistency (Cronbach alpha reliability) for pretest and posttest
for learning environment and attitude scales
Item no. Cooperation Involvement Enjoyment Career interest

Pre Post Pre Post Pre Post Pre Post

1 0.66 0.61
2 0.45 0.51
3 0.54 0.44
4 0.47 0.46
5 0.53 0.67
6 0.68 0.64
7 0.47 0.69
8 0.46 0.55
9 0.56 0.53
10 0.51 0.51
11 0.82 0.89
12 0.75 0.76
13 0.66 0.75
14 0.86 0.91
15 0.81 0.85
16 0.83 0.78
17 0.59 0.43
18 0.62 0.72
19 0.61 0.44
20 0.85 0.86
% Variance 6.22 5.71 12.75 10.76 20.45 27.92 13.22 16.16
Eigenvalue 1.24 1.14 2.55 2.15 4.09 5.58 2.64 3.25
a Reliability 0.90 0.91 0.88 0.94 0.90 0.98 0.88 0.89

N = 1095. Factor loadings less than 0.40 have been omitted from the table
Principal axis factoring with varimax rotation and Kaiser normalisation

both pretest and posttest data. The bottom of this table shows that the proportion of
variance accounted for by different scales ranged from 6.22 to 20.45% (total for the four
scales of 52.64%) for the pretest and from 5.71 to 27.92% (total of 60.55%) for the posttest.
Eigenvalues for the different scales ranged from 1.24 to 4.09 for the pretest and from 1.14
to 5.58 for the posttest.
Cronbach’s (1951) alpha coefficient was used as an index of the internal consistency
reliability of each scale. Table 2 shows that scale alpha reliabilities ranged from 0.88 to
0.90 for the pretest and from 0.89 to 0.94 for the posttest, therefore attesting to the internal
reliability of the constructs.

Effectiveness of STEM activities in terms of individual understanding items

We probed children’s perceptions of E&T by asking them ‘‘What does an engineer do?’’
and ‘‘What is technology?’’, to draw examples of what an engineer does and what tech-
nology is, and to describe their pictures in words. Initially, students generally had a poor

123
294 Learning Environ Res (2018) 21:285–300

idea about the work of engineers but had a stronger understanding about technology. To
systematically probe students’ perceptions of E&T, the questionnaire included captioned
images of working people from which the students had to choose those that showed what
an engineer would be expected to do at work (e.g. designs electrical circuits, designs
machines) and other images that might or might not represent technology (e.g. television,
spaceship).
As part of our evaluation of the E&T teaching, we examined changes in students’
understandings/perceptions of E&T during the instructional activities in terms of effect
sizes (i.e. numbers of standard deviations) and statistical significance tests (see Table 3).
Cohen’s (1992) d effect size is calculated by dividing the difference between pretest and
posttest means by the pooled standard deviation. To ascertain the statistical significance of
pretest–posttest changes for the engineering questionnaire items, we conducted a one-way
MANOVA with repeated measures with the 8 engineering items as the set of dependent
variables. Because pretest–posttest differences for the 8 items as a whole (using Wilks’
lambda criterion) were statistically significant, the one-way ANOVA was interpreted for
each of the individual items. A parallel MANOVA was conducted for the 6 technology
items. Table 3 reports the ANOVA results.
In reporting pretest–posttest changes, Table 3 only includes effect sizes greater than 0.2
standard deviations based on Cohen’s (1992) criteria that suggest that any effect size
smaller than this would involve negligible and educationally-unimportant differences.

Table 3 Mean, standard deviation and pretest–posttest difference (effect size and ANOVA results) for
engineering and technology items
Item Item mean Item SD Differences [ 0.2
SDs

Pretest Posttest Pretest Posttest d F

Engineering items
Design circuits 1.47 1.44 0.49 0.49
Make better food 1.68 1.84 0.46 0.37 0.38 1.24**
Design machines 1.66 1.57 0.47 0.49
Works out ways to make tablets easier to 1.02 1.09 0.15 0.29 0.30 56.40**
take
Work as electrician 1.72 1.67 0.45 0.47
Build houses 1.77 1.95 0.42 0.21 0.54 200.51**
Drive machines 1.19 1.37 0.39 0.48 0.41 121.66**
Repair machines 1.05 1.67 0.22 0.46 1.72 1568.16**
Technology items
TV 1.99 1.97 0.09 0.16
Running shoes 1.98 1.96 0.09 0.21
Tea cup 1.95 1.96 0.21 0.19
Manufacturing plant 1.12 1.81 0.32 0.39 1.93 2397.08**
Spaceship 1.81 1.90 0.39 0.29 0.26 66.26**
Lightning 1.65 1.79 0.47 0.40 0.32 90.25**

Cohen’s d effect size = Difference between means divided by pooled SD

**p \ 0.01, N = 1095

123
Learning Environ Res (2018) 21:285–300 295

Effect sizes for the entries in Table 3 ranged between 0.30 and 1.72 standard deviations for
the engineering test items and between 0.26 and 1.93 standard deviations for the tech-
nology test items. For the engineering items, statistically significant pretest–posttest
changes occurred for the five items ‘make better food’, ‘work out ways to make tablets
easier to take’, ‘build houses’, ‘drive machines’ and ‘repair machines’. For the technology
test, changes between pretest and posttest were significant for ‘manufacturing plant’,
‘spaceship’ and ‘lightning’.

Effectiveness of E&T activities in terms of learning environment, attitudes

and understanding

Table 4 provides some information about the effectiveness of the E&T activities in terms
of pretest–posttest changes in scores on each of the learning environment, attitude and
understanding scales. Changes between pretest and posttest are shown in Table 4 in terms
of their statistical significance (using MANOVA with repeated measures with the 6
learning environment, attitude and understanding scales as the dependent variables) and
their magnitude (using Cohen’s d to express the difference in standard deviation units).
When MANOVA revealed that overall pretest–posttest differences for the set of dependent
variables were statistically significant using Wilks’ lambda criterion, the univariate
ANOVA was interpreted for each individual learning environment, attitude and under-
standing scale, as shown in Table 4. Because Cohen’s (1992) criteria suggest that any
pretest–posttest differences less than 0.2 standard deviations can be considered small and
of little educational importance, only effect sizes larger than 0.2 standard deviations are
recorded in Table 4.
The results in Table 4 show that there was a large change between pretest and posttest
of 0.70 standard deviations in student Career Interest in E&T during the intervention. This
change was statistically significant. The negligible changes between pretest and posttest for
the two learning environment scales (Cooperation and Involvement) and for Enjoyment of

Table 4 Mean, standard deviation and pretest–posttest difference (effect size and ANOVA results) for
learning environment, attitude and understanding scales
Scale Mean SD Differences [ 0.2 SDs

Pretest Posttest Pretest Posttest d F

Learning environment
Cooperation 2.67 2.66 0.31 0.39
Involvement 2.61 2.56 0.37 0.41
Attitudes
Enjoyment of lessons 2.43 2.38 0.57 0.68
Career interest 1.85 2.24 0.53 0.58 0.70 495.06**
Understanding
Engineering 1.44 1.57 0.14 0.18 0.81 510.31**
Technology 1.66 1.76 0.13 0.14 0.74 414.94**

Cohen’s d effect size = Difference between means divided by pooled SD

**p \ 0.01, N = 1095

123
296 Learning Environ Res (2018) 21:285–300

Lessons are considered small and therefore of minor educational importance. A possible
explanation for the very small pretest–posttest changes for Cooperation, Involvement and
Enjoyment is the presence of a ‘ceiling effect’ in that pretest means for these scales already
were relatively close to the maximum possible scale score of 3.0 (see Table 4). Also,
although improvements in learning environment and Enjoyment did not occur, still it is
noteworthy that the large increase in Career Interest was achieved without any deleterious
effect on the two classroom environment variables and Enjoyment.
Table 4 also shows statistically-significant pretest–posttest differences for both under-
standing scales. The magnitude of pretest–posttest improvements were 0.81 standard
deviations for Understanding of Engineering and 0.74 standard deviations for Under-
standing of Technology. Overall, the results in Table 4 support the effectiveness of the
E&T materials on several criteria.

Differential effectiveness for male and female students

We also were interested in investigating whether the improvements between pretest and
posttest experienced by the whole sample (Table 4) were similar or different for male and
female students (i.e. whether the E&T materials might be differentially effective for stu-
dents of different sexes). A one way MANCOVA was used to compare males’ and
females’ posttest scores on each of the six dependent variables while controlling for
corresponding pretest scores. Because the multivariate test revealed a significant difference
overall for the set of dependent variables (using Wilks’ lambda criterion), the univariate
ANCOVA was interpreted for each individual dependent variable (see Table 5). The
ANCOVA results in Table 5 reveal a significant effect only for the two scales of Coop-
eration and Understanding of Engineering.
In order to interpret the significant findings in Table 5, mean scale scores are shown in
Table 5 separately for males and females and separately for pretest and posttest. These
means suggest that males benefitted a little more than females from the E&T materials in

Table 5 ANCOVA results for

Scale Sex Mean ANCOVA
differential effectiveness of E&T
materials for males and females Pretest Posttest F

Learning environment
Cooperation Males 2.64 2.70 26.26**
Females 2.69 2.62
Involvement Males 2.56 2.56 2.81
Females 2.63 2.56
Attitudes
Enjoyment of lessons Males 2.42 2.41 2.69
Females 2.43 2.36
Career interests Males 1.98 3.26 1.67
Females 1.73 3.14
Understanding
Engineering Males 1.42 1.53 44.42**
Females 1.46 1.61
Technology Males 1.67 1.76 0.02
*p \ 0.01, N = 560 (females) Females 1.65 1.75
and 510 (males)

123
Learning Environ Res (2018) 21:285–300 297

terms of Cooperation scores, but females benefitted a little more than males in terms of
engineering understanding scores.

Significance and conclusion

Our study is significant in that it employed learning environment criteria in evaluating

unique instructional materials that integrate engineering and technology ideas for primary-
school students. This involved the development and validation of a widely-applicable
questionnaire that can be used by other researchers and teachers for assessing young
children’s perceptions of learning environment and attitudes in integrated E&T classrooms.
Overall, our study supported the effectiveness of the E&T materials in terms of large
pretest–posttest changes in career interest in E&T (effect size of 0.70 standard deviations)
and understanding of engineering and technology (effect sizes of 0.81 and 0.74 standard
deviations, respectively).
Our study also identified some gaps in primary students’ understanding of E&T that are
likely to serve as deterrents to students taking up E&T careers. Students generally had a
reasonable understanding of what technology is but limited understanding about what
engineers do. This could be because the processes and the knowledge related to education
in engineering and technology are also different. Technology is more a component of
general education and is taught at the primary and lower-secondary levels (Williams 2001).
Engineering studies, being narrower and related to a specific profession, are more voca-
tional and traditionally have not been a part of the primary school curriculum. But our
study also demonstrated that, through the implementation of carefully-designed E&T
curriculum materials, primary-school students can experience large improvements in their
understanding of E&T and in their interest in E&T careers.
In the future, it is recommended that teachers (as well as students) are exposed to
carefully-designed core practices of engineering through design challenges that are spread
over different fields of engineering and incorporate technology. The efforts in our study
involving the implementations of E&T activities with students need to be supplemented by
providing professional development for teachers and by incorporating the topic of STEM
education into teacher education programs. Sustainable courseware can only exist with
teacher support.

References
ACARA (Australian Curriculum, Assessment and Reporting Authority). (2011). The Australian Curriculum.
www.acara.edu.au.
ACER (Australian Council for Educational Research). (2016a, December 6). Latest PISA results: Australia
at the cross-road. [Media release]. Retrieved from: https://www.acer.edu.au/about-us/media/media-
releases/latest-pisa-results-australia-at-the-crossroad.
ACER (Australian Council for Educational Research). (2016b). TIMSS 2015: A first look at Australia’s
results. Retrieved from: https://www.research.acer.edu.au.
Afari, E., Aldridge, J. M., Fraser, B. J., & Khine, M. S. (2013). Students’ perceptions of the learning
environment and attitudes in game-based mathematics classrooms. Learning Environments Research,
16, 131–150.
Aldridge, J. M., & Fraser, B. J. (2008). Outcomes-focused learning environments. Rotterdam: Sense
Publishers.
Aldridge, J. M., Fraser, B. J., & Huang, I. (1999). Investigating classroom environments in Taiwan and
Australia with multiple research methods. Journal of Educational Research, 93, 48–62.

123
298 Learning Environ Res (2018) 21:285–300

Aldridge, J. M., Fraser, B. J., & Sebela, M. P. (2004). Using teacher action research to promote con-
structivist learning environments in South Africa. South African Journal of Education, 24(4), 245–253.
Australian National Engineering Taskforce. (2010). Scoping our future: Addressing Australia’s engineering
skills shortage. www.anet.org.au.
Bundy, A. (1999). 21st century literacy…Still no single measure. Incite, 20(5), 8. Retrieved from: http://
archive.alia.org.au/incite/1999/05/literacy.html.
Burden, R. L., & Fraser, B. J. (1993). Use of classroom environment assessments in school psychology: A
British perspective. Psychology in the Schools, 30(3), 232–240.
Bybee, R. W. (2013). The case for STEM education: Challenges and opportunities. Arlington, VA: National
Science Teachers Association.
Byrne, D. B., Hattie, J. A., & Fraser, B. J. (1986). Student perceptions of preferred classroom learning
environment. The Journal of Educational Research, 80(1), 10–18.
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159.
Cohn, S. T., & Fraser, B. J. (2016). Effectiveness of student response systems in terms of learning envi-
ronment, attitudes and achievement. Learning Environments Research, 19, 153–167.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.
Early Childhood STEM Working Group. (2017). Early STEM matters: Providing high-quality experiences
for all young learners. Retrieved from: http://ecstem.uchicago.edu.
Fraser, B. J. (1981a). Using environmental assessments to make better classrooms. Journal of Curriculum
Studies, 13(2), 131–144.
Fraser, B. J. (1981b). Test of science-related attitude (TOSRA). Melbourne: Australian Council for Edu-
cational Research.
Fraser, B. J. (2012). Classroom learning environments: Retrospect, context and prospect. In B. J. Fraser, K.
G. Tobin, & C. J. McRobbie (Eds.), Second international handbook of science education (pp.
1191–1234). New York: Springer.
Fraser, B. J. (2014). Classroom learning environments: Historical and contemporary perspectives. In N.
G. Lederman & S. K. Abell (Eds.), Handbook of research on science education (II ed., pp. 104–119).
New York: Routledge.
Fraser, B. J., Aldridge, J. M., & Adolphe, F. S. G. (2010). A cross-national study of secondary science
classroom environments in Australia and Indonesia. Research in Science Education, 40, 551–571.
Fraser, B. J., & Butts, W. L. (1982). Relationship between perceived levels of classroom individualization
and science-related attitudes. Journal of Research in Science Teaching, 19(2), 143–154.
Fraser, B. J., & Fisher, D. L. (1983). Development and validation of short forms of some instruments
measuring student perceptions of actual and preferred classroom learning environment. Science
Education, 67(1), 115–131.
Fraser, B. J., Giddings, G. J., & McRobbie, C. J. (1995). Evolution and validation of a personal form of an
instrument for assessing science laboratory classroom environments. Journal of Research in Science
Teaching, 32, 399–422.
Fraser, B. J., & Tobin, K. G. (Eds.). (1987). Exemplary practice in science and mathematics education.
Perth: Key Centre for Teaching and Research in School Science and Mathematics, Curtin University.
Furtak, E. M., Seidel, T., Iverson, H., & Briggs, D. C. (2012). Experimental and quasi-experimental studies
of inquiry-based science teaching: A meta-analysis. Review of Educational Research, 82(3), 300–329.
Gee, J. P. (2012). Social linguistics and literacies: Ideology in discourses (4th ed.). Abingdon: Routledge.
Goh, S. C., & Fraser, B. J. (2000). Teacher interpersonal behaviour and elementary students’ outcomes.
Journal of Research in Childhood Education, 14(2), 216–231.
Goodrum, D., & Rennie, L. J. (2007). Australian school science education national action plan 2008–2012.
Retrieved from http://www.dest.gov.au/NR/rdonlyres/94684C4C-7997-4970-ACAC-5E46F87118D3/
18317/Volume1final_28August2008.pdf.
Goss, P., Sonnemann, J., Chisholm, C., & Nelson, L. (2016). Widening gaps: What NAPLAN tells us about
student progress (Grattan Institute Report No. 2016-3). Retrieved from: http://grattan.edu.au/wp-
content/uploads/2016/03/937-Widening-gaps.pdf.
Harris, K. L., & (for the Australian Council of Deans of Science). (2012). A background in science: What
science means for Australian society. Melbourne: Centre for the Study of Higher Education.
Kennedy, T. J., & Odell, M. R. I. (2014). Engaging students in S.T.E.M. education. Science Education
International, 25(3), 246–258. Retrieved from: http://files.eric.ed.gov/fulltext/EJ1044508.pdf.
Khine, M. S. (Ed.). (2015). Attitude measurements in science education: Classic and contemporary
approaches. Charlotte, NC: Information Age Publishing.
Kind, P. M., Jones, K., & Barmby, P. (2007). Developing attitudes towards science measures. International
Journal of Science Education, 29, 871–893.

123
Learning Environ Res (2018) 21:285–300 299

Koul, R., Fraser, B. J., Maynard, N., Tade, M., & Henderson, D. (2016). Science, technology, engineering
and mathematics (STEM) teaching to primary-school students. In R. V. Nata (Ed.), Progress in
education (pp. 97–119). New York: Nova Science Publishers.
Lachapelle, C. P., & Cunningham, C. M. (2007, March). Engineering is elementary: Children’s changing
understanding of science and engineering. Paper presented at the 114th American Society for engi-
neering education annual conference and exposition, Honolulu, HI.
Laugksch, R. C. (2000). Scientific literacy: A conceptual overview. Science Education, 84(1), 71–94.
Lightburn, M. E., & Fraser, B. J. (2007). Classroom environment and student outcomes among students
using anthropometry activities in high-school science. Research in Science and Technological Edu-
cation, 25, 153–166.
Liu, L., & Fraser, B. J. (2013). Development and validation of an English classroom learning environment
inventory and its application in China. In M. S. Khine (Ed.), Application of structural equation
modeling in educational research (pp. 75–89). Rotterdam: Sense Publishers.
Masny, D., & Cole, D. (2007). Applying multiple literacies in Australian and Canadian contexts. In A.
Simpson (Ed.), Future directions in literacy: International conversations conference proceedings (pp.
190–211). Sydney: Sydney University Press.
Milliken, D., & Adams, J. (2010). Recommendations for science, technology, engineering and mathematics
education (Report by STEM Work Group). Retrieved from: http://www.k12.wa.us/publications.
Murcia, K. (2009). Science in the news: An evaluation of students’ scientific literacy. Teaching Science,
55(3). Retrieved from: http://ro.ecu.edu.au/ecuworks/577/.
Naisbitt, J., & Aburdene, P. (1990). Megatrends 2000. London: Sidwick & Jackson.
Norton, B. (2007). Critical literacy and international development. In L. Mario & T. M. de Souza (Eds.),
Critical literacy: Theories and practices (Vol. 1, pp. 6–15). Nottingham: Centre for the Study of Social
and Global Justice, University of Nottingham.
Office of Chief Scientist. (2016). Australia’s STEM workforce. Canberra: Australian Government.
Office of the Chief Scientist. (November 2014). Benchmarking Australian science, technology, engineering
and mathematics. www.chiefscientist.gov.au.
Ogbuehi, P. I., & Fraser, B. J. (2007). Learning environment, attitudes and conceptual development asso-
ciated with innovative strategies in middle-school mathematics. Learning Environments Research, 10,
101–114.
Riegle-Crumb, C., King, B., Grodsky, E., & Muller, C. (2012). The more things change, the more they stay
the same? Prior achievement fails to explain gender inequality in entry to STEM college majors over
time. American Educational Research Journal, 49, 1048–1073.
Schibeci, R. (2011). Productive partnerships: Advancing STEM education in Western Australian schools (A
report to the Science Education Committee of Western Australian Technology & Industry Advisory
Council, TIAC). Perth: Murdoch University.
Sirrakos, G., Jr., & Fraser, B. J. (2017). A cross-national mixed-method study of reality pedagogy. Learning
Environments Research, 20, 153–174.
Spinner, H., & Fraser, B. J. (2005). Evaluation of an innovative mathematics program in terms of classroom
environment, student attitudes and conceptual development. International Journal of Science and
Mathematics Education, 3(2), 267–293.
Taylor, P. C., Fraser, B. J., & Fisher, D. L. (1997). Monitoring constructivist classroom learning environ-
ments. International Journal of Educational Research, 27, 293–302.
Teh, G. P. L., & Fraser, B. J. (1995). Development and validation of an instrument for assessing the
psychosocial environment of computer-assisted learning classrooms. Journal of Educational Com-
puting Research, 12(2), 177–193.
Treagust, D. F., Duit, R., & Fraser, B. J. (1996). Overview: Research on students’ preinstructional con-
ceptions—The driving force for improving teaching and learning in science and mathematics. In D.
F. Treagust, R. Duit, & B. J. Fraser (Eds.), improving teaching and learning in science and mathe-
matics (pp. 1–14). New York: Teachers College Press.
Tytler, R., & Osborne, J. (2012). Student attitudes and aspirations towards science. In B. J. Fraser, K.
G. Tobin, & C. J. McRobbie (Eds.), Second international handbook of science education (pp.
597–625). New York: Springer.
van Langen, A., & Dekkers, H. (2005). Cross-national differences in participation in tertiary science,
technology, engineering and mathematics education. Comparative Education, 41(3), 329–350.
Walker, S. L. (2006). Development and validation of the test of geography-related attitudes (ToGRA).
Journal of Geography, 105, 175–181.
Williams, P. J. (2001). The teaching and learning of technology in Australian primary and secondary
schools (Department of Education, Science and Technology Working Report). Canberra: Common-
wealth of Australia.

123
300 Learning Environ Res (2018) 21:285–300

Wolf, S. J., & Fraser, B. J. (2008). Learning environment, attitudes and achievement among middle-school
science students using inquiry-based laboratory activities. Research in Science Education, 38,
321–341.
Wong, A. F. L., & Fraser, B. J. (1996). Environment–attitude associations in the chemistry laboratory
classroom. Research in Science and Technological Education, 14, 91–102.
World Economic Forum. (2017). Realizing human potential in the fourth industrial revolution: An agenda
for leaders to shape the future of education, gender and work. Switzerland. http://www3.weforum.org/
docs/WEF_EGW_Whitepaper.pdf.
Zaragoza, J. M., & Fraser, B. J. (2017). Field-study science classrooms as positive and enjoyable learning
environments. Learning Environments Research, 20, 1–20.
Zollman, A. (2012). Learning for STEM literacy: STEM literacy for learning. School Science and Math-
ematics, 112(1), 12–19.

123