Engaging Students in 21st Century Skills through Non-Formal Learning

Lisa A. Moyer

Dissertation submitted to the faculty of Virginia Polytechnic

Institute and State University in partial fulfillment of the
requirements for the degree of

Doctor of Philosophy
In
Curriculum and Instruction

John G. Wells, Committee Chair

Jeremy Ernst
Brett Jones
Kelly Parkes

March 10, 2016

Blacksburg, Virginia

Keywords: Non-formal Education, Informal Education, Experiential Learning, Outdoor

Learning, 21st Century Skills, Essential Skills, Integrative STEM Education, Orienteering
 Engaging Students in 21st Century Skills through Non-Formal Learning

Lisa A. Moyer

ABSTRACT
National reforms, such as the Next Generation Science Standards (NGSS), Common
Core State Standards Mathematical Practices (CCSMP), and Partnership for 21st Century
Learning (P-21) challenge educators to provide students with dynamic learning experiences that
address the needs of learners in today’s society. These new standards represent a paradigm shift
away from the meticulous content memorization of many state standards, toward more dynamic
measures addressing the whole learner. To truly develop the leaders, innovators and thinkers of
tomorrow, educators are beginning to look beyond the traditional schoolhouse walls to intertwine
intentionally designed non-formal learning experiences within formal education. These non-
formal experiences serve to connect seemingly disparate skills and knowledge through real-life,
hands-on, minds-on learning. Embracing partnerships with individuals and organizations beyond
the classroom fosters an environment seamlessly connecting life, work, and school.
Although the importance of student engagement in 21st century skills is at the forefront of
current educational reforms, little has been done to assess this engagement. While standards such
as Common Core State Standards and NGSS have measures in place for domain-specific 21st
century skills, aside from PISA’s cross-curricular problem solving test, there are few resources to
measure non-domain specific engagement in these skills. Without a viable measure, detractors
can argue that the term 21st century skills is meaningless and it distracts students from learning
core content. Bridging the divide between skills and content is essential to build support for skills
that reach far beyond isolated subject-matter knowledge. Engaging students in these skills
through non-formal learning, and measuring the extent of student engagement in these skills will
drive the development of future opportunities for students to hone them in creative ways.
The purpose of this study was to measure student engagement in 21st century skills while
they participate in a non-formal learning experience. Once a viable measurement was developed,
it was utilized to measure student percent of engagement in each specific 21st century Learning
and Innovation skill (creativity and innovation, critical thinking, problem solving), Life and
Career skill (flexibility and adaptability, initiative, self-direction and productivity, leadership,
responsibility and accountability), and Socio-Cultural skill (communication and collaboration)
while students participated in the intentionally designed non-formal learning experience of
orienteering. The study also described what characterizes a viable non-formal learning
experience facilitating student engagement in 21st century skills.
Analysis of data revealed the non-formal learning experience of orienteering engages
students in 21st century Learning and Innovation Skills, Life and Career Skills and Socio-
Cultural Skills. Specifically, communication and collaboration, critical thinking skills and
initiative, self-direction and productivity comprise the largest student engagement. Engagement
in leadership, responsibility and accountability, problem solving, and flexibility and adaptability
are also evident. This particular non-formal learning experience facilitates very little student
engagement of creativity and innovation. While not generalizable to a larger population, this
study confirms that students immersed in a non-formal learning activity will become engaged in
essential 21st century skills for school, life and work, therefore, this type of learning is a valuable
part of instructional time within the formal instructional day and beyond.
 Dedication

I dedicate this work to my two daughters, who inspire me with their independence,

curiosity, and amazing life perspective. They graciously endured my absences from childhood

weekends, evening events and soccer games throughout my career as a teacher and as a student.

They learned to figure things out for themselves while I spent long evenings studying, grading,

and writing papers. I am constantly amazed at the thoughtful and independent women they have

become, and I look forward to being a part of their future journeys.

iii
 Acknowledgements

I was so fortunate to have a dynamic and caring committee who supported my work and

helped me sift through countless ideas, research and data to finally arrive at the point of

completion. I especially want to thank my advisor, Dr. John G. Wells for not only spending

hours mentoring my work, but for pushing me to develop the grit and perseverance I always look

for in my students. Through downed trees, bad Chinese food, and all other manner of challenges,

you stuck with me and helped me to persist. I am grateful to Dr. Brett Jones, who’s thoughtful

support and wealth of knowledge helped me to navigate through many difficult questions. I am

so thankful to Dr. Jeremy Ernst, who always made practical sense of things when they became

too convoluted. Special thanks to Dr. Kelly Parkes, who’s time I stole at the 11th hour to guide

my qualitative data, and who always found the time to be a part of my committee, no matter

where in the world she was.

A very special thank you to my colleagues, Michelle and Anita, who graciously gave

countless hours of their busy schedules to help me code and arbitrate piles of data, as well as the

amazing middle school teacher who stood in the freezing wind to make the data collection

possible. Special thanks to Tim, who kept our business running while I dropped off the map to

work, and who helped me keep some semblance of sanity throughout this long process.

I could not have accomplished this without the love and support of Emily and Kendall,

who graciously endured countless hours of an absent mom throughout my journey as a student. I

am grateful to David, who’s patience and support helped get me through multiple degrees and

too many classes to count. And finally, many thanks to my parents for their acceptance and

patience throughout my years as a perpetual student. I can finally say this goal was

accomplished, and mom- you’re still here!

iv
 Table of Contents

Abstract ................................................................................................................................ i
Dedication ........................................................................................................................... ii
Acknowledgements ............................................................................................................ iii
Table of Contents ............................................................................................................... iv
List of Tables .................................................................................................................... vii

CHAPTER I: INTRODUCTION

Nature of the Problem ..............................................................................................1

Educational Paradigm Shift .....................................................................................2
Next Generation Science Standards ..............................................................2
Common Core State Standards….…………………………………...…….3
Partnership for 21st Century Learning………………...…..………….........5
Assessment...................................................................................................6
Rationale of the Study..............................................................................................8
Problem Statement ...................................................................................................9
Purpose of the Study ................................................................................................9
Research Questions ................................................................................................10
Limitations of the Study..................................... Error! Bookmark not defined.10
Operational Definitions ..........................................................................................11

CHAPTER II: LITERATURE REVIEW

Overview ................................................................................................................14
21st Century Skills ..................................................................................................14
A Pedagogical Framework for Non-formal Education ..........................................15
Philisophical Underpinnings of Non-formal Education........................................16
Constructivism .......................................................................................... 16
Constructing the Whole Learner ................................................................17
Theoretical Framework..........................................................................................18
Integration of Domain Specific Knowledge, Skills, and Attitudes ........... 19
Situated Learning ...................................................................................... 20
Collaborative Construction of Knowledge. .............................................. 21
The Role of Teacher and Learner ............................................................. 22
Problem Solving Knowledge, Skills, and Attitudes.................................. 23
Continuous Formative and Authentic Assessment. .................................. 24
Reflection and Articulation . ..................................................................... 24
Problem Solving, Metacognition, and Transferability. ............................. 25
A Pedagogical Framework for Orienteering ..........................................................26
Necessary Teacher Pedagogical Knowledge..............................................28
Conducting an Orienteering Problem Solving Experience.................................... 31
Creating a Course........................................................................................31
Student Preparation.................................................................................... 31
Strategy.......................................................................................................32
And They're Off!.........................................................................................33
v
 Debriefing...................................................................................................34
Summary.....................................................................................................35
CHAPTER III: METHOD
Research Questions ................................................................................................36
Research Design.....................................................................................................37
Participants……………………………………………………………………….39
Procedure…………………………………………………………………………40
Intervention. .............................................................................................. 40
Pre-Orienteering. .............................................................................40
Course Construction........................................................................40
Data Collection ...........................................................................................42
Video Capture .................................................................................42
Post Interview………….…..……………………………………...42
Data Analysis……………………………………………………………………..43
Quantitative………………………………………………………………43
Descriptive Statistics……………………………………………..43
Qualitative……………………………………………………………….44
Summary………………………………………………………………………….44

CHAPTER IV: FINDINGS

Main Study .............................................................................................................47
Research Question .........................................................................................47
Data Sources ..................................................................................................48
Pilot Study......................................................................................................48
Data Analysis ........................................................................................................51
Research Question 1 .....................................................................................51
Learning and innovation and life and career skills ...............................52
Socio-cultural skills ..............................................................................54
Reasearch Question 2 ....................................................................................54
Protocol ................................................................................................55
Summary of Findings ............................................................................................62

CHAPTER V: CONCLUSIONS, IMPLICATIONS, AND RECOMMENDATIONS

Research Question 1 .......................................................................................65
Conclusion: Learning and Innovation Skills.........................................65
Conclusion: Life and Career Skills .......................................................66
Conclusion: Socio-Cultural Skills.........................................................66
Reasearch Question 2 .....................................................................................66
Conclusion: Characteristics of Viable Non-Formal Learning
Experiences ...........................................................................................67
Summary of Conclusions .....................................................................................67
Implications..........................................................................................................68
Recommendations ................................................................................................68
vi
 Recommendations for Researchers ...............................................................68
Recommendations for Practitioners ..............................................................69

REFERENCES.................................................................................................................70
APPENDICES ................................................................................................................. 83
Appendix A IRB Approval Letter (Pilot Study) ....................................................83
Appendix B IRB Approval Letter (Actual Study) .................................................86
Appendix C P21 Skills Coding Scheme (Original) ...............................................89
Appendix D Revised Coding Scheme for Panelists' Review…………….………94
Appendix E P21 Framework for 21st Century Learning…………………………98
Appendix F Panelists Comments on Coding Scheme…………………………..102
Appendix G Coding Scheme 1 and 2 (First Iteration with Panelists).………….105
Appendix H NGSS, 21st Century Skills, CCSS Math Practices Cross-
Referenced………………………………………………………...109
Appendix I Orienteering Maps……………………………………………….…111
Appendix J Orienteering Score Card...………………………………………….113
Appendix K Control Markers, Punch and Keychain…………………………....115
Appendix L Class Visit Lesson ………………………………………………...117
Appendix M Pre-Course Briefing………………………………………………119
Appendix N Orienteering Post-Interview Protocol……… ……………………121
Appendix O Finalized Coding….……………………………………………….124
Appendix P Coded Pilot Transcripts………..…………………………………. 128
Appendix Q Pilot Stud……………………………………………………….…134
Appendix R Initial Post-Experience Interview Notes..…………………………143
Appendix S Student Utterances by Theme ..................................................……173

vii
 List of Tables

Table 1: Alignment Between Research Questions, Data Sources, and Analysis

Procedure ............................................................................................................ 39
Table 2: Weber’s Eight-Step Coding Protocol (1990) ...................................................... 49
Table 3: Student Percentage of Engagement in Learning & Innovation and Socio-Cultural
Skills ................................................................................................................... 52
Table 4: Student Percentage Engagement in Socio-Cultural Skills .................................. 54
Table 5: Characterization of Interviews ............................................................................ 55
Table 6: Theme Analysis Summary of Viable Non-Formal Learning Characteristics ..... 57

viii
 CHAPTER 1: INTRODUCTION

“The map is not a substitute for personal experience. The map does not take the place of the
actual journey” (Dewey, 1956, p. 20).
Overview

“To successfully face rigorous higher education coursework, career challenges and a

globally competitive workforce, U.S. schools must align classroom environments with real world

environments…” (P21 Mission section, n.d., para 2). This requires not only hands-on, problem-

based simulation activities, but actual experiences beyond the classroom walls. Today’s ideal

learning environments are rich in creative thinking, problem solving and critical thinking

opportunities. Integrative Science, Technology, Engineering and Mathematics (I-STEM)

Education provides the framework to facilitate this shift, ushering in an era of integrative,

problem based and design based experiences at the forefront of today’s “classroom.”

There is a profound disconnect between the knowledge and skills most students learn in

school and those needed for real life. Gone are the days when compartmentalized knowledge and

fragmented learning are deemed sufficient for our students. Essential skills for success in today’s

world are critical thinking, problem solving, communication, collaboration, creativity and

innovation (P21, 2011). To prepare for work and life, students must learn to think for themselves

while in the field, solve complex, often multi-faceted problems, and be able to communicate

their decisions effectively. These multi-layered, unscripted problems that arise within authentic

contexts are inherently integrative in nature (Humphreys, 2005). Participation in authentic,

integrative problem solving experiences builds necessary life and career skills, such as flexibility

and adaptability, initiative, self-direction, leadership and responsibility (P21, 2011). Educators

must bridge the gap between knowing and doing, no longer treating knowledge as independent

of the situation in which it is learned and used (Brown, Collins & Duguid, 1989).

1
 Educational Paradigm Shift

National reforms, such as the Next Generation Science Standards (NGSS), Common

Core State Standards Mathematical Practices (CCSMP), and Partnership for 21st Century

Learning Framework (P-21) challenge educators to provide students with dynamic learning

experiences that address the needs of learners in today’s society. These new standards represent a

paradigm shift away from the meticulous content memorization of many state standards under

No Child Left Behind, toward more dynamic measures addressing the whole learner.

Next Generation Science Standards

The Next Generation Science Standards (NGSS) have introduced a conceptual shift in the

vision for American science education. According to NGSS, K-12 standards emphasize “the

interconnected nature of science as practiced and experienced in the real world” (Appendix A,

p.1, 2013). Based on the Framework for K-12 Science Education developed by the National

Research Council, NGSS stresses students’ active engagement in scientific and engineering

practices. Crosscutting concepts deepen student understanding of core ideas. According to

NGSS, these science and engineering practices and crosscutting concepts are designed “to be

taught in context-not in a vacuum” (National Research Council (NRC), 2012, para 2). NGSS

emphasizes that scientific and engineering practices must not be treated as an afterthought.

Combining practice with content provides context for learning, instead of practices alone

(activities) and content alone (memorization). This integration sets the stage for meaningful

learning experiences. The focus is not simply on teachers covering content, but on students

developing deeper understanding and application of content.

NGSS (2013) core ideas are designed to “provide a key tool for understanding or

investigating more complex ideas and solving problems” as well as “relate to the interests and

2
life experiences of students.” (NGSS Introduction, p.3). NGSS acknowledges that with increased

frequency, scientists are working in interdisciplinary teams that “blur traditional boundaries”

(NGSS Introduction, 2013, p.3). Students must be provided experiences that encourage

collaborative critical thinking and problem solving across disciplines.

Furthermore, NGSS acknowledge the importance of the affective domain in the

development of concepts and skills in science and engineering. According to the Framework for

K-12 Science Education “Research suggests that personal interest, experience, and enthusiasm—

critical to children’s learning of science at school or in other settings— may also be linked to

later educational and career choices” (NRC, 2012, p. 28). Preparation for college, careers and

citizenship is core to NGSS as well. Standards are intentionally designed to prepare students to

innovate, lead and create jobs of the future.

Common Core State Standards

Another important feature of NGSS is the alignment with Common Core State Standards

(CCSS) in English Language Arts and Mathematics (NGSS, Appendix A, 2013) and Partnership

for 21st Century Learning (P21) skills. All three sets of standards overlap, providing a continuity

of learning and the ability for educators to integrate subjects within meaningful, real-world

learning experiences. This overlap is depicted in Appendix D.

The Standards for Mathematical Practice focus on “processes and proficiencies”

important in mathematics education (Common Core State Standards for Mathematics, 2010, p.6).

The process standards consist of problem solving, reasoning and proof, communication,

representation, and connections. The proficiencies include adaptive reasoning, strategic

competence, conceptual understanding, procedural fluency, and productive disposition.

Productive disposition specifies that students should be inclined to perceive mathematics as

3
sensible, useful, and worthwhile. It also emphasizes a belief in diligence, and one’s own self-

efficacy (CCSS, 2010). The CCSS processes and proficiencies embolden educators to develop

expertise in their students far beyond traditional mathematical computation and memorization.

Developed with the goal of ensuring that every student is prepared to succeed in college,

career, and life, the CCSS specifically address 21st century needs (2010). For example,

CCSS.Math.Practice.MP1, Make sense of problems and persevere in solving them, discusses

student analysis of givens, constraints, relationships and goals. While this standard is applicable

to multiple types of mathematical problems, the skills described extend far beyond any single

school subject. This is further reinforced with CCSS.Math.Practice.MP3, Construct viable

arguments and critique the reasoning of others. This standard encourages students to analyze

situations, justify conclusions, reason inductively, communicate their reasoning to others, and

respond effectively to the arguments of others (CCSS, 2010). CCSS.Math.Practice.MP5, Use

appropriate tools strategically, and CCSS.Math.Practice.MP6, Attend to precision, also discuss

skills applicable far beyond the scope of basic mathematics. They discuss the proficiency of

students considering appropriate tools, visualizing the results of varying assumptions, exploring

consequences, and communicating their reasoning to others precisely (CCSS, 2010). These

standards were informed by the best existing state standards, the expertise of teachers, content

specialists, states, leading thinkers, and public feedback. They further illustrate the shift in

educational thinking, away from isolated content area competencies, toward a more holistic

development of students prepared for today’s work and life.

Partnership for 21st Century Learning

Founded in 2002 as a coalition to bring together the business community, educational

leaders, and policymakers, the Partnership for 21st Century Learning (P21) serves as a catalyst to

4
position 21st century readiness in the center of educational policy and reform. With such

founding organizations as The U.S. Department of Education, AOL Time Warner Foundation,

Apple, Inc., National Education Association, and many more, P21 has grown into a force for

educational reform, bridging business, education, and communities with a mission to foster the

knowledge and skills all learners need to be life-long learners in a constantly changing world

(P21 Vision and Mission, 2015). The basic tenets of P21 are the belief that learning takes place

throughout life, and in many, varied places and spaces. A broad range of experiences is

necessary for learners to thrive in today’s world. All learners deserve 21st century learning

opportunities, and “a strong foundation for success is rooted in learning that happens in and out

of school” (P21 Vision and Mission, 2015). Preparing learners for the challenges of work, life,

and citizenship in the 21st century and beyond requires 21st century learning environments and

opportunities. Learners who possess these skills are essential for the innovations that drive our

economy and the health of our democracy (P21 Vision and Mission, 2015).

P21 developed the Framework for 21st Century Learning to help practitioners integrate

21st century skills. It describes the knowledge, skills, and expertise necessary for success in work

and life today. P21 divides these elements into “21st century student outcomes” that are “critical

systems necessary to ensure 21st century readiness for every student” (P21 Framework for 21st

Century Learning, 2015, p. 1). These elements are: Content Knowledge and 21st Century

Themes, Learning and Innovation Skills, Information, Media, and Technology Skills, and Life

and Career Skills (P21 Framework for 21st Century Learning, 2015, p.2). While content

knowledge is still a key element, P21 describes Learning and Innovation Skills such as creativity,

critical thinking, problem solving, and collaboration as equally important. Life and Career Skills

such as flexibility and adaptability, and initiative and self-direction are also essential. According

5
to P21, “Today’s life and work environments require far more than thinking skills and content

knowledge” (P21 Framework for 21st Century Learning, 2015, p.2).

Assessment

Global and national assessment agencies are recognizing that today’s complex and

diverse society calls for a fundamental change in how we educate and assess our students. The

Program for International Assessment (PISA) is a global assessment that measures 15 year-old

students’ reading, mathematics and science literacy. In 2012, PISA added a cross-curricular

problem solving test to its repertoire. While American students scored slightly above average on

exams assessing their problem solving skills, students who took the problem solving tests in

other countries, including Singapore, South Korea, Japan, several provinces of China, Canada,

Australia, Finland and Britain, all outperformed American students. The problem solving results

revealed that students in the highest performing nations were also able to think flexibly. Even on

interactive tasks, the American students’ strength, all the Asian countries that participated

outperformed the United States (OECD, 2012). “To understand how to navigate a complex

problem and exercise abstract reasoning is actually a very strong point for the Asian countries,”

said Francesco Avvisati, an analyst on the PISA team at the Organization for Economic

Cooperation and Development (Motoko, 2014, p.A17). The fact that PISA now identifies

problem solving as a discrete, measurable and essential competency illustrates a shift in the focus

of the educational needs of our students today, from mere knowledge acquisition to the synthesis

and analysis of information and flexible thinking necessary to solve problems, innovate and

create.

Recognizing PISA’s shift toward a better prepared future generation, the National

Assessment of Educational Progress (NAEP) moved toward broadening its assessments to

6
include college and career readiness and “21st Century Skills” such as communication and

collaboration, problem solving, and persistence after failure which “cut across subject domains”

(NAEP, 2012, p.6). In 2009, NAEP introduced a new science framework providing the

theoretical basis for student assessment. The assessment measures content knowledge, but also

goes beyond the basics, with hands-on components and interactive computer tasks measuring

students’ knowing and doing (procedural, schematic and strategic knowledge). Assessment of

this content employs cross-cutting concepts among Physical Science, Life Science and Earth and

Space Science. Science practices assessed take into account their cognitive complexity.

Technological design questions are included, requiring students to apply their knowledge and

skill to solve problems in real world context (NAEP, 2012).

The National Center for Education Statistics (NCES), which administers NAEP is

committed to seeing these changes in its new procurement cycle for 2017. Among the changes is

a new Technology and Engineering Literacy (TEL) Assessment being given to students in grade

8. This framework emphasizes student’s problem solving process. Lonnie Smith, an assessment

specialist at the Educational Testing Service explained “What we’re looking at is not, are you

able to solve the problem, but how did you go about solving the problem…we’re trying to assess

not whether students can arrive at a solution, which in my opinion is not telling of their abilities,

but what is interesting is how do they get there?” (Heitin, 2014, p. 7). This metacognitive

approach is consistent with the hands-on, active, intentional teaching of problem solving and the

transferability of this skill. If students are to perform well on these assessments, they must be

given multiple opportunities to integrate their knowledge into larger constructs and utilize

problem solving skills to apply this knowledge to real problems in the field.

7
 Rationale for the Study

Student success in college, career, and life depends upon much more than learning

information, subjects and skills in isolation. To truly develop the leaders, innovators and thinkers

of tomorrow, educators are beginning to look beyond the traditional schoolhouse walls to

intertwine intentionally designed non-formal learning experiences within formal education.

These non-formal experiences serve to connect seemingly disparate skills and knowledge

through real-life, hands-on, minds-on learning. Embracing partnerships with individuals and

organizations beyond the classroom fosters an environment seamlessly connecting life, work,

and school.

When asked to rank skills in terms of importance, employers put professionalism,

teamwork, and oral communication at the top of their list (Are They Really Ready to Work,

2006). While knowing how to problem solve, think critically, communicate effectively and

collaborate with flexibility and adaptability are not unique to the 21st century, today’s citizens

must be prepared with these skills, as modern work and life requires their daily use for such tasks

as analysis of information, decision making and collaborating to create new ideas (Silva, 2009).

21st century skills are critical to today’s success, yet students are rarely engaged in them while

sitting in a traditional classroom.

Even when students are engaged in 21st century skills, these skills are rarely measured.

While there are some measures in place for use of these skills within specific subject areas, it is

difficult to find assessments for student engagement in skills while participating in cross-

curricular or non-domain specific activities. This study addresses the need to engage students in

experiences that practice 21st century skills, as well as the need to measure student engagement

in these skills during intentionally designed non-formal educational experiences.

8
 Problem Statement

Although the importance of student engagement in 21st century skills is at the forefront of

current educational reforms, little has been done to assess this engagement. While standards such

as Common Core State Standards and Next Generation Science Standards have measures in

place for domain-specific 21st century skills, aside from PISA’s cross-curricular problem solving

test, there are few resources to measure non-domain specific engagement in these skills. Without

a viable measure, detractors can argue that the term 21st century skills is meaningless and it

distracts students from learning core content. Bridging the divide between skills and content is

essential to build support for skills that reach far beyond isolated subject-matter knowledge.

Engaging students in these skills through non-formal learning, and measuring the extent of

student engagement in these skills will drive the development of future opportunities for students

to hone them in creative ways.

Purpose of Study

The purpose of this study was to measure student engagement in 21st century Learning

and Innovation skills, Life and Career skills, and Socio-Cultural skills while participating in a

non-formal learning experience. Once a viable measurement was developed, it was utilized to

measure student percent of engagement in each specific 21st century Learning and Innovation

skill (creativity and innovation, critical thinking, problem solving), Life and Career skill

(flexibility and adaptability, initiative, self-direction and productivity, leadership, responsibility

and accountability), and Socio-Cultural skill (communication and collaboration) while students

participated in the intentionally designed non-formal learning experience of orienteering.

Through these measures, educators can demonstrate the importance of non-formal learning

9
experiences for student engagement in 21st century skills. The study also characterizes a viable

non-formal learning experience facilitating student engagement in 21st century skills.

Research Questions

RQ#1: To what extent are students engaged in the following 21st century skills while

participating in a non-formal learning experience: Learning and Innovation Skills

(creativity and innovation, critical thinking, and problem solving), Life and Career Skills

(flexibility and adaptability, initiative, self-direction and productivity, and leadership,

responsibility and accountability) and Socio-Cultural Skills (communication and

collaboration)?

RQ#2: What characterizes a viable non-formal learning experience facilitating student

engagement in 21st century skills?

Limitations of the Study

This research investigation was a case study involving ten participants who were seventh

grade students at Blacksburg Middle School. The results of this study, therefore, cannot be

generalized across populations, but they can provide an understanding, from the students’

perspective, of their engagement in 21st century skills (Yin, 2014). Time constraints limited the

length of time students could participate in the orienteering experience. Students participated

during their science class period, which was 50 minutes long. Due to time constraints, detailed

orienteering instruction was provided the day before the experience, with a short review on the

day of the experience. Participants gained permission to stay after class for the debriefing

interview. It is difficult to determine if this had any effect on the results, but if the students had

block scheduling, they would have participated in all three parts of the experience seamlessly

within one block.

10
 Operational Definitions

21st Century Skills: a broad set of knowledge, skills, work habits, and character traits that

are believed—by educators, school reformers, college professors, employers, and others—to be

critically important to success in today’s world, particularly in collegiate programs and

contemporary careers and workplaces. 21st century skills can be applied in all academic subject

areas, and in all educational, career, and civic settings throughout a student’s life (Abbott, 2014,

para. 1).

Behavioral engagement: Student involvement and participation in academic, social or

extracurricular activities. Behavioral engagement is crucial for achieving positive academic

outcomes and preventing dropping out. (Connell & Wellborn,1991; Finn,1989).

Cognitive Engagement: the psychological level of investment and the effort a student

directs toward learning. It includes being thoughtful, strategic, and willing to exert the necessary

effort for comprehension of complex ideas or mastery of difficult skills (Corno & Mandinach,

1983; Fredricks, Blumenfield & Paris, 2004; Newmann, Wehlarge, & Lamborn, 1992).

Creativity and Innovation: Individuals who demonstrate this use a wide range of creation

techniques, create new and worthwhile ideas (both incremental and radical concepts),

demonstrate originality and inventiveness in work and understand the real world limits to

adopting new ideas, create new and worthwhile ideas (both incremental and radical concepts),

and act on creative ideas to make a tangible and useful contribution to the field in which the

innovation will occur (P21 Framework Definitions, 2015).

Critical Thinking: Individuals who demonstrate this use various types of reasoning

(inductive, deductive, etc.) as appropriate to the situation, effectively analyze and evaluate

evidence, arguments, claims and beliefs, analyze how parts of a whole interact with each other to

11
produce overall outcomes in a complex system (systems thinking), reflect critically on learning

experiences and processes, analyze and evaluate major alternative points of view, synthesize and

make connections between information and arguments , and interpret information and draw

conclusions based on the best analysis (P21 Framework Definitions, 2015).

Flexibility and Adaptability: Individuals who demonstrate these skills adapt to varied

roles, jobs responsibilities, schedules and contexts, work effectively in a climate of ambiguity

and changing priorities, incorporate feedback effectively, reflect critically on past experiences in

order to inform future progress, deal positively with praise, setbacks and criticism, exercise

flexibility and willingness to be helpful in making necessary compromises to accomplish a

common goal, and view failure as an opportunity to learn; understand that creativity and

innovation is a long-term, cyclical process of small success and frequent mistakes (P21

Framework Definitions, 2015).

Initiative, Self-Direction, and Productivity: Individuals who demonstrate these skills set

goals with tangible and intangible success criteria, balance tactical (short-term) and strategic

(long-term) goals, go beyond basic mastery of skills and/or curriculum to explore and expand

one’s own learning and opportunities to gain expertise, set and meet goals, even in the face of

obstacles and competing pressures, prioritize, plan and manage work to achieve intended results,

monitor, define, prioritize and complete tasks without direct oversight, multi-task, and manage

time and projects effectively (P21 Framework Definitions, 2015).

Leadership, Responsibility, and Accountability: Individuals who demonstrate these skills

use interpersonal and problem-solving skills to influence and guide others toward a goal,

leverage strengths of others to accomplish a common goal, inspire others to reach their very best

via example and selflessness, demonstrate integrity and ethical behavior in using influence and

12
power, act responsibly with the interests of the larger community in mind, and are accountable

for results (P21 Framework Definitions, 2015).

Learning:

Formal education: the hierarchically structured, chronologically graded education

system, running from primary school through the university and including, in addition to general

academic studies, a variety of specialized programs and institutions for full-time technical and

professional training (Coombs, Prosser, & Ahmed,1973).

Informal education: the truly lifelong process whereby every individual acquires

attitudes, values, skills and knowledge from daily experience and the educative influences and

resources in his or her environment - from family and neighbors, from work and play, from the

market place, the library and the mass media (Coombs, Prosser, & Ahmed,1973).

Non-formal education: any organized educational activity outside or beyond the

established formal system - whether operating separately or as an important feature within some

broader activity - that is intended to serve identifiable learners and learning objectives (Coombs,

Prosser, & Ahmed,1973).

Learning and Innovation Skills: Essential skills that are increasingly being recognized as

those that separate students who are prepared for the more complex life and work environments

of the 21st century from those who are not. These skills include creativity, innovativeness, critical

thinking, problem solving, communication and collaboration (P21 Framework Definitions,

2015).

Life and Career Skills: Essential skills that require more than thinking skills and content

knowledge. These skills allow one to navigate the complex life and work environment of the

globally competitive information age. They include flexibility and adaptability, initiative and

13
self-direction, social and cross-cultural skills, productivity and accountability, and leadership and

responsibility (P21 Framework Definitions, 2015).

Problem Solving: Individuals who demonstrate this solve different kinds of non-familiar

problems in both conventional and innovative ways, and identify and ask significant questions

that clarify various points of view and lead to better solutions (P21 Framework Definitions,

2015).

CHAPTER II: LITERATURE REVIEW

The separation between knowing and doing becomes an issue in the formal learning

environment. (Resnick, 1987). Principles, concepts and facts are often taught in an abstract and

decontextualized form. In this atmosphere, knowledge itself becomes seen as the final product

of education rather than a tool for solving problems. Non-formal education, by contrast, can be

seen as education that occurs outside of this formalized, decontextualized setting, or simply put,

learning that goes on in daily life. Bell, in conjunction with the National Research Council

(NRC), discussed non-formal education in science as “rich with real-world phenomena…where

people can pursue and develop science interests, engage in science inquiry, and reflect on their

experiences through sense-making conversations” (2009). Non-formal education is the real-

world, social, hands-on learning that is sometimes forgotten within the confines of formal

education.

21st Century Skills

According to the Glossary of Education Reform, 21st century skills are a “broad set of

knowledge, skills, work habits and character traits” (Abbott, 2015). Educators, school reformers,

college professors, employers and others agree that these skills are critical for success in today’s

world, and they can be applied to all academic areas. They are sometimes referred to by such

14
related terms as applied skills, cross-curricular skills, cross-disciplinary skills, interdisciplinary

skills, transferable skills, transversal skills and others, but while some of these names may be

fairly synonymous, they also have other, more specialized meanings. The concept of 21st century

skills is fostered by the belief that our schools and other institutions of learning should prioritize

the most useful, relevant and in-demand skills for today’s world (2015).

In his article Rigor Redefined, Wagner states “It's time to hold ourselves and all of our

students to a new and higher standard of rigor, defined according to 21st-century criteria” (2008,

p.25). He asserts that out of the myriad of classes he has observed in recent years, less than 1 in

20 engaged students in instruction deliberately designed to teach them to think rather than merely

drilling for a test. Wagner touts the following “seven survival skills” (p.20) for students to

become productive citizens dealing with 21st century issues: problem solving and critical

thinking, collaboration and leadership, agility and adaptability, initiative and entrepreneurialism,

effective oral and written communication, access and analyzing information, and curiosity and

imagination (2008).

While some educators and school reformers refer to these skills as new, they have in fact

been a part of human progress throughout history. According to Rotherham and Willingham

(2009), what is actually new is the extent to which the changes in our world and our economy

mean that success (collectively and individually) is dependent upon these skills. While there are

many students who are being taught these skills, Rotherham and Willingham assert that up to this

point in education it has been a matter of chance rather than a deliberate design of our school

systems (2009). This deliberate design is difficult, however, as the body of knowledge and skills

known as 21st century skills is not simple to define and has not been officially codified or

categorized (Abbott, 2015). Assessment challenges will require deliberate attention from today’s

15
policy makers and a deviation from current ways of thinking about educational assessment

(Rotherham and Willingham, 2009).

A Pedagogical Framework for Non-Formal Education

A clear definition of specifically what non-formal education is can be elusive, as it is usually

approached as a residual category- everything that is not formal education. This, however does a

great disservice to this educational approach, as there are many rich and complex modes of

learning involved in non-formal education. While formal education may fulfil learning needs

prescribed by government, employment, or the education system, non-formal education can

serve to fulfil broader goals as well as people’s own life purposes, and can take place in forms

chosen by the learner (Cairns, Merrifield & McGivney, 2000).

Since non-formal education is often described as what formal education is not, it is useful to

clarify basic tenets of what would be described as “formal” education. In general, formal

education:

 Has a prescribed learning framework

 Is an organized learning event or package
 Has a designated teacher or trainer
 Is offered for credit or a qualification
 Has specified outcomes (Eraut, 2000, p. 12)

Non-formal learning occurs across all of the different domains of our lives, with transfer and

integration between the domains of work, home and community (Cairns et al., 2000).

Philosophical Underpinnings of Non-Formal Education

Constructivism

Any discussion of non-formal education must move beyond the simple focus on

context or setting and look at the processes and experiences involved. Non-formal learning is

inherently an active process, where the learner makes choices and acts upon them, constructing

16
meaning from experiences as they unfold. Drawing on the ideas of cognitive scientists such as

John Dewey, Jean Piaget and Lev Vygotsky, constructivism is a learning theory about the nature

of reality and how people understand the world around them. According to Dewey, “every

experience lives on in further experiences” (1938, p.16). Constructivism asserts that people make

(or construct) their own knowledge based on their experiences. Ideas do not have an exact true or

false, but their viability is variable depending upon their context. Solutions to problems are

dependent upon the situation and the factors involved in that situation (Callison, 2001).

Constructivist teaching is a type of “discovery learning,” where the learners are actively

involved in problem solving, as opposed to behaviorist teaching, where the learner is a passive

reactor to the teacher (Callison, 2001). Non-formal education embraces the student-driven,

discovery and idea-generated nature of constructivism. Learners bring previous experiences and

expectations to the problem, and consider ideas relative to the situation. They construct meaning

as they work through the problem. Fosnot and Perry define constructivism as a psychological

learning theory that describes learning as “an interpretive, recursive, non-linear building process

by active learners interacting with their surround- the physical and social world” (1996, p.23).

These constructivist characteristics are woven throughout non-formal education. Educators bear

the responsibility for facilitating these experiences, allowing students to build knowledge and

expand their problem solving skills.

Constructing the Whole Learner

20th Century cognitive scientist Jerome Bruner described people as constantly seeking to

balance their internal representation of the outside world through their observations (1960).

Bruner’s inquiry-based, constructivist learning theory asserts that learning occurs in problem

solving situations. Students become problem solvers by interacting with their environment,

17
testing hypotheses and developing generalizations. Bruner believed the science curriculum

should foster the development of problem solving skills through discovery and inquiry (Bruner,

1960). He outlined the five-stage process of perception, selection, inference, prediction and

action. Bruner recognized that learning is an information-seeking process that involves thoughts,

actions and feelings (Kuhlthau, 2004).

Lev Vygotsky (1978) declared that social learning is essential for driving cognitive

development. His Zone of Proximal Development (ZPD) consists of tasks that lie between those

a learner can accomplish without assistance, and those which require assistance. Optimal

learning occurs when a learner is challenged, and the learning involves facilitated social

construction of knowledge (Vygotsky, 1978). Bruner also discusses this level of uncertainty,

believing that student’s desire to learn and problem solve is fostered through problem activities

in which the learner could explore alternative solutions (Bruner, 1966).

Robert Gagne’s work with intellectual skills established prerequisite knowledge; essential

knowledge one must possess to master subsequent higher order learning. This prerequisite

knowledge is crucial for problem solving (1962). Albert Bandura’s theory of self-efficacy

describes our belief in our ability to perform a task. Self-efficacy is developed over time through

the successes a learner experiences when interacting with his/her environment. Through the

process of self-regulation, a learner sets goals for him/herself, monitors performance toward

those goals, judges the adequacy of that performance, and makes modifications (Bandura, 1971).

These pivotal learning theories espouse the cognitive, affective and social aspects of learning

inherent in problem solving through non-formal education.

Theoretical Framework

18
 Although there are multiple, and sometimes conflicting definitions of non-formal

education, my research revealed seven basic commonalities that foster meaningful learning and

form the underlying principles of the non-formal learning process. Students build usable

knowledge through learning environments that:

 Foster integration of skills and knowledge to solve problems

 Provide meaningful activities within the context of an authentic learning environment
 Support collaborative construction of knowledge
 Promote the role of the teacher as facilitator
 Embolden the student to be an active, autonomous, self-regulated decision maker
 Provide continuous, formative and authentic assessment within tasks
 Promote reflection and articulation to foster the formation of abstractions and enable
tacit knowledge to become explicit.

Non-Formal Learning

collaborative construction
of knowledge

continuous Domain specific

reflection & knowledge, skills &
articulation attitudes

Metacognition Active
Integration
Learner
continuous formative general problem
assessment solving knowledge,
skills & attitudes

facilitator
coaching &
scaffolding

Authentic Learning Environment

Figure 1: Non-Formal Learning

19
Integration of Domain Specific Knowledge, Skills, and Attitudes

Integrative learning is vital for preparing students to deal effectively with the complex

issues they will face in work and everyday life. Integrative thinkers can see connections in

information that seems disparate. They draw on an extensive range of knowledge to make

decisions, and they can adapt what they learn from one situation to problems in other situations

(Greater Expectations National Panel, 2002). According to Nussbaum (1997), students prepared

for twenty first century life must be able to work through complex interdependencies,

synthesizing from many, varied sources. They should learn from experience and be able to make

productive connections between theory and practice. Integrative teaching approaches must foster

students’ capacity to begin making connections for themselves. Both fully integrated learning

and learning with a purpose are fundamental elements of the non-formal learning process

(Griffin, 1998).

Situated Learning

Learning is a complex and continuous life-long process. It is not only situated within a

series of contexts, it results from acting within situations (Brown, et al, 1989, Falk & Dierkin,

2000). Simply put, learning is a contextually driven effort to make meaning. It involves a

continuous exchange between the learner and his/her physical and sociocultural environment.

(Falk & Storksdeick, 2005).

Coherent, meaningful, and purposeful learning activities are authentic. Classroom

activities that take place within the culture of schools are not the same as the activities or the

culture of actual practitioners (Brown, et al., 1989). The formal definitions, clear-cut, well

defined problems, and symbol manipulation within the school culture do not allow learners to

develop the general strategies for intuitive reasoning, negotiating meaning and resolving issues

20
that are experienced in everyday life and work (Lave, 1988). This culture limits student’s access

to the important structuring and supporting cues that are developed from context. Classroom

tasks often fail to provide the contextual features that allow authentic activity. As a result,

instead of the desired goal of in-school learning, success within the formal school culture often

has little effect on a learner’s performance elsewhere (Brown, et al., 1989).

Due to the nature of classroom tasks, transfer of knowledge within the confines of the

school culture is extremely difficult. When a learner experiences concepts in multiple contexts,

he/she learns different ways the knowledge can be used. From these various experiences, the

learner begins to generalize his/her knowledge. This knowledge can then be utilized in multiple

ways and within various contexts. This is opposed to the abstract way knowledge is often taught

within the structure of school. All too often, school’s abstract structure encourages students to

employ substandard strategies like storing information just long enough to retrieve it for a test

(Collins, 1989). It is never truly internalized or placed within a usable context, and it is unlikely

to be used to solve problems outside of the constructs of school. Learning through multiple

contexts, both within and outside of the general school culture encourages students to generalize

across domains, and apply their knowledge in unique and flexible ways throughout various

situations. Learning that is meaningful for solving real world problems is complex and situated

within a series of contexts.

Collaborative Construction of Knowledge

Learning comes from experience, but both learning and experience are socially

constructed and reconstructed. (Cairns et al., 2000). Learning is defined by the cultural contexts

in which it occurs. Through participation in various cultures, people develop behaviors, beliefs

and practice that make meaning of their world. All of these are established as social practices

21
within a given context. The process of learning from experience shapes the learner’s

development. Since learning is a social process, the learner’s development is molded by his/her

cultural system of social knowledge (Kolb, 1984). Because of the socio-culturally situated nature

of learning, learners are powerfully influenced by the interactions and collaborations they have

with learners from their own social group (Falk & Storksdieck, 2005).

In their multimedia program study, Herrington and Oliver (2000), discussed the benefit

students found when articulating, reflecting and scaffolding with a partner. They discovered that

if not given a partner with whom to work, students will covertly seek opportunities to

collaborate. According to Vygotsky, people develop individual cognition as a result of their

social interaction (1978). Resnick, in her 1987 Presidential Address pointed out that much of

formal school learning and measurement of performance is individual, but most of our activity

outside of school is socially shared. Real life takes place within social systems, and a learner’s

ability to function successfully depends upon working within this social system (Resnick, 1987).

Sharing of knowledge is necessary to solve the real, complex, ill-structured problems we face

every day.

The Role of Teacher and Learner

Learning is optimized when students have choices and feel they are in control of their

own learning (Falk & Dierking, 2000). Self-directed learning involves students choosing what

they will learn based on need, anticipated use, or interest. It also involves self-determination of

pacing both physically (how quickly a student moves through a physical process) and mentally

(when to slow down and focus on details and when to superficially take in the whole picture).

Griffin (1998), in her baseline study found that most teachers on a field trip or in a non-formal

learning situation had a task oriented approach. When their strategies moved from teacher-

22
centered to learner-centered, the outcome was self-directed learning. Griffin determined that that

the key to successful learning experiences is not just teacher strategies, but the full set of self-

directed learning conditions which are provided throughout the non-formal learning experience.

One of her most significant findings was the extent and level of learning achieved by students

when given freedom to learn through self-direction. When exploration is encouraged, learners

are able to try different hypotheses, methods and strategies to see what happens. This puts

learners in control of problem solving. The facilitator is there to guide the learners and show

them how to explore productively. Through autonomous, facilitator-guided exploration, students

learn how to problem-find and set achievable goals (Griffin, 1988).

Learners thrive in environments that acknowledge their needs and experiences.

Students should be asked questions whose answers need to be figured out, not just recited from

memorized facts. “They need to be treated like sense-makers rather than rememberers and

forgeters” (Lampert, 1986, p.340). It is imperative that students make connections between what

they are supposed to be learning and the things that they care about understanding. Students who

are given the freedom to think for themselves, figure things out, and make decisions based on

their learning develop the capacity to gather information, organize it strategically, generate and

test hypothesis, and produce as well as evaluate solutions. They become comfortable discussing

what they are thinking, they develop the ability to listen, appreciate, and build from others’ ways

of understanding, and they learn to invent problem solving procedures that are sensible and

useful (Lampert, 1986).

Problem Solving Knowledge, Skills, and Attitudes

Problem finding has been identified as the most critical skill for success (Getzels, 1979),

yet it is rarely explicitly taught or even practiced in the formal school environment. Learners

23
need practice in setting reasonable goals and revising their goals as they progress deeper into a

problem. Schools emphasize well defined tasks instead, unlike any problems in the real world.

Practicing how to form and test hypotheses within the context of real problems is pivotal for

making sense of the world and solving problems. Learners who practice this within many

different domains and with the guidance of a facilitator will not only learn to effectively navigate

real-world problems, they will start making discoveries on their own. Learners will get a feel for

what it is like to be a scientist while they make and test hypotheses. They develop a sense of

enjoyment and accomplishment from creating their own ideas and seeing if they work (Collins,

1989).

Continuous Formative and Authentic Assessment

Learning is an emergent process, and therefore cannot be assessed strictly through a set

of expected outcomes. Ideas are not fixed and incontrovertible elements of thought. They are

formed and reformed through experience. Every piece of understanding is the result of the

process of continuous construction and invention. The interaction process of assimilation and

accommodation allows this to occur (Kolb, 1984). This is why continuous, formative and

authentic assessment is so important.

An authentic assessment gives students a task that is worthwhile, significant, and

meaningful (Hart, 1994). It confronts student with the same types of activities that are carried out

in professional practices. When students are given a meaningful learning task, they must utilize

higher-order thinking skills and integrate a broad range of knowledge to build competencies

instead of memorizing information and basic skills. Instead of a passive listener, the student is an

active participant, building skills throughout the task. The assessment is continuous and

24
formative, giving students the responsibility for their own learning, reflecting, collaborating, and

engaging in continuous dialogue with the teacher (Gulikers, Bastiaens, & Kirschner, 2004).

Reflection and Articulation to Foster the Formation of Abstractions and Enable Tacit

Knowledge to Become Explicit.

Learning only becomes generalizable to new situations when the student can recognize

elements of the context in which the original learning occurred within that new context (Perkins

& Salomon, 1989). Therefore, transfer needs to be facilitated. When knowledge is tacit, it can

only be used in contexts that elicit the knowledge because the contexts are very similar to the

conditions in which it was acquired. When learners must articulate their knowledge, it

generalizes the knowledge from a particular context so it can be used in other circumstances.

This allows knowledge to be more available and utilized in other tasks. Articulated knowledge

becomes part of a set of interconnected ideas and becomes more easily available.

When learners articulate strategies, they can begin to realize how their strategies apply in

different contexts. Insight into alternative perspectives is achieved when learners discuss their

strategies in groups. Not only do learners hear various perspectives, their own articulation

promotes insight into alternative perspectives. When a learner attempts to explain an idea or a

problem, he/she begins to understand it from different perspectives. Responses from other

learners allow insight into difficulties that others may have, and the way different people view

the same issue (Collins, 1989).

Problem Solving, Metacognition and Transferability

The ability to think about one’s problem solving activities can mean the difference

between being a poor problem solver and an effective one (Gardner, 1991). Metacognition

(awareness of one’s own cognitive processes) allows students to identify and work strategically

25
within the parts of the problem solving process. Metacognitive skills foster students’ ability to

strategically encode the nature of the problem and form a mental representation of its elements.

Students may then choose appropriate plans and strategies for a solution, and identify and

overcome obstacles that hamper progress (Davidson & Sternberg, 1998).

Metacognition includes the planning, monitoring and evaluation of the problem solving

process, and improving skill in these areas can increase performance in a wide variety of

situations. When students must give reasons for their decisions, and verbalize the steps they took

while problem solving, it focuses their attention on their own solution process. Use of intentional

metacognition will not only improve current performance, it will transfer to other problems for

improved performance in multiple tasks (Dominowski & Dallob, 1995). Students’ verbalization

of their problem solving practice facilitates the discovery of the general process of solving

problems and it assists in allowing them to better utilize these skills in future problem situations

(Gagne & Smith, 1962). Knowledge about their own mental processes and the process of

problem solving itself helps students become better problem solvers in school and work

situations, as well as everyday life.

A Pedagogical Framework for Orienteering

According to Ericsson, Krampe, and Tesch-Romer (1993), expert performance is developed

through deliberate practice. Orienteering provides an optimal environment for fostering 21st

century skills. Students gain practice in ill-structured problem solving, developing declarative,

procedural, schematic and strategic knowledge. Orienteering encourages flexibility and

motivates students to persist in difficult situations. It encourages positive collaboration and

teamwork, leadership skills, initiative and self-direction, and it brings tacit knowledge to light

through facilitated articulation. As orienteering is a form of non-formal learning, constructivism

26
offers a solid foundation for the development of essential 21st century skills within the

experiential, outdoor learning context of orienteering.

Orienteering is situated learning, immersing students in the outdoor environment, while

challenging them to think strategically and problem solve with peers. In an experiential learning

orienteering experience, participants are given a detailed map with control points marked

throughout the course. Students must plan the best route to all control points in the least amount

of time. The experience challenges students both mentally and physically to navigate new and

diverse terrain, problem solve, think strategically, work collaboratively, and make decisions in

the field. Throughout the experience, students monitor their progress, often changing their course

of action multiple times. Original plans become less than ideal when terrain is more difficult than

anticipated, obstacles prevent progress, or time dictates a necessity to change the route. Students

must constantly regroup, analyze their strategies, and devise varied solutions. They test their

solutions immediately in the field, and must re-evaluate and problem solve again if the solution

fails to produce the desired outcome. The facilitator scaffolds the students as needed to assist in

providing a zone of proximal development (Vygotsky, 1978) that is neither too obvious nor

frustrating to a point where students are unable to make meaning of their situation. This

deliberate practice in strategic planning, problem solving and decision making serves to develop

the 21st century skills necessary for success not only within the rigor of changing future

assessments, but in higher education and everyday work and life.

The pedagogical framework for orienteering specifically is reflective of the basic tenets of

non-formal education in general. Students work in collaborative teams to create strategies for

locating control points and navigating diverse terrain. The process is iterative, with students

moving between specific domain knowledge, skills and attitudes and general problem solving

27
knowledge, skills and attitudes. Students continuously collaborate to reflect upon their actions,

articulate their progress and assess strategies. Through this process of self-assessment,

metacognition and articulation, they make their tacit knowledge explicit. The facilitator scaffolds

as needed, filling in knowledge if a task lies beyond the scope of the learners. The facilitator also

poses thought-provoking questions, guiding student’s problem solving process. A facilitated

debriefing at the end of the experience provokes additional meaning-making, continuing the

cycle of collaborative construction of knowledge and enminding students into a community of

practice. The following model illustrates this process:

Figure 2: Orienteering Learning Process

28
Necessary Teacher Pedagogical Knowledge

Requisite pedagogical knowledge that teachers must possess in order to engage students in

21st century skills through orienteering experiences follows the heuristics of constructivist

teaching. First and foremost, teachers must remember that the experience is student-centered.

The teacher is not the purveyor of knowledge, and the purpose of this experience is for students

to learn to problem solve and make meaning for themselves. This requires that the teacher create

a non-threatening environment that encourages flexible and creative thinking. Students must feel

comfortable to innovate and encouraged to express their cognitive and metacognitive processing,

including frustrations, throughout the experience. This approach includes recognizing prior

learning, attitudes, perceptions, and expectations (Griffin 1998).

Teachers must have specific knowledge of 21st Century Learning and Innovation Skills and

Life and Career Skills. They must recognize the importance of developing students’ creativity

and innovation, problem solving and critical thinking, and communication and collaboration. It is

paramount that they encourage student flexibility and adaptability, initiative and self-direction,

productivity and accountability, and leadership and responsibility. Teachers must recognize the

importance of the social aspects of solving problems as a team. They must understand that

success in today’s life and work environment requires students to possess more than simply

thinking skills and content knowledge (P21 Framework for 21st Century Learning, 2015, p.2).

Teachers must also possess the pedagogical content knowledge to provide a short, basic map

and compass lesson prior to the experience, showing students how to orient their maps. They

must also provide basic directions for the activity, as well as informing students of the overall

goal to find as many control points as possible within the allotted time. Teachers may also set up

the course and draw a map, or utilize an existing map to plot points. This is fairly basic map and

29
compass content knowledge, and teachers who do not feel confident setting up a course can rely

on help from a local orienteering club or outdoor company who provides these services.

After this basic instruction, the teacher becomes the facilitator or coach, only scaffolding

students as needed and providing questioning to encourage innovation, collaboration and

problem solving. This type of cognitive apprenticeship facilitates learning through guided

experience, focusing on 21st century skills. This allows conceptual and factual knowledge to be

situated within the context of its use. When this occurs, conceptual knowledge can become

known through its uses in a variety of contexts. This type of coaching facilitates transfer of

knowledge and deeper understanding (Collins, Brown, & Newman, 1987). Teachers who

facilitate a meaningful orienteering experience maintain a dual focus on these expert processes

and situated learning.

Further pedagogical knowledge includes recognition of Vygotsky’s zone of proximal

development (1978). The teacher must have the ability to monitor the students’ level of

frustration, and decide when scaffolding is necessary, as well as when he/she should wait and

observe whether students can problem solve through difficult situations unassisted. This decision

will be made easier when teachers possess pedagogical knowledge in problem solving

competency, and the process of developing through the stages from novice to exert.

Assessment of student engagement in 21st century skills within a meaningful orienteering

experience requires that teachers be well-versed in practices of on-going, formative assessment.

They should also be able to ask thought-provoking questions that encourage rich discussion and

articulated metacognition through varied debriefing strategies. Drawing from this debriefing,

teachers must be able to gauge student engagement in 21st century skills.

30
 Teachers who can draw upon constructionist pedagogical knowledge and knowledge of

21st century skills can successfully conduct a meaningful orienteering experience with minimal

additional pedagogical content knowledge, or help from experts within the field of orienteering.

While it does not always happen in the formal educational setting, nurturing students’ 21st

century skills within the context of a motivating and meaningful experience should be a key

focus for all educators. In the current standardized testing culture, it is easy to lose sight of our

goal as educators to foster autonomous life-long learners who enjoy creatively and

collaboratively solving problems.

Conducting an Orienteering Problem Solving Experience

The best example to illustrate the framework so a teacher can intentionally structure a

student’s learning in an orienteering problem solving activity is to walk through the experience

step-by-step. There are important considerations for structuring the learning throughout each step

of the process.

Creating a Course

An orienteering course can be set up in a schoolyard, park, wilderness area, or even

inside a building if weather is a factor. Course difficulty depends on age level and experience.

Typical middle school students with little or no orienteering experience will benefit from a

moderately difficult course, with some control markers more easily found, while other

“challenge” markers will push the limits of the group and in some cases may not be found. As

the teacher places the control markers (similar to flags) with punches throughout the course,

he/she will plot them on a map. An existing map may be used, preferably containing

topographical information, or teachers may create their own. If this step is too time consuming or

31
difficult, teachers may seek help from experts in the field through orienteering clubs or outdoor

businesses who conduct orienteering competitions.

Student Preparation

The teacher will divide students into dyads or small groups, providing each group with a

compass, a punch card, and a copy of the map. He/she will show the students a control marker

and model how to punch their card in the numbered squares. Students can practice this on their

own punch card by punching a designated box for the example. The teacher explains that each

punch has a unique hole pattern, and this is how we know that you have visited a control marker.

The teacher then guides the students through the process of orienting the map by asking

questions such as:

a) Can you find where we are on this map?

b) Is it easier to read the map when it is turned in a different direction (upside down,
etc.)?
c) Can you match your compass north to north on the compass rose of your map?
d) Which way does this require you to turn your map?
e) Can you see any visual landmarks in real life and on your map that help you orient the
map?
f) How can orienting your map in this way help you while you are looking for control
points?

The Teacher can scaffold this activity by pointing out obvious visual landmarks and helping the

students orient their maps to them. He/she can also give more detailed instruction on compass

skills and topography.

Strategy

The teacher explains that the goal of orienteering is to accumulate as many points as

possible in the shortest amount of time. Each control marker is worth a certain number of points

depending upon level of difficulty. There is a time limit for the course, and at the end of the

activity, teams must arrive back at the base camp (starting point) within that time limit. If they

32
are late, they will be docked a significant amount of points for each minute beyond the time limit

they arrive. The teacher may emphasize that teams are not required to find every single control

marker, although they will want to try, and that teams can start with any control marker and they

can search for and find control markers in any order. The ultimate goal in an orienteering

competition is to “clean” the course (get to all of the control markers) and be the first team back

to base camp. Students can decide whether to rush off and start searching, or whether to take a

few minutes to study the map and devise a plan. The teacher can scaffold this by asking

questions such as:

a) Is there a logical order to these based on terrain, placement and/or points?

b) Is it worth taking the time to create a strategy? Why?
c) Which control markers would you skip if you ran out of time? Why?

This is where students begin to collaborate and think critically, coming up with creative and

innovative methods for gaining as many points as possible.

And They’re Off!

Students are heavily engaged in each of the 21st Century Learning and Innovation, Life

and Career, and Socio-cultural skills. This is also where it is important to keep in mind students’

frustration level, zone of proximal development (Vygotsky, 1978), and students’ problem solving

processes. Examples of scaffolding problem solving and metacognition through authentic

problems in the field are as follows:

Scenario 1

The students are in the vicinity of a control marker, but cannot seem to locate it. They are
frustrated, worried about time, and are trying to decide whether to continue the search or move
on to another control marker.

Examples of questions to scaffold problem-solving and metacognition:

a) Where do you think the control marker should be? Why?

33
 b) Take another look at your map. Can you find a visual landmark to help you orient it?
So now where do you think the control marker should be?
c) From which direction did you come? Were you looking along the way? So where
could you look now?
d) How many points is this control marker worth? Is it worth your time to take a few
more minutes to search since you know you must be near it?

Scenario 2
The students had a strategy for reaching a control marker that is worth several points, but
they find that their path is blocked by large brush and thorny brambles.

Examples of questions to scaffold problem-solving and metacognition:

a) What was your strategy for this one? It looks like you’ll need to rethink that strategy.
Any ideas?
b) Going that way to find the control marker puts you far away from the next one you
planned to find. Is it worth it to change your order or skip one of them?

Scenario 3
The students have only two control markers left, but they only have 15 minutes to arrive
back at base camp. One member of their team is extremely tired, and not sure if he can make it
much farther.

Examples of questions to scaffold problem-solving and metacognition:

a) How far away do you estimate each control marker to be? How fast have you been
traveling? Does that allow you enough time for both?
b) Does everyone in the group have to go all the way to punch the marker?
c) If you need to choose, which control marker should you go for? Why?
d) How do you feel about your point total right now? Would it be safer to head back to
base camp?
Debriefing
Facilitated debriefing is an important facet of developing students’ knowledge of their

engagement in 21st century skills through orienteering. It also encourages student reflection

about their thinking, collaboration and problem solving process (metacognition), articulation

(making tacit knowledge explicit) and comparison of strategies (promoting insight into

alternative perspectives). The teacher can facilitate these outcomes through thoughtful

questioning and open discussion. Questions that encourage productive reflection include:

a) Did your group begin with a strategy or develop one along the way? What was it?
b) Why was it helpful to have a strategy?

34
 c) Did your strategy change during the course? What changed? Why?
d) Were control markers where you originally thought they were? What did you do when
they were not? Why? Did this work? Why/why not?
e) Did all team members agree on strategies and decisions throughout the experience?
What did you do when they did not? Did it work? Why/why not?
f) What could you have done differently?
g) What do you think were your strengths? Why do you think this was something you
were particularly good at?
Which of your strategies or decisions were especially inventive or creative? Did you
come up with a way of doing something that was different than what others did?
h) What did you learn or figure out as you went?
i) What knowledge would help you do better next time you try this?
j) Tell me the process you used to problem solve when you had difficulties.
k) How could this experience help you solve a different kind of problem at home or at
school?
l) Is there anything else you want to tell me about the experience?

While the task may seem daunting at first, an orienteering experience provides a simple, fun,

yet challenging opportunity to get students active, outside and engaging in 21st century skills.

Summary

As Wells (2008) asserts, I-STEM education stresses content in real, meaningful contexts,

intentionally designed to guide students through purposeful inquiry and develop meaningful

connections. Engaging in 21st century skills through orienteering is an example of an I-STEM

experience that is not constrained by classroom content domains. As in the real world, solutions

to problems are not predictable or convergent. People must be flexible and adaptable, and solve

problems through collaboration, critical thinking, creativity, flexibility, and innovation. The

orienteering experience fosters student engagement in authentic learning, encouraging students

to recognize relationships among disciplines and effectively utilize them to solve real, complex

problems. For success in 21st century life, students need to be cognizant of these complex

interdependencies. Students must learn from experience to be able to connect theory and practice

(Humphreys, 2005; Nussbaum 1997).

35
 CHAPTER III: METHOD
Presented in this chapter is the methodological framework for this investigation which

includes the following sections: research questions, research design, pilot study, participants,

data collection procedures, data analysis (quantitative, qualitative, and mixed), and summary.

Research Questions

For success in our dynamic, global society, today’s students require far more than the

traditional education delivered within the classroom walls. It can be argued that non-formal

learning, whether conducted within school hours or beyond, is integral to the development of 21st

century skills. This authentic engagement allows students to acquire the real-world expertise

necessary to thrive in today’s ever-changing society. Guided by the following questions, this

research seeks to demonstrate the potential for authentic non-formal learning experiences to

serve as viable educational pathways for student acquisition of 21st century skills.

RQ#1: To what extent are students engaged in the following 21st century skills while

participating in a non-formal learning experience: Learning and Innovation Skills

(creativity and innovation, critical thinking, and problem solving), Life and Career Skills

(flexibility and adaptability, initiative, self-direction and productivity, and leadership,

responsibility and accountability) and Socio-Cultural Skills (communication and

collaboration)?

RQ#2: What characterizes a viable non-formal learning experience facilitating student

engagement in 21st century skills?

Research Design

Yin (2003) describes the case study as an empirical inquiry that “investigates a

contemporary phenomenon within its real-life context, especially when the boundaries between

the phenomenon and context are not clearly evident” (p. 13). Following Yin’s definition and

36
based on the literature review presented in chapter 2, it was determined that case study was the

best method to answer the types of research questions posed. It allows the researcher to study

small group behavior in a holistic, real-world setting (Yin, 2014). Because this research was

conducted with several small groups, a multiple-case sampling was the appropriate case study

approach to answer the types of research questions posed (Miles & Huberman, 1994) and a

logical fit when considering the type of information sought in this study. Specifically, the goal of

this research was not to generalize across populations, but to establish a better understanding of

the relationship between the acquisition of 21st century skills and non-formal learning. As well,

because the research was also looking to characterize what might constitute viable non-formal

learning experiences for student acquisition of 21st century skills, a multiple case sampling was

appropriate for determining such viability.

Data collected within the multiple case studies followed a fully integrated mixed method

monostrand conversion research design (Teddlie & Tashakkori, 2006). Both qualitative and

quantitative components influenced the conceptualization stage of the study (establishment of

relationships), with crossover analysis directly influencing the formulation of meta-inferences

(viability), thus necessitating this integrated mixed method monostrand conversion design.

Multiple phases were necessary for triangulating the qualitative and quantitative data from this

study (Figure 3).

37
This model represents the process of methods triangulation followed in this study. Presented in

Table 1 below are techniques employed in data collection, sources of those data, and method

used in analyzing those data.

38
Table 1.

Alignment Between Research Questions, Data Sources, and Analysis Procedures

Question Data Collection Method Data Analysis

RQ#1: To what extent are  GoPro Video Recordings Quantitative
students engaged in the  Interview Video  Frequency counts
following 21st century skills Recordings  percentage
while participating in a non- occurrences
formal learning experience?  measures of central
1. Learning and Innovation tendency
A. Creativity and
innovation Qualitative
B. Critical thinking  Content analysis
C. Problem solving (coding)
2. Life and Career  Corroborative
analysis
A. Flexibility and
adaptability
B. Initiative, self-direction
and productivity
C. Leadership,
responsibility and
accountability
3. Socio-Cultural
A. Communication
B. Collaboration

RQ#2: What characterizes a  GoPro Video Recordings  Content analysis

viable non-formal learning  Interview Video (clustered summary
experience facilitating student Recordings table)
acquisition of 21st century  Corroborative
skills? analysis

Video recordings from the orienteering experience were transcribed and analyzed

qualitatively using a coding rubric developed by the researcher (Appendix O). The coded data

were then analyzed quantitatively to produce frequency counts and frequency distribution.

Corroborative analysis was conducted on this data, along with content analysis to develop further

meaning. The post-interview was also analyzed qualitatively through conceptual and relational

content analysis, and analyzed corroboratively with both the qualitative and quantitative data

39
from the orienteering experience. A clustered summary table, based on the coding rubric, ordered

the cross-case data sets and facilitated deeper understanding of the phenomenon, thus ensuring

that the study was rich, robust, comprehensive, and well-developed (Creswell, 1998).

To establish the fidelity of the method used in conducting this research a pilot study was

first conducted (Appendix Q). The pilot study resulted in the instruments, coding scheme, and

procedures for data analysis.

Participants

The participants in this study consisted of 20 seventh grade students ages 12-13 enrolled

in a Life Science class at a mid-sized suburban middle school. The population was composed of

55% (11/20) male and 45% (9/20) female. Students in each class were all randomly assigned to

dyads. Only two dyads per class were randomly assigned to wear GoPro® cameras and

participate in post-interviews. This approach was designed to provide 10 data sets.

Procedure

After a short introduction to basic orienteering, students participated in a 35-minute

orienteering experience with 15 control markers on the school grounds. The course presented

several natural challenges, including distance, terrain, thick branches, thorns, and underbrush.

Higher point values motivated students to seek the more difficult control markers within the

allotted time provided for the orienteering experience. The 35-minute time limit and large size of

the course encouraged students to have a plan and run through the course as quickly as possible.

The following section details the experience provided for participants.

Intervention

Pre-Orienteering. Pre-experience instruction is critical for preparing participants in the

basics of orienteering such as how to read a map, what to look for, where to look, and basic

40
rules. This requires approximately 20 minutes to complete. Because the middle school schedule

allows only 57 minutes of class time, it was necessary to prepare the students in advance so they

would have an entire 57-minute period on the day of the actual orienteering experience. The

Thursday before the orienteering experience (scheduled for the following Monday), the

researcher visited each class for approximately 20 minutes to explain orienteering to the students

and to teach them how to “orient” a map (Appendix H).

Course Construction. A course location on school grounds was determined after a

meeting with the teacher and a visit to the school two weeks prior to the orienteering experience.

A map was created using Google Maps (2015) earth view of the middle school grounds and

adding fifteen red numbered circles to indicate where the control markers would be placed

(Appendix I). Point values for each control marker were indicated at the bottom of the map. A

scorecard (Appendix J) was also created consisting of numbered boxes intended for students to

hole punch (Appendix K). A hole punch was attached to each control marker. Each punch

created a unique pattern when hole-punched into the scorecard. The unique pattern in the

scorecard indicates to the facilitator that the students located that particular control marker.

The course, consisting of 15 control markers of various sizes (Appendix K), was set up

the Sunday afternoon before the Monday orienteering experience. Control markers of three

different sizes (small, medium, and large) were placed based on terrain and visibility. Control

markers in open areas were smaller, so they would be more difficult to spot from a distance,

while those hung within trees or brush were larger. This placement necessitated student use of

the map, rather than merely visual location of control markers. Each control marker was assigned

a point value based on difficulty. Ten of the control markers were assigned a value of ten points

each, three control markers that were more difficult and/or farther from the start/finish were

41
valued at twenty points each, while the two most difficult control markers were valued at thirty

points each. Students were tasked with accumulating as many points as possible within a 35-

minute time limit. Points were accumulated by finding a numbered control marker and using the

attached hole punch to punch a unique hole pattern into the box on the scorecard with the

corresponding number.

To incentivize students’ locating the lower point markers, an extra challenge was added

consisting of a keychain (Appendix K) included with one of the 10-point control markers. The

purpose of the keychain was to ensure that students did not go only for high point markers.

Students were made aware that a keychain existed, but there was no indication on the map of the

placement of the keychain. The keychain was worth 40 extra points, and the first dyad to find

this keychain could take it and return it to the facilitator at the end of the course, thus receiving

the extra 40 points. The keychain was there to encourage students to devise an overall strategy.

Data Collection

Video Capture. Prior to students beginning the orienteering activity, the teacher

projected a randomly generated list of student dyads on a screen in the front of the classroom for

the students to see when they arrived for class. On this list the dyads randomly selected for data

collection were highlighted. One student in each randomly selected dyad volunteered to wear the

GoPro® Hero4 camera on his/her head using a head strap and quick clip. The GoPro® Hero4

cameras were used to capture video and audio data during the orienteering experience, which

provided the non-formal experience data from the participant’s viewpoint. Students were briefly

reminded of the rules of the course (Appendix M). They walked outside, found their teammates,

and were given one course map (Appendix I) and one scorecard per team (Appendix J). The

teacher carried a whistle to notify students of the time limit when five minutes remained in the

42
experience and again when one minute remained. As soon as all maps and scorecards were

handed out, the 35-minute time limit for the course began. Two short blasts from the teacher’s

whistle notified students when five minutes remained, and one long whistle blast notified

students when one minute remained in the time limit.

Post-Interview. Post-interviews occurred in the field directly after the video-recorded

dyads completed the course. Student dyads were led to a nearby quiet area where the stationary

video cameras were set up, and they answered the eleven questions from the Orienteering Post-

Interview Protocol (Appendix N). This process was repeated for dyads in each of the teacher’s

five classes throughout the day, resulting in ten video-recorded interviews of two students each.

Data Analysis

This study utilized a fully integrated mixed method monostrand conversion design

(Teddlie & Tashakkori, 2006), which entails the conversion of qualitative data to quantitative

data with analysis and comparison between the qualitative and quantitative data. Multiple phases

were utilized to triangulate the qualitative and quantitative data from this study.

To answer RQ1, audio/video recordings collected throughout the orienteering

experiences and post-interviews were manually transcribed and entered verbatim into alternating

rows of a Microsoft Excel (2014) spreadsheet, utilizing the identical process as the pilot study.

Three coders conducted concurrent analysis of each orienteering experience transcript. As with

the pilot transcript, each of the coders segmented student utterances based on meaning conveyed

and coded each segment utilizing the coding rubric for 21st century skills (Appendix O). Once all

coders completed segmentation and coding of a given transcript, they met to arbitrate and assign

final codes to each segment. A final code was assigned where agreement of independently coded

segments occurred. Where agreement had not occurred, independent coders compared,

43
discussed, and justified the 21st century codes they assigned to each segment. Coders followed

this arbitration process until they reached agreement, generating final protocol data. Multiple

arbitrations resulted in a final data set that was readied for use in statistical analysis.

RQ 2 was answered through both conceptual and relational content analysis and

corroboration of the orienteering experience and interview data as compared to the seven basic

commonalities that foster meaningful learning and form the underlying principles of the non-

formal learning process discussed in the literature review. Both the quantitative (descriptive

statistics) and qualitative data (coding and other content analysis) were analyzed, seeking

relationships between the experience and the principles of a quality non-formal learning

experience, while the interviews elicited further explanation of the students’ perspectives related

to the experience and what they valued.

Quantitative Analysis

Descriptive statistics. Final protocol data were analyzed to generate descriptive

statistics, including frequency counts, percentage occurrences, and measures of central tendency.

For each coding scheme, individual codes were tallied and percentage occurrences were

calculated by dividing the total quantity of codes used for that particular coding category.

Percent occurrences were calculated as a means of accounting for protocols of different lengths.

This increases the validity of the study and facilitates comparison of differences between dyads.

Percentage occurrences were then used to calculate measures of central tendency for each code

of each dyad. Percentage frequency distribution specified the percentage of observations that

exist for each data point and grouping of data points (Shapiro, 2008).

44
Qualitative Analysis

Data from the interviews were analyzed qualitatively by interpreting and characterizing

student responses, looking for relationships. This analysis consisted of generating natural units of

meaning, classifying, categorizing and ordering these units of meaning, structuring narratives to

describe the interview contents, and interpreting the interview data (Cohen, Manion & Morrison,

2000). Specifically, qualitative analysis involved counting frequencies of occurrence for words,

themes, and pieces of data, noting patterns and themes, clustering items into categories, building

a logical chain of evidence through noting causality and making inferences, and making

conceptual/theoretical coherence- building constructs and theories to explain the phenomena

(Miles & Huberman, 1994). These data were organized in a clustered summary table. Through

methods triangulation, the data from the interviews were then corroborated with the orienteering

experience qualitative and quantitative data. Methods triangulation combined the descriptive

statistics, content analysis, and corroborative analysis of the interview and orienteering

experience data to obtain a deeper understanding and further demonstrate concurrent validity.

(Denzin, 1978; Patton, 1999).

Summary

This chapter described the purpose and design of this study as it related to the data and

analysis. It also described the study participants, materials utilized and the pilot study that

informed the purpose and data collection of the study. Data analysis, including specific statistical

analysis and triangulation of both qualitative and quantitative data from the orienteering

experience and qualitative data from the post-interviews was also included. For detailed

description of the development of a viable coding scheme for student acquisition of 21st century

45
skills, and the iterative process by which intercoder reliability was established, including average

pairwise percent agreements (APPA), see Appendix Q.

46
 CHAPTER IV: FINDINGS
This chapter presents the results of data analysis of video recordings from a middle

school orienteering experience and post-interview. Quantitative data derived from coding of

student utterances recorded during the orienteering experience addresses research question 1,

while qualitative data analysis from the post-experience interview addresses research question 2.

Mixing of these data occurred at the analysis phase using corroborative data analysis to gain a

richer understanding of what characterizes a viable non-formal learning experience.

This chapter opens with a discussion of the main study regarding data sources and

establishment of intercoder reliability, followed by a presentation of the quantitative findings of

student engagement in 21st century skills and qualitative findings of characteristics of a viable

non-formal learning experience. Additionally, statistical and corroborative analysis of research

questions is provided.

Main Study

Research Questions

The two research questions that guided this study were:

RQ#1: To what extent are students engaged in the following 21st century skills while

participating in a non-formal learning experience: Learning and Innovation Skills

(creativity and innovation, critical thinking, and problem solving), Life and Career Skills

(flexibility and adaptability, initiative, self-direction and productivity, and leadership,

responsibility and accountability) and Socio-Cultural Skills (communication and

collaboration)?

RQ#2: What characterizes a viable non-formal learning experience facilitating student

engagement in 21st century skills?

47
Data Sources

Video recordings of the orienteering experience were transcribed, coded by three

independent coders, and arbitrated to establish final codes. This process generated the descriptive

statistics necessary to address research question 1. Video recordings of the post-experience

interviews were transcribed and qualitatively analyzed through theme analysis. These data were

used to address research question 2. Corroboration of both the quantitative and qualitative data

analysis further addressed viability of non-formal learning experiences for engagement in 21st

century skills, and also resulted in implications for instructional design of non-formal learning

experiences.

Pilot Study

A pilot study was conducted to inform the main study and assist in establishing a valid

coding scheme and intercoder reliability. Video recordings of a dyad orienteering experience and

post-interview were transcribed manually with individual utterances from each dyad member

entered verbatim into alternating rows of a spreadsheet. A preliminary coding scheme (Appendix

C) was developed utilizing the Framework for 21st Century Learning Definitions of Learning &

Innovation Skills and Life & Career Skills (P21, 2015). After comparing it to the pilot study

transcript, it became obvious that this coding scheme contained copious overlap, making it

unsuitable for establishing inter-coder reliability. The coding scheme was then paired down, and

Communication & Cross-Cultural Skills was separated as a third unique category (Socio-

Cultural Skills), rather than combined within Learning & Innovation Skills and Life & Career

Skills (Appendix D). After comparing it once again to the pilot study transcripts, this coding

scheme was sent to a panel of two individuals. Criteria for panelist selection included extensive

knowledge of P21, NGSS, and Common Core Standards, previous experience teaching 21st

48
century skills, experience coding and arbitrating transcripts, and experience in non-formal

learning. The panelists were provided with the P21 Framework Definitions (P21, 2015) for

Learning & Innovation and Life & Career Skills (Appendix E), and a chart of the P21 skills

cross-referenced with Science and Engineering Practices in the NGSS and Common Core

Standards for Mathematical Practice MP1, MP2, MP3, MP6, and MP8 (Appendix H). The

original coding scheme (Appendix C), pilot study transcripts (Appendix P), and the coding

scheme developed for calibration (Appendix D) were also provided. Data were coded using

Weber’s (1990) eight-step coding protocol (Table 1).

Table 2.

Weber’s Eight-Step Coding Protocol (1990)

____________________________________________________________________________
1. Definition of coding units (e.g., word, phrase, sentence, and paragraph)
2. Definition of coding categories
3. Test of coding on a sample
4. Assessment of the accuracy and reliability of the sample coding
5. Revision of the coding rules
6. Return to step 3 repeatedly until sufficient reliability
7. Coding of all text
8. Assess the achieved reliability or accuracy
______________________________________________________________________________

The panelists read the P21 definitions and the pared down coding rubric, ensuring that each code

and definition were in sync and codes were not redundant. Panelists were asked to make

notations about the definitions and suggested modifications. (Appendix F). The panelists and

lead researcher then met and reviewed all comments, arbitrating results for each skill. Through

this process, the lead researcher developed a newly revised coding scheme (Appendix G).

The panelists then used the refined definitions with the codes to conduct concurrent

analysis of 10% of the pilot transcript. This process required panelists to simultaneously divide

student utterances until each individual segment contained a single code that reflected only one

49
of the eight 21st century skills. Once independent coders completed the segmentation and coding

with the revised coding scheme, they met to arbitrate all codes. A final code was automatically

assigned where 100% agreement of independently coded segments occurred before arbitration.

During arbitration, the independent coders compared, discussed, and justified the 21st century

codes they assigned to each segment. Segments that differed in assigned codes required coders to

engage in arbitration to dispute the assigned coding and reach agreement on the 21st century skill

addressed. Independent coders arbitrated until they reached 100% agreement on the final code,

necessitating refinement of the coding scheme in some instances.

Co-coders and the lead researcher then individually coded another 10% (approximately

25 lines) of the pilot study transcript using the newly redefined coding scheme. Again, a final

code was assigned only when there was 100% agreement among co-coders. All codes without

100% agreement were arbitrated until consensus was reached or the coding scheme was refined

to facilitate agreement. The process of coding and arbitrating 10% of the pilot study transcript

was then repeated for a third time. The researcher and co-coders again looked for discrepancies

within the coding schemes and worked to further clarify assignments of codes. An additional

30% of the pilot transcript was then coded and arbitrated. The researcher and co-coders once

again examined the coding schemes, clarified and further developed heuristics for definitions,

and then coded and arbitrated the remaining 40% of the pilot study transcript.

The measure of inter-coder reliability validates the interpretation of content and quality of

research. It requires independent coders to agree on interpretation of the content based on the

coding scheme (Cho, 2008). The percent agreement method, which computed the average

pairwise percent agreement (APPA) was used to measure the agreement between coders. To

compute this statistic, the average of cases of agreement between coders was calculated by all

50
possible codes in the final protocol. A percentage agreement above 75% was used as a

benchmark for coder consensus within the context of the study. This is deemed an acceptable

measure in social sciences (Klenke, 2008; Schloss & Smith, 1999; Stemler, 2004). Inter-coder

agreement of the last 40% of the transcript was 76% for the Socio-Cultural coding scheme and

77% for the Learning and Innovation and Life and Career coding scheme. Inter-coder agreement

for the entire pilot transcript was 78% for the Socio-Cultural coding scheme and 75% for the

Learning and Innovation and Life and Career coding scheme. The iterative process described

above resulted in the final coding scheme, heuristics, and example utterances that produced

acceptable intercoder agreements for the main study (Appendix O). Further details of this

process are included in the pilot study description (Appendix Q).

Data Analysis

Research Question 1

The following tables illustrate student engagement in each of the 21st century skills

discussed in Research Question 1. Two separate coding schemes were employed, one for

Learning and Innovation Skills and Life and Career Skills, and another for Socio-Cultural Skills.

This was necessary, as communication and collaboration were the particular Socio-Cultural

Skills examined, and all student utterances could be described as communication. The separate

communication and collaboration coding scheme allowed for coding of strictly communication

and coding of communication resulting in collaboration. Two dyads per class were video

recorded during the learning experience resulting in a total of 10 video recordings. However, due

to technical difficulties five recordings were unusable, leaving five viable video recordings that

were transcribed and coded. The five coded transcriptions with two coding schemes resulted in

data Tables 3 and 4.

51
Table 3.

Student Percentage of Engagement in Learning & Innovation Skills and Life & Career Skills

__________________________________________________________________
Learning & Innovation Skills Life & Career Skills
CI CT PS Total FA ISP LRA Total
__________________________________________________________________
Dyad 1 1% 31% 11% 43% 8% 36% 13% 57%

Dyad 2 6% 35% 8% 49% 8% 30% 13% 51%

Dyad 3 7% 22% 8% 37% 14% 38% 11% 63%

Dyad 4 4% 24% 4% 32% 9% 43% 16% 68%

Dyad 5 3% 37% 6% 46% 8% 35% 11% 54%

___________________________________________________________________
Note. CI= Creativity & Innovation; CT= Critical Thinking; PS= Problem Solving; FA=
Flexibility & Adaptability; ISP= Initiative, Self-Direction & Productivity; LRA= Leadership,
Responsibility, & Accountability

Learning and innovation skills and life and career skills. The Learning and Innovation

Skills and Life and Career Skills component of Research Question 1 asks

To what extent are students engaged in the following 21st century skills while

participating in a non-formal learning experience: Learning and Innovation Skills

(creativity and innovation, critical thinking, and problem solving), and Life and Career

Skills (flexibility and adaptability, initiative, self-direction and productivity, and

leadership, responsibility and accountability)?

The above question was analyzed by calculating the percentage of observed occurrences in

which the students were engaged in each skill during the non-formal learning experience.

Frequency of individual codes were tallied and percentage occurrences were calculated by

dividing the total quantity of codes used for that particular coding category. Percent occurrences

were calculated as a means of accounting for protocols of different lengths.

52
 Within Learning and Innovation Skills, critical thinking showed consistently and

significantly higher student engagement than creativity and innovation or problem solving.

Critical thinking (highlighted in yellow) was clearly the dominant skill across all cases, ranging

from 22% to 37%, while student engagement in problem solving skills and creativity and

innovation was by comparison fairly minimal. Problem solving skills, ranging from 4% to 11%,

showed slightly higher engagement than creativity and innovation, which ranged from 1% to 7%.

Within Life and Career Skills, student engagement in initiative, self-direction and productivity

(highlighted in blue) far exceeded engagement in leadership, responsibility and accountability or

flexibility and adaptability, with student engagement ranging from 30% to 43%. Leadership,

responsibility and accountability (11%-16%) was not as distinctly low as flexibility and

adaptability, showing a greater contribution to student engagement. Patterns among cases were

consistent, and they revealed that during the non-formal learning experience, the 21st century

skills with the largest extent of engagement were by far initiative, self-direction and productivity,

as well as critical thinking. Students were also engaged in leadership, responsibility and

accountability skills, followed by problem solving skills, and flexibility and adaptability. The

study revealed that this non-formal learning experience engaged student very little in creativity

and innovation. All data sets were fairly consistent in distribution of student engagement, with

the largest discrepancy between dyads at 15%.

53
Table 4.

Student Percentage Engagement in Socio-Cultural Skills

_____________________________________________
Socio-Cultural Skills
Communication Collaboration
(non-collaborative)
_____________________________________________

Dyad 1 59% 41%

Dyad 2 47% 53%

Dyad 3 46% 54%

Dyad 4 42% 58%

Dyad 5 47% 53%

______________________________________________

Socio-cultural skills. The Socio-Cultural component of Research Question 1 “To what

extent are students engaged in the following 21st century skills while participating in a non-

formal learning experience: communication and collaboration,” was also analyzed by calculating

the percentage of observed occurrences in which the students were engaged in each skill during

the non-formal learning experience. The data show a relatively even split between

communication and collaboration, with four out of the five dyads engaged in collaboration

slightly more than communication.

Research Question 2

Research Question 2 “What characterizes a viable non-formal learning experience

facilitating student engagement in 21st century skills?” was answered through both conceptual

and relational content analysis and corroboration of the orienteering experience and interview

data. This was compared to the seven basic commonalities that foster meaningful learning and

form the underlying principles of the non-formal learning process discussed in the literature

54
review. Directly following the non-formal learning experience, each dyad participated in a 5

to15-minute video-recorded interview. The following table characterizes the interview length

and dyads.

Table 5.

Characterization of Interviews

Dyad Time Length Transcribed Lines Gender

1 6 minutes 30 seconds 59 M/M

2 7 minutes 70 M/F

3 13 minutes 15 seconds 140 M/M

4 5 minutes 4 seconds 59 M/F

5 8 minutes 47 seconds 145 M/M

Protocol. The researcher transcribed each interview and employed the process of

progressive focusing (Parlett & Hamilton, 1976) to sift, sort, review and reflect on the data

collected. Through this process, the researcher and a panel member independently analyzed each

interview and compared all five interviews simultaneously. After extensive discussion, results

were compared and merged, producing one set of dyad interviews with highlights and notations

of meaningful units (Appendix R). Each unit, or utterance, was divided based on meaning

conveyed. Significant features within each interview emerged, as did noticeable overlaps

between interviews. Domain analysis was then conducted, grouping the utterances into clusters,

patterns and themes to form domains. Codes were analyzed and concurrently modified as themes

emerged. This process assisted the researcher in moving from description to explanation and

theory generation (Cohen, Manion & Morrison 2000). Finally, utilizing constant comparison

55
(Glaser & Strauss, 1967), the researcher compared data that were applicable to each category to

data within and across cases that were in the same category, and integrated these categories.

Only themes that appeared consistently across all five dyad interviews were retained. (Appendix

S). These themes were compared to 21st century skills, setting out a theory of viable

characteristics for a non-formal learning experience facilitating student engagement in 21st

century skills. The results of this process are presented in Table 6, with exemplar utterances

illustrative of the six characteristics of the non-formal learning experience and the 21st century

skills that emerged from all five dyad interviews.

56
Table 6.
Theme Analysis Summary of Viable Non-Formal Learning Characteristics

Theme Utterance Representative Samples 21st Century Viable Characteristics Percent of

Skill Utterances

Thinking “Uh, when we had a problem, we kinda looked at Critical thinking Foster integration of 17%
critically to solve the map closer and closer, to see like where this Problem solving skills and knowledge
problems road was like the curve and stuff. So say there's to solve problems
like, I guess all those trees and we'd look at the
curve just to know exactly where it was.” (Dyad 4,
p. 4)
“…we had to sort of accept and understand that
they are there. They're not like mislabeled. Cause
it got, sometimes it got a little frustrating cause
we'd be like ‘we're right here’” (Dyad 2, p. 3)

Making “…each time we got one we were like, alright, we Initiative, self- Embolden the student 13%
decisions, can get this one, we can get this one each time, so direction, to be an active,
scaffolding that's what we did, like help us push on and get productivity autonomous, self-
our…” (Dyad 5, p.5) regulated decision-
“So after we got the ones that, after we got the two Leadership, maker
that were down here, we cut up the field and we responsibility,
were sitting in the middle of the field talking and accountability
we were like, alright, I know we missed number
15, but I think we should go back and try to find
that because it might help us out with points since,
uh, more points the better, and we both agreed on
it so” (Dyad 5, p. 6)

57
Table 6. (cont.)
Theme Analysis Summary of Viable Non-Formal Learning Characteristics

Theme Utterance Representative Samples 21st Century Viable Characteristics Percent of

Skill Utterances

Authentic A:/ “In group work, you need teamwork, and I Collaboration Support collaborative 27%
collaboration mean the teachers can tell you to work” B:/ Communication construction of
“Together” A:/ “To work together but that's not knowledge
gonna- they can't make you and when you're” B:/
“This is fun so…” A:/ “When you're doing
something that's fun, but also challenging” B:/
“You want to work” A:/ “Yeah you want to work
together and it teaches good teamwork” (Dyad 2,
p.1-2)
“…I’d point, he’d run over there, stamp it and I’d
still try to figure out where the map was going.”
(Dyad 4, p.2)

Motivation A:/ “When you're doing something that's fun, but Initiative, self- Meaningful activities 13%
also challenging” B:/ “You want to work” (Dyad direction and within the context of
2, p.4) productivity an authentic learning
“It's like recess with hidden learning” (Dyad 2, environment
p.7)

Evaluation, “Sometimes on the way, you realize that you are Flexibility and Continuous formative 21%
flexibility not pinpoint where you think you are, so you'd adaptability and authentic
have to” A:/ “Yeah we'd be running and he'd be assessment within
like...” B:/ “So if you're running one direction and Critical thinking tasks
you thought you were running the other direction
to get one then it kinda changes of how you do
throughout the map” A:/ “It really changes” (Dyad
3, p. 1-2)

58
Table 6. (cont.)
Theme Analysis Summary of Viable Non-Formal Learning Characteristics

Theme Utterance Representative Samples 21st Century Viable Characteristics Percent of

Skill Utterances

Self-reflection, “Actually it's a little bit easier than it looks Communication Reflection and 9%
intrinsic positive because the map can be a little bit confusing but Critical articulation to foster
reinforcement once you get the hang of where the bird's eye view Thinking the formation of
building skill/ is, like how far apart things are from each other Problem abstractions and
knowledge then it gets a lot easier to move to the different Solving enable tacit knowledge
points” (Dyad 3, p.5) to become explicit

____________________________________________________________________________________________________________________

59
 Throughout the process of analyzing each dyad interview, several common themes

repeatedly emerged. Students consistently discussed how they worked as a team to make

decisions collaboratively, how they constantly evaluated and re-evaluated their decisions and

strategies based on their progress and level of success throughout the non-formal learning

activity, and the critical thinking processes they used to solve problems. Additionally, every dyad

discussed the positive nature of communication and collaboration throughout the experience,

some even highlighting how the specific non-formal learning environment fosters this effort, for

example, “When you’re doing something fun and also challenging…you want to work together,

and it teaches good teamwork.” Initiative, self-direction, and productivity were evident through

the interviews, with incremental student successes discussed as scaffolds to build the knowledge

and confidence necessary to support further problem solving within the experience. One dyad

described this scaffolding and initiative as “Each time we got one we were like, alright, we can

get this one…like help us push on…” There was no mention of any teacher or facilitator

providing answers or direct assistance in any interview. Conversely, every interview discussed

abundant student-made decisions and problem solving. Phrases in student interviews such as

“…the map can be a little bit confusing but once you get the hang of where the bird's eye view is,

like how far apart things are from each other then it gets a lot easier to move to the different

points” are illustrative of student learning independent of teacher-directed instruction.

The only 21st century skill not consistently evident in all five dyad interviews was

creativity and innovation. Two of the five dyads asserted that a strategy they employed was

especially creative or inventive, for example, “…to stay behind the tree line, because we could've

hopped on the bandwagon just kind of gone through and out the trees but we decided it was

60
getting a little scraped up, we were getting a little scraped up...” but three of the dyads could not

come up any engagement creativity or inventiveness.

As exemplified in Table 5, student descriptions of their experiences while participating in

the learning activity show consistent alignment with what P21 characterizes as 21st century

Learning and Innovation, Life and Career, and Socio-Cultural Skills, excluding creativity and

innovation. Six of the seven basic commonalities that foster meaningful learning and form the

underlying principles of the non-formal learning process were also discussed in the five dyad

interviews (Viable Characteristics column). The seventh characteristic, the role of the teacher as

facilitator, was also evident throughout student descriptions of their learning. This role is

apparent as there was no mention of teacher intervention in any of the interviews, rather, students

consistently described how they took the initiative to collaborate, think critically, and problem

solve, consistently figuring out difficult situations for themselves, without direct teacher

intervention.

The researcher revisited the learning experience transcripts to match student responses in

interviews to observed behaviors, looking for any disconfirming cases as well as confirmation of

student descriptions. Learning experience transcripts corroborated student interview utterances,

as each of the five dyads described experiences in the interview that were directly found in each

of the experience transcripts. Learning experience data analysis also corroborated the findings of

the student interview, with heavy emphasis on collaboration and communication as well as

critical thinking skills and initiative, self-direction and productivity. Leadership, responsibility,

and accountability, problem solving, and flexibility and adaptability were found to a lesser

extent, but were described in all five dyad interviews as well as the learning experience data

analysis. While use of creativity and innovation was a direct question in the interviews, students

61
could recall little or no use of it during the learning experience. This was supported by the

learning experience transcripts, as creativity and innovation only accounted for between 1% and

7% of student utterances.

Summary of Findings

After utilizing the pilot study to inform the research and develop a coding scheme with

intercoder reliability, quantitative data analysis findings derived from coded transcripts of the

non-formal learning experience were presented in chapter 4 to answer Research Question 1.

Qualitative analysis findings of student post-interview transcripts were presented to answer

Research Question 2. Descriptive statistics, content analysis, and corroborative analysis of the

interview and orienteering experience data were utilized to obtain a deeper understanding and

further demonstrate concurrent validity.

According to the five coded transcripts of the non-formal learning experience, 21st

century skills with highest student engagement by far were critical thinking and initiative, self-

direction and productivity. Qualitative analysis of the post-experience interviews corroborates

these findings, as all five dyads consistently discuss multiple engagements in these skills.

Specific elaboration of critical thinking was characterized in student discussions of how they

utilized various types of reasoning, analysis, synthesis, and critical reflection to solve problems

as they came about. Discussion of initiative, self-direction and productivity was also abundant in

the post-experience interviews. In each interview, students discussed how they balanced tactical

goals with strategic goals, and monitored, defined, and prioritized tasks. They re-evaluated their

prioritizations (critical thinking) throughout the experience, and changed priorities as needed

(flexible thinking). Students described how they set goals for success, even in the face of

obstacles and managed the tasks to gain the most points in the least amount of time. Student

62
engagement in productivity was also evident in the post-interviews through students’ discussion

of how they managed their time to most efficiently complete tasks.

Leadership, responsibility and accountability, with student engagement between 11% and

16% during the learning experience, was evidenced in the interviews through student

descriptions of how they leveraged their strengths and the strengths of their teammates to

accomplish their common goals. Ethical behavior was also evident throughout, with dyads even

helping other teams through difficulties, and students clearly held themselves accountable for

results. There was little evidence of creativity and innovation in the post-experience interview,

even though students were specifically asked about it. Creativity and innovation accounted for

the least amount of engagement (between 1% and 4%) during the learning experience.

Quantitative learning experience data determined between 41% and 58% student

engagement in collaboration during the experience. Students identified these skills within the

post-experience interview through utterances such as “I think it was a good exercise in building

teamwork skills and cooperation, and respect toward your partner’s decisions” and “I'd point,

he'd run over there, stamp it and I'd still try to figure out where the map was going.” Students

characterized collaboration as both aiding problem solving (leveraging each other’s strengths)

and assisting in completing tasks more quickly.

While engagement in initiative, self-direction and productivity, critical thinking skills,

and communication and collaboration clearly dominated the non-formal learning experience,

students also showed engagement in leadership, responsibility and accountability, problem

solving, flexibility and adaptability, with a very small percentage of engagement in creativity and

innovation. Copious student cognitive engagement in these 21st century Learning and Innovation

Skills, Life and Career Skills and Socio-Cultural Skills, excluding creativity and innovation, was

63
evident, and the viability of the non-formal learning experience was clearly characterized

through student articulation of this engagement.

64
 CHAPTER V: CONCLUSIONS, IMPLICATIONS, AND RECOMMENDATIONS

This chapter discusses conclusions, implications, and recommendations derived from the

research. Conclusions are based on findings from data analysis as presented in the previous

chapter, and focus on answering the two research questions. Implications resulting from these

conclusions are presented for future research and practice, followed by recommendations for

application of these findings to research and practice. Recognizing the inherent limitations of this

research, the following conclusions, implications, and recommendations are presented below.

Research Question 1

To what extent are students engaged in the following 21st century skills while

participating in a non-formal learning experience: Learning and Innovation (creativity

and innovation, critical thinking, problem solving), Life and Career (flexibility and

adaptability, initiative, self-direction and productivity, leadership, responsibility and

accountability) and Socio-Cultural (communication and collaboration)?

The above question was answered through analyses of data collected in video-recordings of

students participating in a 35-minute orienteering experience. What follows are conclusions

drawn from this analysis.

Conclusions: Learning and Innovation Skills. Within Learning and Innovation Skills,

one can conclude that this orienteering experience is extremely effective for engaging students in

critical thinking (Table 3, p. 52). One can also conclude that several steps of critical thinking are

necessary to move toward each problem solution (PS). From this data it can also be concluded

that this orienteering experience was not very conducive to student engagement in creativity and

innovation (CI).

65
 Conclusions: Life and Career Skills. From these data, it can be concluded that as

designed, this orienteering experience is a very strong instructional approach for engaging

students in initiative, self-direction and productivity (ISP) (Table 3, p.52). It can also be

concluded that students marginally engage in the 21st century skills of leadership, responsibility

and accountability (LRA) through this experience.

While flexibility and adaptability (FA) comprised a small part of total overall

engagement, its existence is important to the findings of this study. These skills were

demonstrated consistently across all five dyad experiences and corroborated in the interviews

(Table 6, p.57).

Conclusions: Socio-Cultural Skills. Based on the data (Table 4, p. 54), it can be

concluded that the design of this non-formal instruction (task interdependence requiring

teammates to rely on each other to solve problems) leads to high student engagement in

collaboration.

Research Question 2

The second research question in the study, “What characterizes a viable non-formal

learning experience facilitating student engagement in 21st century skills?” was answered

through qualitative analysis of data from the post-experience interviews. The findings from

analysis of the interviews corroborated the findings from analysis of the learning experience.

Conclusions: Characteristics of Viable Non-Formal Learning Experiences. From

analysis of student interviews, it can be concluded that the following characteristics reflect a

viable non-formal learning experience. The experience is characterized as one that fosters

integration of skills and knowledge to solve problems, emboldens students to be active,

autonomous, self-regulated decision-makers, supports collaborative construction of knowledge,

66
provides meaningful activities within the context of an authentic learning environment, includes

continuous formative and authentic assessment within tasks, and encourages reflection and

articulation to foster the formation of abstractions and enable tacit knowledge to become explicit

(Table 6, p. 57).

Summary of Conclusions

Based on findings as presented in Chapter IV, the following main conclusions were

reached.

1. This particular non-formal learning experience was strong in engaging students in

collaboration, critical thinking and initiative, self-direction and productivity. It also

engages students in leadership, responsibility and accountability, problem solving, and to

a lesser extent, flexibility and adaptability.

2. This particular non-formal learning experience was not strong in engaging students in

creativity and innovation.

3. A viable non-formal learning experience is characterized as one that fosters integration of

skills and knowledge to solve problems, emboldens students to be active, autonomous,

self-regulated decision-makers, supports collaborative construction of knowledge,

provides meaningful activities within the context of an authentic learning environment,

includes continuous formative and authentic assessment within tasks, and encourages

reflection and articulation to foster the formation of abstractions and enable tacit

knowledge to become explicit.

Implications

The conclusions reached in this study reveal the following set of specific implications for

researchers, educators, administrators, and teacher educators.

67
 1. Given this study found non-formal learning to be a viable approach for engaging students

in 21st century skills, the implications are that there now exists a potential avenue for

exploring the measurement of non-domain specific engagement in 21st century skills.

Measurement of non-domain specific skills would complement the current measurement

of domain-specific skill as assessed in Common Core State Standards and Next

Generation Science Standards, and ultimately situate non-formal learning as a parallel

instructional approach to that of traditional formal learning.

2. This study revealed that students were consistently engaged in the same non-domain

specific 21st century skills targeted in Common Core State Standards and Next

Generation Science Standards, therefore the implications are that non-formal learning is a

pathway for achieving current educational initiatives and a viable use of student

instructional time. Focusing instructional time on non-formal learning both within the

formal school day and outside of it are valuable means for achieving current educational

initiatives.

3. As students were engaged in the specific 21st century skills that this particular non-formal

learning activity was intentionally designed to foster, the implication exists that

instructional design for non-formal learning deliberately target certain non-domain

specific skills. Existing non-formal learning activities can be adjusted to more fully

engage students in targeted non-domain specific 21st century skills, and new learning

opportunities can be designed with these specific goals in mind.

Recommendations

Based on the implications of this study, the following actions are recommended for

researchers, educators, administrators, and teacher educators.

68
Recommendations for Researchers

The following recommendations for further research resulted from the findings and

conclusions of this study.

1. Further research should be conducted studying non-formal learning experiences with

larger populations and a variety of demographics and age groups.

2. Research should be conducted to determine whether working in dyads or larger groups

more effectively fosters engagement in 21st century skills.

3. Research should be conducted to determine how longer and/or more challenging

experiences would affect the outcomes.

4. Both single-gender and mixed-gender groups should be studied to determine if there are

gender-based differences in engagement in 21st century skills through non-formal

learning.

Recommendations for Practitioners

The following recommendations for teacher educators, supervisors/administrators, and

formal and non-formal educators were generated by the findings and conclusions of this study.

1. Professional development for formal educators in how to design and implement non-

formal learning instruction to engage students in 21st century skills should be provided.

2. Professional development for non-formal educators in how to better incorporate standards

such as Common Core, NGSS, and P21 into their teaching should be provided.

3. Practitioners should integrate non-formal learning practices into the formal school

instruction.

4. Practitioners should design instruction with specific 21st century skills in mind.

69
 References
Abbott, S. (Ed.). (2014, August 26). Hidden curriculum. Retrieved October 19, 2015.

Baker, E.L. & O’Neil Jr, H.F. (1994). Performance assessment and equity: A view from the

USA. Assessment in Education, 1(1), 11-26.

Bandura, A. (1971). Social learning theory. New York: General Learning Press.

Bell, P., & National Research Council (U.S.). Committee on Learning Science in Informal

Environments. (2009). Learning science in informal environments: People, places, and

pursuits. Washington, D.C: National Academies Press.

Berman, S. (2008). Performance-based learning: Aligning experiential tasks and assessment to

increase learning. Thousand Oaks, CA: Corwin Press.

Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in

Education, 5, 7–74.

Bozhovich, E. D. (2009). Zone of proximal development. Journal of Russian and East European

Psychology, 47(6), 48-69.

Bransford, J., Brown, A. L., Cocking, R. R., & National Research Council (U.S.). Committee on

Developments in the Science of Learning. (1999). How people learn: Brain, mind,

experience, and school. Washington, D.C: National Academy Press.

Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning.

Educational Researcher, 18 (1), 32-42.

Bruner, J.S. (1960). The process of education. Cambridge, MA: Harvard University Press.

Bruner, J.S. (1966). Toward a theory of instruction. Cambridge, MA: Harvard University

Press.

Byun, H., Lee, J., & Cerreto, F.A. (2014). Relative effects of three questioning strategies in ill-

structured, small group problem solving. Instructional Science, 42, 229-250.

70
Cain, J., Cummings, M., & Stanchfield J. (2005). A teachable moment: A facilitator’s guide to

activities for processing, debriefing, reviewing and reflection. Dubuque, IA: Kendall

Hunt Publishing.

Cairns, T., Merrifield, J. & McGivney, V. (2000). Informal Learning: Report of a Search

Conference held in July 2000. Discussion Paper. Learning from Experience Trust.

Callison, D. (2001). Constructivism. School Library Media Activities Monthly, 18(4), 35.

Casson, A. (2009). Assessment in outdoor education (Doctoral dissertation) Retrieved from

Qspace library. (200906)

Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R.J. Sternberg

(Ed.), Advances in the psychology of human intelligence. (p. 7-75). Hillsdale, NJ:

Earlbaum.

Cho, Y. (2008). Intercoder reliability. In P. Lavrakas (Ed.), Encyclopedia of survey research

methods. (pp. 345-346). Thousand Oaks, CA: SAGE Publications, Inc.

Cohen, L., Manion, L., & Morrison, K. (2000). Research methods in education. New York, NY:

RoutledgeFalmer.

Collins, A. (1989) Cognitive apprenticeship and instructional technology. Retrieved from

https://www.ideals.illinois.edu/bitstream/handle/2142/17920/ctrstreadtechrepv01989i004

74_opt.pdf?sequence=1

Collins, A., Brown, J.S., & Newman, S.E. (1987). Cognitive apprenticeship: Teaching the craft

of reading, writing, and mathematics. Center for the Study of Reading. Technical Report

No. 43.

Coombs, P. H., Prosser, C. & Ahmed, M. (1973). New Paths to Learning for Rural Children

And Youth, New York: International Council for Educational Development.

71
The Conference Board, Inc., the Partnership for 21st Century Skills, Corporate Voices for

Working Families, and the Society for Human Resource Management. (2006). Are the

Ready to Work? Employers’ Perspectives on the Basic Knowledge and of New Entrants

to the 21 Century U.S. Workforce. Retrieved from

http://www.p21.org/storage/documents/FINAL_REPORT_PDF09-29-06.pdf

Corno, L., & Mandinach, E. (1983). The role of cognitive engagement in classroom learning and

motivation. Educational Psychologist, 18, 88–108.

Costa, Arthur L., & Kallick, B. (2003). Assessment strategies for self-directed learning.

Thousand Oaks: Corwin Press.

Creswell, JW. (1998). Qualitative Inquiry and Research Design Choosing Among Five

Traditions. Thousand Oaks, CA: Sage Publications.

Daniel, R. C., & Embretson, S. E. (2010). Designing cognitive complexity in mathematical

problem-solving items. Applied Psychological Measurement, 35(5), 348-364.

Darling-Hammond, L., & Snyder, J. (2000). Authentic assessment in teaching in context.

Teaching and Teacher Education. 16, 523-545.

Davidson, J. E., & Sternberg, R. J. (1998). Smart problem solving: How metacognition helps. In

D. J. Hacker, J. Dunlosky & A. C. Graesser (Eds.), Metacognition in educational theory

and practice (pp. 47–68). Mahwah, NJ: Lawrence Erlbaum.

Denzin, NK. (1978). Sociological methods. New York: McGraw-Hill.

Desforges, C. (1995). How does experience affect theoretical knowledge for teaching? Learning

and Instruction, 5(4), 385-400.

Dewey, J. (1938). Logic, the theory of inquiry. New York: H. Holt and company.

Dominowski, R.L., & Dallob, P. (1995). Insight and problem solving. In R.J. Sternberg

72
 & J.E. Davidson (Eds.), The nature of insight (pp. 33–62). Cambridge, MA: MIT Press.

Driscoll, M. P. (1994). Psychology of learning for instruction. Boston, MA: Allyn and Bacon.

Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the

acquisition of expert performance. Psychological Review, 100(3), 363-406.

Falk, J. & Dierkin, L. (2000). Learning from museums: Visitor experiences and the making of

meaning. Lanham, MD: AltaMira Press.

Falk, J., & Storksdieck, M. (2005). Using the contextual model of learning to understand visitor

learning from a science center exhibition. Science Education, 89(5), 744-778.

Feichas, H. (2010). Bridging the gap: Informal learning practices as a pedagogy of integration.

Brazilian Journal of Music Education, 27, (1), 47-58.

Fosnot, C. T., & Perry, R. S. (1996). Constructivism: A psychological theory of learning.

Constructivism: Theory, perspectives, and practice, 8-33.

Fredricks, J. A., Blumenfeld, P. C., & Paris, A. (2004). School engagement: Potential of the

concept: State of the evidence. Review of Educational Research, 74, 59–119.

Gabbei, R. (2004). Generating effective facilitation questions for team building/personal

challenge activities. Journal of Physical Education, Recreation and Dance. 75(9), 20-24,

49.

Gagné, R. M. & Smith Jr, E. C. (1962) A study of the effects of verbalization on problem

solving. Journal of Experimental Psychology, 63(1), 12-18.

Gardner, H. (1991). The unschooled mind: How children think and how schools should teach.

New York, NY: Basic.

Gay, L., Mills. G., & Airasian, P. (2006). Educational research: Competencies for

analysis and application (8th ed.). New York: Prentice Hall.

73
Ge, X., & Land, S. M. (2003). Scaffolding students' problem-solving processes in an ill-

structured task using question prompts and peer interactions. Educational Technology

Research and Development, 51 (1), 21-38.

Ge, X., & Land, S. (2004). A conceptual framework for scaffolding ill-structured problem-

solving process using question prompts and peer interactions. Educational Technology

Research and Development, 52 (2), 5–22.

Getzels, J. W. (1979). Problem finding: A theoretical note. Cognitive Science, 3(2), 167-171.

Gielen, S., Dochy, F. & Dierick, S. (2003) Evaluating the consequential validity of new

modes of assessment: The influence of assessment on learning, including the pre-, post-,

and true assessment effects, in: M. Segers, F. Dochy & E. Cascallar (Eds) Optimising

new modes of assessment: In search of quality and standards (pp. 37-54). Netherlands:

Kluwer Academic Publishers.

Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for

qualitative research. Chicago: Aldine Pub. Co.

Google Maps (2015). Retrieved from https://www.google.com/maps/place/Blacksburg+Middle

+School/@37.2150969,-80.4633669,17z/data=!3m1!4b1!4m2!3m1!1s0x884dead5edd

74adb:0x3933bd225c4d0826

Greater Expectations National Panel. (2002). Greater expectations: A new vision for learning as

a nation goes to college. Retrieved from

http://www.greaterexpectations.org/pdf/gex.final.pdf

Greiff, S. (2012). From interactive and collaborative problem solving: Current issues in the

Griffin, J.M. (1998). School-museum integrated learning experiences in science: A learning

journey. Retrieved from UTS Publishing. (AAT 2100254)

74
Gronlund, N. E. (2003). Assessment of Student Achievement, 7 t h ed. Boston, MA: Allyn &

Bacon.

Guilkers, J.T.M., Bastiaens, T.J., & Kirschner. (2004). A five-dimensional framework for

authentic assessment. Education Technology research and Development, 52(3), 67-86.

Hall, W. A., Long, B., Bermback, N., Jordan, S. and Patterson, K. (2005). Qualitative teamwork

issues and strategies: Coordination through mutual adjustment. Qualitative Health

Research, 15, 394–410.

Hart, D. (1994). Authentic assessment. New York, NY: Addison-Wesley.

Herrington, J., & Oliver, R. (2000). An instructional design framework for authentic learning

environments. Educational Technology Research and Development, 48(3), 23-48.

Herrington, J., Oliver, R., & Reeves, T. C. (2003). Patterns of engagement in authentic online

learning environments. Australian Journal of Educational Technology, 19(1), 59–71.

Hibbard, M. (1996). A teacher’s guide to performance-based learning and assessment.

Alexandria, VA: ASCD.

Hoover, J. D & Whitehead, C. (1975), An experiential-cognitive methodology in the first

course in management: Some preliminary results, Simulation Games and Experiential

Learning in Action, Richard H. Buskirk (ed.), 25-30.

Horn, J. L., & Blankson, N. (2005). Foundations for better understanding of cognitive abilities.

In D. P. Flanagan, & P. L. Harrison (Eds.), Contemporary intellectual assessment:

Theories, tests, and issues (pp.41–68), New York: Guilford Press.

Humphreys, D. (2005). Why integrative learning? Why now? Peer Review, 7(4), 30.

Johnson, E.J. (1988). Expertise and decision under uncertainty: Performance and process.

75
 In M.T.H. Chi, R. Glaser & M.J. Farr (Eds.), The nature of expertise (pp.209-228).

Hillsdale, NJ: Erlbaum.

Jonassen, D.H. (1997). Instructional design model for well-structured and ill-structured problem-

solving learning outcomes. Educational Technology: Research and Development, 45 (1),

65-95.

Jonassen, D.H. (2000). Toward a Design Theory of Problem Solving. Educational Technology

Research and Development, 48(4), 63-85.

Joplin, L. (1995). On defining experiential education. In K. Warren, M. Sakofs & Hunt, J.S.

(Eds.). The theory of experiential education (15-22). Dubuque, IA:

Kendall/Hunt Publishing Company.

Kane, M., Crooks, T. & Cohen, A. (1999). Validating measures of performance. Educational

Measurement: Issues and Practice, 18(2), 5-17.

Kelly, J. (2000). Rethinking the elementary science methods course: a case for content,

pedagogy, and informal science education. International Journal of Science Education,

22 (7), 755-767.

Klenke, K. (2008). Qualitative research in the study of leadership. Bingley, UK: Emerald.

Kolb, A.Y. & Kolb, D.A. (2005). Learning styles and learning spaces: Enhancing experiential

learning in higher education. Academy of Management Learning & Education, 4(2), 193-

212.

Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and

development. Englewood Cliffs, N.J: Prentice-Hall.

Kuhlthau, C. (2004). Seeking meaning: a process approach to library and information services

(Second edition ed.). Westport, CT: Libraries Unlimited, Inc.

76
Lampert, M. (1986). Knowing, doing, and teaching multiplication. Cognition and Instruction,

3(4), 305-342.

Lave, J. (1988). Cognition in practice: Mind, mathematics, and culture in everyday life.

Cambridge; New York: Cambridge University Press.

Lave, J. & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation.

Cambridge: Cambridge University Press.

Layton, D. (1993). Technology’s challenge to science education: Cathedral, quarry or company

store? Open University Press: Buckingham, Philadelphia.

Lombardi, M. M. (2007). Authentic learning for the 21st century: an overview. Retrieved from

https://net.educause.edu/ir/library/pdf/ELI3009.pdf

Lu, C. and Schulman, S. W. (2008). Rigor and flexibility in computer-based qualitative

research: Introducing the Coding Analysis Toolkit. International Journal of

Multiple Research Approaches, 2, 105–117.

Martens, R. L., Gulikers, J., & Bastiaens, T. (2004). The impact of intrinsic motivation on

e-learning in authentic computer tasks. Journal of Computer Assisted Learning, 20(5),

368–376.

Mayer, R.E. (1998). Cognitive, metacognitive, and motivational aspects of problem solving.

Instructional Science. 26, 49-63.

McCormick, R. (2004). Issues of learning and knowledge in technology education. International

Journal of Technology and Design Education, 14(1), 21-44.

Microsoft. (2014). Microsoft Excel [computer software]. Redmond, Washington: Microsoft.

Miles, M.B., & Huberman, A.M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks,

CA: Sage.

77
Mims, C. (2003). Authentic learning: A practical introduction and guide for implementation.

Meridian: A Middle School Computer Technologies Journal, 6(1), 1-3.

Nadler, R. S., & Luckner, J. L. (1992). Processing the adventure experience: Theory and

practice. Dubuque, Iowa: Kendall/Hunt Publishing Company.

National Governors Association Center for Best Practices & Council of Chief State School

Officers. (2010). Common Core State Standards for Mathematics. Retrieved from:

http://www.nctm.org/uploadedFiles/Standards_and_Positions/Common_Core_State_

Standards/Math_Standards.pdf

National Research Council (NRC). 2009. Learning science in informal environments.

Washington, DC: National Academies Press.

National Research Council. (2012). A Framework for K-12 Science Education: Practices,

Crosscutting Concepts, and Core Ideas. Committee on a Conceptual Framework for New

K-12 Science Education Standards. Board on Science Education, Division of Behavioral

and Social Sciences and Education. Washington, DC: The National Academies Press.

Newmann, F. M., Wehlage, G. G., & Lamborn, S. D. (1992). The significance and sources of

student engagement. In F. Newmann (Ed.), Student engagement and achievement in American

secondary schools (pp. 11-39). New York, NY: Teachers College Press.

Next Generation Science Standards. (2013) Appendix A- Conceptual shifts in the next generation

science standards. Retrieved from:

http://www.nextgenscience.org/sites/ngss/files/Appendix%20A%20-

%204.11.13%20Conceptual%20Shifts%20in%20the%20Next%20Generation%20Scienc

e%20Standards.pdf

Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A

78
 model and seven principles of good feedback practice. Studies in Higher Education, 31,

199–218.

Nokes, T.J., Schunn, C.D. & Chi, M.T.H. (2010). Problem solving and human expertise.

International Encyclopedia of Education. 5, 265-272.

Nussbaum, M. C. (1998). Cultivating humanity. Liberal Education, 84(2), 38.

Organization for Economic Co-operation and Development. (2014). Assessing problem-solving

skills in PISA 2012. PISA 2012 Results: Creative Problem Solving (Volume V): Students’

Skills in Tackling Real-Life Problems, OECD Publishing.

Organization for Economic Co-operation and Development. (2010). PISA 2012 field trial

problem solving framework. Retrieved from:

http://www.oecd.org/dataoecd/8/42/46962005.pdf

Orienteering USA. (2014). Orienteering lingo. Retrieved from

http://www.us.orienteering.org/new-o/o-lingo

Pajares, F. (2002). Gender and perceived self-efficacy in self-regulated learning. Theory into

Practice, 41(2), 116-125.

Palm, T. (2008). Performance assessment and authentic assessment: A conceptual analysis of the

literature. Practical Assessment, Research & Evaluation 13(4), 1-11.

Parlett, M., & Hamilton, D. (1976). Illuminative evaluation: A new approach to the study

of innovatory programs. In: G. Glass (Ed.) Evaluation Studies Reviewed Annual, Vol. 1.

Beverly Hills, CA: Sage.

Partnership for 21st Century Skills (P-21). (2015). Framework for 21st century learning.

Retrieved from http://www.p21.org/storage/documents/1.__p21_framework_2-pager.pdf

Partnership for 21st Century Skills (P-21). (2015). P21 framework definitions. Retrieved from:

79
 http://www.p21.org/storage/documents/docs/P21_Framework_Definitions_New_Logo_2

015.pdf

Patton, MQ. (1999). Enhancing the quality and credibility of qualitative analysis. HSR: Health

Services Research, 34 (5), 1189-1208.

Perkins, D. N. & Salomon, G. (1989). Are cognitive skills context-bound? Educational

Researcher, 18 (1), 16-25.

Pintrich, P. R., & Zusho, A. (2002). The development of academic self-regulation-chapter 10:

The role of cognitive and motivational factors. In A. Wigfield & and J.S. Eccles (Eds),

Development of achievement motivation. A volume in the educational psychology series

(pp. 249-284). San Diego, CA: Academic Press.

Pretz, J.E., Naples, A.J., & Sternberg, R.J. (2003). Recognizing, defining, and representing

problems. In J.E. Davidson & R.J. Sternberg (Eds.), The psychology of problem solving,

(pp. 3-30). New York: Cambridge University Press.

Rennie L.J. (2007). Learning science outside of school. In: Abell SK, Lederman NG (eds)

Handbook of research on science education. Lawrence Erlbaum, Mahwah, pp 125–167.

Resnick, L. B. (1987). The 1987 presidential address learning in school and out. Educational

Researcher, 16(9), 13-54.

Rotherham, A., and Willingham, D. (2009) 21st century skills: The challenges ahead.

Educational Leadership, 67(1), 16-21.

Ryan, Gery W. (1999) Measuring the typicality of text: Using multiple coders for more than just

reliability and validity checks. Human Organization, 58, 313–322.

Schloss, P. J., & Smith, M. A. (1999). Conducting research. Columbus, Ohio:

Merrill / Prentice Hall.

80
Schoenfeld, A.H. (2011). How we think: A theory of goal-oriented decision-making and its

Educational applications. New York, NY: Routledge

Shapiro, J. (2008). Percentage Frequency Distribution. In Paul J. Lavrakas (Ed.), Encyclopedia

of Survey Research Methods. (pp. 578-579). Thousand Oaks, CA: Sage Publications, Inc.

Silva, Elena. (2009). Measuring skills for 21st century learning. The Phi Delta Kappan, 90. (9).

Retrieved from http://www.jstor.org/stable/27652741

Simon, D. P. and Simon, H. A. (1978). Individual differences in solving physics problems. In

Siegler, R. (Ed.), Children’s Thinking: What Develops? Hillsdale, NJ: Erlbaum.

Singh, Mandhu. (2015). Global Perspectives on Recognising Non-formal and Informal Learning:

Why recognition Matters Springer International Publishing: New York.

Sinnott, J. D. (1989). A model for solution of ill-structured problems: Implications

for every day and abstract problem solving. In J. D. Sinott (Ed.), Everyday problem

solving: Theory and application (pp. 72-99). New York: Praeger.

Smith, M.K. (2008). Informal learning. The encyclopedia of informal education. Retrieved from

http://infed.org/mobi/informal-learning-theory-practice-and-experience

Stemler, Steven E. (2004). A comparison of consensus, consistency, and measurement

approaches to estimating interrater reliability. Practical Assessment, Research &

Evaluation, 9, 4. Retrieved from http://PAREonline.net/getvn.asp?v=9&n=4.

Sternberg, R. J. (1995). Expertise in complex problem solving: A comparison of alternative

concepts. In P. A. Frensch & J. Funke (Eds.), Complex problem solving. The European

perspective (pp. 295-321). Hillsdale, NJ: Erlbaum.

Teddlie, C., & Tashakkori, A. (2006). A general typology of research designs featuring mixed

methods. Research in the Schools, 13(1), 12-28.

81
Tennant, M. (1997). Psychology and Adult Learning. 2e, London: Routledge.

Voss, J. F. and Post, T. A. (1988). On the solving of ill-structured problems. In M. T. H. Chi,

R. Glaser, and M.J. Farr (Eds.), The nature of expertise (pp 261–285). Hillsdale, NJ:

Erlbaum.

Vygotsky, L. (1978). Mind in society: The development of higher mental processes. M.

Cole, V. John-Steiner, S. Scribner, and E. Souberman (Ed.). Reprint, Cambridge:

Harvard University Press.

Wagnor, T. (2008). Rigor Redefined. Educational Leadership,66 (2), 20-25.

Weber R.P. (1990) Basic Content Analysis. Sage Publications, Newbury Park, CA.

Wells, J.G. (2015). A century of professional organization influence: Findings from content

analyses of MVTTEC annual meetings. Journal of Technology Education, 25(3), 3-37.

Wells, J.G. (2008). A technology education perspective on the potential of STEM education.

Invited paper presented at the 95th Mississippi Valley Technology Teacher Education

Conference, St. Lois, MO.

Wenke, D., Frensch, P. A., & Funke, J. (2005). Complex Problem Solving and intelligence:

Empirical relation and causal direction. In R. J. Sternberg & J. E. Pretz (Eds.), Cognition

and intelligence: Identifying the mechanisms of the mind (pp.160-187). New York:

Cambridge University Press.

Wiggins, G. (1993). Assessment: Authenticity, context, and validity. Phi Delta Kappan, 75(3),

200-214.

Yin, Robert K. (2014) Case Study Research: Design and Methods. Sage, Thousand Oaks, CA.

82
 APPENDIX A

IRB Approval Letter (Pilot Study)

83
 Office of Research Compliance
Institutional Review Board
North End Center, Suite 4120, Virginia Tech
300 Turner Street NW
Blacksburg, Virginia
24061 540/231-4606
Fax 540/231-0959
email irb@vt.edu
website http://www.irb.vt.edu

MEMORANDUM
DATE: June 22, 2015

TO: John Wells, Lisa Ann Moyer

FROM: Virginia Tech Institutional Review Board (FWA00000572, expires April 25, 2018)

PROTOCOL TITLE: Problem Solving Metacognition though Orienteering

IRB NUMBER: 15-603

Effective June 22, 2015, the Virginia Tech Institution Review Board (IRB) Chair, David M Moore,
approved the New Application request for the above-mentioned research protocol.

This approval provides permission to begin the human subject activities outlined in the IRB-
approved protocol and supporting documents.

Plans to deviate from the approved protocol and/or supporting documents must be submitted
to the IRB as an amendment request and approved by the IRB prior to the implementation of
any changes, regardless of how minor, except where necessary to eliminate apparent immediate
hazards to the subjects. Report within 5 business days to the IRB any injuries or other
unanticipated or adverse events involving risks or harms to human research subjects or others.

All investigators (listed above) are required to comply with the researcher requirements

outlined at: http://www.irb.vt.edu/pages/responsibilities.htm

(Please review responsibilities before the commencement of your research.)

PROTOCOL INFORMATION:
Approved As: Expedited, under 45 CFR 46.110 category(ies) 5,6,7
Protocol Approval Date: June 22, 2015
Protocol Expiration Date: June 21, 2016
Continuing Review Due Date*: June 7, 2016

84
 *Date a Continuing Review application is due to the IRB office if human subject activities covered
under this protocol, including data analysis, are to continue beyond the Protocol Expiration Date.

FEDERALLY FUNDED RESEARCH REQUIREMENTS:

Per federal regulations, 45 CFR 46.103(f), the IRB is required to compare all federally funded
grant proposals/work statements to the IRB protocol(s) which cover the human research
activities included in the proposal / work statement before funds are released. Note that this
requirement does not apply to Exempt and Interim IRB protocols, or grants for which VT is not
the primary awardee.

The table on the following page indicates whether grant proposals are related to this IRB
protocol, and which of the listed proposals, if any, have been compared to this IRB protocol, if
required.
IRB Number 15-603 page 2 of 2 Virginia Tech Institutional Review Board

Date* OSP Sponsor Grant Comparison Conducted?

Number

* Date this proposal number was compared, assessed as not requiring comparison, or
comparison information was revised.

If this IRB protocol is to cover any other grant proposals, please contact the IRB office
(irbadmin@vt.edu) immediately.

85
 APPENDIX B
IRB Approval Letter (Actual Study)

86
 Office of Research Compliance
Institutional Review Board
North End Center, Suite 4120, Virginia Tech
300 Turner Street NW
Blacksburg, Virginia
24061 540/231-4606
Fax 540/231-0959
email irb@vt.edu
website http://www.irb.vt.edu

MEMORANDUM
DATE: September 24, 2015

TO: John Wells, Lisa Ann Moyer

FROM: Virginia Tech Institutional Review Board (FWA00000572, expires July 29, 2020)

PROTOCOL TITLE: Developing 21st Century Skills through Informal Learning Experiences

IRB NUMBER: 15-692

Effective September 23, 2015, the Virginia Tech Institution Review Board (IRB) Chair, David M
Moore, approved the New Application request for the above-mentioned research protocol.

This approval provides permission to begin the human subject activities outlined in the IRB-
approved protocol and supporting documents.

Plans to deviate from the approved protocol and/or supporting documents must be submitted
to the IRB as an amendment request and approved by the IRB prior to the implementation of
any changes, regardless of how minor, except where necessary to eliminate apparent immediate
hazards to the subjects. Report within 5 business days to the IRB any injuries or other
unanticipated or adverse events involving risks or harms to human research subjects or others.

All investigators (listed above) are required to comply with the researcher requirements

outlined at: http://www.irb.vt.edu/pages/responsibilities.htm

(Please review responsibilities before the commencement of your research.)

PROTOCOL INFORMATION:
Approved As: Expedited, under 45 CFR 46.110 category(ies) 6,7
Protocol Approval Date: September 23, 2015
Protocol Expiration Date: September 22, 2016
Continuing Review Due Date*: September 8, 2016
*Date a Continuing Review application is due to the IRB office if human subject activities covered
under this protocol, including data analysis, are to continue beyond the Protocol Expiration Date.

87
 FEDERALLY FUNDED RESEARCH REQUIREMENTS:
Per federal regulations, 45 CFR 46.103(f), the IRB is required to compare all federally funded
grant proposals/work statements to the IRB protocol(s) which cover the human research
activities included in the proposal / work statement before funds are released. Note that this
requirement does not apply to Exempt and Interim IRB protocols, or grants for which VT is not
the primary awardee.

The table on the following page indicates whether grant proposals are related to this IRB
protocol, and which of the listed proposals, if any, have been compared to this IRB protocol, if
required.
IRB Number 15-692 page 2 of 2 Virginia Tech Institutional Review Board

Date* OSP Sponsor Grant Comparison Conducted?

Number

* Date this proposal number was compared, assessed as not requiring comparison, or
comparison information was revised.

If this IRB protocol is to cover any other grant proposals, please contact the IRB office
(irbadmin@vt.edu) immediately.

88
 APPENDIX C
P21 Skills Coding Scheme (Original)

89
Category
Learning and Innovation Skills
Creativity and Innovation
Think Creatively
Work Creatively with Others
Implement Innovations
Critical Thinking and Problem
Solving
Reason Effectively
Use Systems Thinking
Make Judgements and
Decisions
Code Definition
CI
CI-TC  use a wide range of idea creation techniques
(such as brainstorming)
 Create new and worthwhile ideas (both
incremental and radical concepts)
 Elaborate, refine, analyze and evaluate their
own ideas in order to improve and maximize
creative efforts.
CI-CO  Develop, implement and communicate new
ideas to others effectively
 Be open and responsive to new and diverse
perspectives; incorporate group input and
feedback into the work
 Demonstrate originality and inventiveness in
work and understand the real world limits to
adopting new ideas.
 View failure as an opportunity to learn;
understand that creativity and innovation is a
long-term, cyclical process of small success and
frequent mistakes
CI-II  Act on creative ideas to make a tangible and
useful contribution to the field in which the
innovation will occur
TPS
TPS-R  Use various types of reasoning (inductive,
deductive, etc.) as appropriate to the situation
TPS-S  Analyze how parts of a whole interact with each
other to produce overall outcomes in a complex
system
TPS-D  Effectively analyze and evaluate evidence,
arguments, claims and beliefs
 Analyze and evaluate major alternative points of
view
 Synthesize and make connections between
information and arguments
 Interpret information and draw conclusions
based on the best analysis
90

Solve Problems:
Communication and
Collaboration
Communicate Clearly
Collaborate with Others
Life and Career Skills
Flexibility and Adaptability
Adapt to Change
Be Flexible
Initiative and Self-Direction
Manage Goals and Time
 Reflect critically on learning experiences and
processes
TPS-P  Solve different kinds of non-familiar problems in
both conventional and innovative ways
 Identify and ask significant questions that clarify
various points of view and lead to better
solutions
CC
CC-CC  Articulate thoughts and ideas effectively using
oral, written and nonverbal communication skills
in a variety of forms and contexts
 Listen effectively to decipher meaning, including
knowledge, values, attitudes and intentions
 Use communication for a range of purposes
(e.g. to inform, instruct, motivate and persuade)
 Utilize multiple media and technologies, and
know how to judge their effectiveness a priori as
well as assess their impact
 Communicate effectively in diverse
environments (including multi-lingual)
CC-CO  Demonstrate ability to work effectively and
respectfully with diverse teams
 Exercise flexibility and willingness to be helpful
in making necessary compromises to
accomplish a common goal
 Assume shared responsibility for collaborative
work, and value the individual contributions
made by each team member
FA
FA-A  Adapt to varied roles, jobs responsibilities,
schedules and contexts
 Work effectively in a climate of ambiguity and
changing priorities
FA-F  Incorporate feedback effectively
 Deal positively with praise, setbacks and
criticism
 Understand, negotiate and balance diverse
views and beliefs to reach workable solutions,
particularly in multi-cultural environments
ISD
ISD-M  Set goals with tangible and intangible success
criteria
91

Work Independently
Be Self-directed Learners
 Balance tactical (short-term) and strategic
(long-term) goals
 Utilize time and manage workload efficiently
ISD-WI  Monitor, define, prioritize and complete tasks
without direct oversight
ISD-SD  Go beyond basic mastery of skills and/or
curriculum to explore and expand one’s own
learning and opportunities to gain expertise
 Demonstrate initiative to advance skill levels
towards a professional level
 Demonstrate commitment to learning as a
lifelong process
 Reflect critically on past experiences in order to
inform future progress

Social and Cross-Cultural SCS

Skills
Interact Effectively with Others: SCS-IO  Know when it is appropriate to listen and when
to speak
 Conduct themselves in a respectable,
professional manner
Work Effectively in Diverse SCS-DT  Respect cultural differences and work effectively
Teams with people from a range of social and cultural
backgrounds
 Respond open-mindedly to different ideas and
values
 Leverage social and cultural differences to
create new ideas and increase both innovation
and quality of work
Productivity and PA
Accountability
Manage Projects PA-P  Set and meet goals, even in the face of
obstacles and competing pressures
 Prioritize, plan and manage work to achieve
intended results

Produce Results PA-R  Demonstrate additional attributes associated

with producing high quality products
including the abilities to:
- Work positively and ethically
- Manage time and projects effectively
- Multi-task
- Participate actively, as well as be reliable and
punctual

92
 - Present oneself professionally and with proper
etiquette
- Collaborate and cooperate effectively with teams
- Respect and appreciate team diversity
 - Be accountable for results
Leadership and LR 
Responsibility
Guide and Lead Others LR-LO  Use interpersonal and problem-solving skills to
influence and guide others toward a goal
 Leverage strengths of others to accomplish a
common goal
 Inspire others to reach their very best via
example and selflessness
 Demonstrate integrity and ethical behavior in
using influence and power
Be Responsible to Others LR-RO  Act responsibly with the interests of the larger
community in mind

93
 APPENDIX D
Revised Coding Scheme for Panelists’ Review

94
CATEGORY CODE DEFINITION
Learning and
Innovation Skills
Creativity and CI  Use a wide range of creation techniques
Innovation  Demonstrate originality and inventiveness in
work and understand the real world limits to
adopting new ideas
 Create new and worthwhile ideas (both
incremental and radical concepts)
 Act on creative ideas to make a tangible and
useful contribution to the field in which the
innovation will occur
Critical Thinking and TPS
Problem Solving
Critical Thinking TPS-C  Use various types of reasoning (inductive,
deductive, etc.) as appropriate to the situation
 Effectively analyze and evaluate evidence,
arguments, claims and beliefs
 Analyze how parts of a whole interact with
each other to produce overall outcomes in a
complex system (systems thinking)
 Reflect critically on learning experiences and
processes
Make Judgements and TPS-D  Analyze and evaluate major alternative points
Decisions of view
 Synthesize and make connections between
information and arguments
Interpret information and draw conclusions
based on the best analysis
Problem Solving TPS-P  Solve different kinds of non-familiar problems
in both conventional and innovative ways
 Identify and ask significant questions that
clarify various points of view and lead to better
solutions
Communication and CS
Collaboration/Social
and Cultural Skills
Communication/Social CS-CS  Articulate thoughts and ideas effectively using
Skills oral, written and nonverbal communication
(Take out ALL of this? skills in a variety of forms and contexts
Everything they do  Listen effectively to decipher meaning,
involves including knowledge, values, attitudes and
communication) intentions

95
  Use communication for a range of purposes
(e.g. to inform, instruct, motivate and
persuade)
 Interact effectively with others
 Communicate effectively in diverse
environments (including multi-lingual)
 Know when it is appropriate to listen and when
to speak
 Conduct themselves in a respectable,
professional manner
 Present oneself professionally and with proper
etiquette
 communicate new ideas to others effectively
Collaboration CS-CL  Be open and responsive to new and diverse
perspectives; incorporate group input and
feedback into the work
 Assume shared responsibility for collaborative
work, and value the individual contributions
made by each team member
 Work effectively in diverse teams
 Respect cultural differences and work
effectively with people from a range of social
and cultural backgrounds
 Respond open-mindedly to different ideas
and values
 Collaborate and cooperate effectively with
teams
Life and Career Skills 
Flexibility and FA  Adapt to varied roles, jobs responsibilities,
Adaptability schedules and contexts
 Work effectively in a climate of ambiguity and
changing priorities
 Incorporate feedback effectively
 Reflect critically on past experiences in order
to inform future progress
 Deal positively with praise, setbacks and
criticism
 Exercise flexibility and willingness to be helpful
in making necessary compromises to
accomplish a common goal
 View failure as an opportunity to learn;
understand that creativity and innovation is a
long-term, cyclical process of small success
and frequent mistakes

96
Initiative and Self- ISD  Set goals with tangible and intangible success
Direction criteria
 Balance tactical (short-term) and strategic
(long-term) goals
 Go beyond basic mastery of skills and/or
curriculum to explore and expand one’s own
learning and opportunities to gain expertise
 Set and meet goals, even in the face of
obstacles and competing pressures
Prioritize, plan and manage work to achieve
intended results
Productivity and PA  Multi-task
Accountability  Be accountable for results
(Manage Projects and
Produce Results)
Leadership and LR  Use interpersonal and problem-solving skills to
Responsibility influence and guide others toward a goal
 Leverage strengths of others to accomplish a
common goal
 Inspire others to reach their very best via
example and selflessness
 Demonstrate integrity and ethical behavior in
using influence and power
 Act responsibly with the interests of the larger
community in mind

97
 APPENDIX E
P21 Framework for 21st Century Learning

98
LEARNING AND INNOVATION SKILLS
Learning and innovation skills increasingly are being recognized as those that separate
students who are prepared for a more and more complex life and work environments in
the 21st century, and those who are not. A focus on creativity, critical thinking,
communication and collaboration is essential to prepare students for the future.

CREATIVITY AND INNOVATION

Think Creatively
• Use a wide range of idea creation techniques (such as brainstorming)
• Create new and worthwhile ideas (both incremental and radical concepts)
• Elaborate, refine, analyze and evaluate their own ideas in order to improve and
maximize creative efforts
Work Creatively with Others
• Develop, implement and communicate new ideas to others effectively
• Be open and responsive to new and diverse perspectives; incorporate group input and
feedback into the work
• Demonstrate originality and inventiveness in work and understand the real world
limits to adopting new ideas
• View failure as an opportunity to learn; understand that creativity and innovation is a
long-term, cyclical process of small successes and frequent mistakes
Implement Innovations
• Act on creative ideas to make a tangible and useful contribution to the field in which
the innovation will occur
CRITICAL THINKING AND PROBLEM SOLVING
Reason Effectively
• Use various types of reasoning (inductive, deductive, etc.) as appropriate to the
situation
Use Systems Thinking
• Analyze how parts of a whole interact with each other to produce overall outcomes in
complex systems
Make Judgments and Decisions
• Effectively analyze and evaluate evidence, arguments, claims and beliefs
• Analyze and evaluate major alternative points of view
• Synthesize and make connections between information and arguments
• Interpret information and draw conclusions based on the best analysis
• Reflect critically on learning experiences and processes
Solve Problems
• Solve different kinds of non-familiar problems in both conventional and innovative ways
• Identify and ask significant questions that clarify various points of view and lead to better
solutions

99
COMMUNICATION AND COLLABORATION
Communicate Clearly
• Articulate thoughts and ideas effectively using oral, written and nonverbal
communication skills in a variety of forms and contexts
• Listen effectively to decipher meaning, including knowledge, values, attitudes and
intentions
• Use communication for a range of purposes (e.g. to inform, instruct, motivate and
persuade)
• Utilize multiple media and technologies, and know how to judge their effectiveness a
priori as well as assess their impact
• Communicate effectively in diverse environments (including multi-lingual)
Collaborate with Others
• Demonstrate ability to work effectively and respectfully with diverse teams
• Exercise flexibility and willingness to be helpful in making necessary compromises to
accomplish a common goal
• Assume shared responsibility for collaborative work, and value the individual
contributions made by each team member

LIFE AND CAREER SKILLS

Today’s life and work environments require far more than thinking skills and content knowledge.
The ability to navigate the complex life and work environments in the globally competitive
information age requires students to pay rigorous attention to developing adequate life and career
skills.

FLEXIBILITY AND ADAPTABILITY

Adapt to Change
• Adapt to varied roles, jobs responsibilities, schedules and contexts
• Work effectively in a climate of ambiguity and changing priorities
Be Flexible
• Incorporate feedback effectively
• Deal positively with praise, setbacks and criticism
• Understand, negotiate and balance diverse views and beliefs to reach workable solutions,
particularly in multi-cultural environments
INITIATIVE AND SELF-DIRECTION
Manage Goals and Time
• Set goals with tangible and intangible success criteria
• Balance tactical (short-term) and strategic (long-term) goals
• Utilize time and manage workload efficiently
Work Independently
• Monitor, define, prioritize and complete tasks without direct oversight
Be Self-directed Learners
• Go beyond basic mastery of skills and/or curriculum to explore and expand one’s own
learning and opportunities to gain expertise
• Demonstrate initiative to advance skill levels towards a professional level
• Demonstrate commitment to learning as a lifelong process
• Reflect critically on past experiences in order to inform future progress

100
SOCIAL AND CROSS-CULTURAL SKILLS
Interact Effectively with Others
• Know when it is appropriate to listen and when to speak
• Conduct themselves in a respectable, professional manner
Work Effectively in Diverse Teams
• Respect cultural differences and work effectively with people from a range of social and
cultural backgrounds
• Respond open-mindedly to different ideas and values
• Leverage social and cultural differences to create new ideas and increase both innovation
and quality of work
PRODUCTIVITY AND ACCOUNTABILITY
Manage Projects
• Set and meet goals, even in the face of obstacles and competing pressures
• Prioritize, plan and manage work to achieve the intended result
Produce Results
• Demonstrate additional attributes associated with producing high quality products
including the abilities to:
- Work positively and ethically
- Manage time and projects effectively
- Multi-task
- Participate actively, as well as be reliable and punctual
- Present oneself professionally and with proper etiquette
- Collaborate and cooperate effectively with teams
- Respect and appreciate team diversity
- Be accountable for results
LEADERSHIP AND RESPONSIBILITY
Guide and Lead Others
• Use interpersonal and problem-solving skills to influence and guide others toward a goal
• Leverage strengths of others to accomplish a common goal
• Inspire others to reach their very best via example and selflessness
• Demonstrate integrity and ethical behavior in using influence and power
Be Responsible to Others
• Act responsibly with the interests of the larger community in mind

101
 APPENDIX F
Panelists Comments on Coding Scheme

102
Category 1. Need If yes, how would you 2. Need If yes, how would
to modify? to you modify?
modify? modify?
(yes/no) (yes/no)
Learning and
Innovation
Skills
Creativity and No No
Innovation

Critical
Thinking and
Problem
Solving
Critical No yes I would combine all
Thinking three sub-categories
of TPS – they are
very similar and all
represent parts of
the non-linear
problem-solving
process.
Making No yes
Judgements
and Decisions
Problem No yes
Solving

Communication
and
Collaboration/
Social and
Cultural Skills
Communication Yes Include only: yes Everything involves
/Social Skills  Articulate communication –
thoughts and new drop this category.
ideas effectively
using oral, written
and nonverbal
communication
skills in a variety

103
 of forms and
contexts
 Listen effectively
to decipher
meaning,
including
knowledge,
values, attitudes
and intentions
Collaboration No Maybe? Will these concepts
be captured in the
participants’
statements?
Life and Career
Skills
Flexibility and No
Adaptability no

Initiative and Yes Add: (add productivity)

Self-Direction  Monitor, define, no
prioritize and
complete tasks
without direct
oversight
Productivity Yes Add: I think productivity
and  Manage time and yes would fit better
Accountability projects under self-direction.
effectively If they choose to
multi-task, that
should fit in with
their goal.

Accountability is
totally different – I
think it should go
with leadership and
responsibility
Leadership and No (add accountability)
Responsibility no

104
 APPENDIX G
Coding Scheme 1 and 2 (First Iteration with Panelists)

105
Coding Scheme 1
CATEGORY CODE DEFINITION Example
Utterances
Communication/ 1  Articulate thoughts and ideas effectively using oral,
Social Skills written and nonverbal communication skills in a
variety of forms and contexts
 Listen effectively to decipher meaning, including
knowledge, values, attitudes and intentions
 Use communication for a range of purposes (e.g. to
inform, instruct, motivate and persuade)
 Communicate effectively in diverse environments
(including multi-lingual)
 Know when it is appropriate to listen and when to
speak
 Conduct themselves in a respectable, professional
manner
 Present oneself professionally and with proper
etiquette
 communicate new ideas to others effectively
Collaboration 2  Be open and responsive to new and diverse
perspectives; incorporate group input and feedback
into the work
 Assume shared responsibility for collaborative work,
and value the individual contributions made by each
team member
 Work effectively in diverse teams
 Respect cultural differences and work effectively with
people from a range of social and cultural
backgrounds
 Respond open-mindedly to different ideas and
values
 Collaborate and cooperate effectively with teams

106
Coding Scheme 2
CATEGORY CODE DEFINITION Example
Utterances
Learning and
Innovation Skills
Creativity and CI  Use a wide range of creation techniques
Innovation  Demonstrate originality and inventiveness in work
and understand the real world limits to adopting new
ideas
 Create new and worthwhile ideas (both incremental
and radical concepts)
Act on creative ideas to make a tangible and useful
contribution to the field in which the innovation will occur
Critical Thinking CT  Use various types of reasoning (inductive, deductive, Analyzing what
etc.) as appropriate to the situation we’re doing
 Effectively analyze and evaluate evidence, “…because it’s
arguments, claims and beliefs and major alternative closer”
points of view
 Analyze how parts of a whole interact with each
other to produce overall outcomes in a complex
system (systems thinking)
 Synthesize and make connections between
information and arguments
 Reflect critically on learning experiences and
processes
Problem Solving PS  Interpret information and draw conclusions based on
the best analysis
 Solve different kinds of non-familiar problems in both
conventional and innovative ways
 Identify and ask significant questions that clarify
various points of view and lead to better solutions
Life and Career 
Skills
Flexibility and FA  Adapt to varied roles, jobs responsibilities, schedules (when running
Adaptability and contexts out of time)
 Work effectively in a climate of ambiguity and “…so let’s go to
changing priorities #3 instead of
 Deal positively with praise, setbacks and criticism; #5”
Incorporate feedback effectively
 Exercise flexibility and willingness to be helpful in
making necessary compromises to accomplish a
common goal
 Reflect critically on past experiences in order to
inform future progress
View failure as an opportunity to learn; understand
that creativity and innovation is a long-term, cyclical
process of small success and frequent mistakes

107
Initiative, Self- ISP  Set goals with tangible and intangible success Thinking about
Direction and criteria, even in the face of obstacles and competing what to do
Productivity pressures
 Balance tactical (short-term) and strategic (long-
term) goals
 Prioritize, plan and manage work to achieve intended
results
 Monitor, define, prioritize and complete tasks without
direct oversight
 Multi-task
 Manage time and projects effectively
 Go beyond basic mastery of skills to explore and
expand one’s own learning and opportunities to gain
expertise
Leadership and LRA  Use interpersonal and problem-solving skills to
Responsibility and influence and guide others toward a goal
Accountability  Leverage strengths of others to accomplish a
common goal
 Inspire others to reach their very best via example
and selflessness
 Demonstrate integrity and ethical behavior in using
influence and power
 Act responsibly with the interests of the larger
community in mind
 Be accountable for results

108
 APPENDIX H
NGSS, 21st Century Skills, CCSS Math Practices Cross-referenced

109
 NGSS
1. Asking questions (for
science) and defining
problems (for engineering)
6. Constructing explanations
(for science) and designing
solutions (for engineering)
7. Engaging in argument
from evidence
8. Obtaining, evaluating, and
communicating information
P21 CCSS MP
• Communication and CCSS.Math.Practice.MP1
Collaboration Make sense of problems and
• Critical Thinking and persevere in solving them.
Problem Solving CCSS.Math.Practice.MP2
Reason abstractly and
quantitatively.
• Communication and CCSS.Math.Practice.MP1
Collaboration Make sense of problems and
• Critical Thinking and persevere in solving them.
Problem Solving CCSS.Math.Practice.MP2
• Creativity and Innovation Reason abstractly and
• Productivity and quantitatively.
Accountability CCSS.Math.Practice.MP6
• Leadership and Attend to precision.
Responsibility CCSS.Math.Practice.MP8
Look for and express
regularity in repeated
reasoning.
• Communication and CCSS.Math.Practice.MP3
Collaboration Construct viable arguments
• Flexibility and Adaptability and critique the reasoning of
• Initiative and Self-Direction others.
• Leadership and CCSS.Math.Practice.MP6
Responsibility Attend to precision.
• Communication and CCSS.Math.Practice.MP3
Collaboration Construct viable arguments
• Initiative and Self-Direction and critique the reasoning of
• Leadership and others.
Responsibility CCSS.Math.Practice.MP6
Attend to precision.
CCSS.Math.Practice.MP8
Look for and express
regularity in repeated
reasoning.
110
 APPENDIX I
Orienteering Map

111
112
 APPENDIX J
Orienteering Score Card

113
114
 APPENDIX K
Control Markers, Punch, and Keychain

115
116
 APPENDIX L
Class Visit Lesson

117
Introduction to Orienteering

I. Why are we doing this? – Nature of the research (non-formal learning, P21 skills –

problem solving, critical thinking, creativity and innovation, communication and

collaboration, flexibility and adaptability, productivity and accountability, and

leadership and responsibility).

II. What are we doing?

A. Orienteering as a sport (very popular in Europe)

B. Fun problem solving activity involving strategy and teamwork

C. Great for those who like to hike, run cross country or on trails, wander in the

woods

D. Video clip- https://youtu.be/LRJgU0I5E

III. How are we doing it?

A. Map (ours will be a recent aerial photo of the school and grounds with “control

markers” indicated)

B. Discuss and demonstrate how to “orient” a map by turning it the appropriate

direction based on landmarks

C. Control markers (show example) – explain points, time constraints and the need

for a flexible strategy

D. Teams (pairs) – teacher will use random group generator; two teams in each class

will be filmed and interviewed afterwards (explain how GoPro® will work)

E. Appropriate dress – running outside in cold weather – bring hat, gloves, etc.

118
 APPENDIX M
Pre-course Briefing

119
 Object of the game is to get the most points in the least amount of time

 Time limit is 35 minutes – late penalty is 15 points/minute. It never pays to be late!

 This is a team activity – you must stay with your partner the entire time.

 15 control markers (CM) – you may find them in any order you choose (show CM

example); having a strategy before you start is highly recommended!

 Punch your scorecard in the appropriate box at each CM; leave the CM where you found

it! (show punch and pass out scorecards)

 Point values for each CM are listed at the bottom of the map

 Bonus points – “Peaks of Otter” keychain is hanging at one of the “10 point” CMs; the

first team to find that CM should take the keychain and bring it back to the finish for an

extra 40 points (only 1 keychain is on the course)

 You may cross the road ONLY at the designated “road crossing” (marked on your map)

 Off limit areas = ALL pavement except for designated “road crossing,” including parking

lots!

 You MAY travel on/through woods, fields, and sidewalks!

 Maps will be distributed once we get outside; finish line is in the same location as the

start

 Be safe – watch for cars, mud, and low branches!

 Have fun

120
 APPENDIX N
Orienteering Post-Interview Protocol

121
Introduction:

You did a great job with the orienteering experience today. I hope you enjoyed it! I’d like to ask
you a few questions about your experience. It should only take about 15 minutes. I’m video
recording it so we can discuss your answers without taking the extra time to write everything
down.

Questions:

1. Did your team begin with a strategy or develop one along the way?
Probes: What was it?
Why was it helpful to have a strategy?
How did the point values of control markers influence your strategy? (including
bonus points)
A. Initiative, Self-Direction, and Productivity (ISP)
B. Critical thinking (CT)

2. Did your strategy change during the course?

Probes: What changed? Why?
A. Flexibility and Adaptability (FA)
B. Problem Solving (PS)
C. Initiative, Self-Direction, and Productivity (ISP)

3. Were control markers where you originally thought they were? What did you do when
they were not?
Probes: Why?
Did this work? Why/why not?
A. Flexibility and Adaptability (FA)
B. Critical Thinking (CT)
C. Problem Solving (PS)

4. Did all team members agree on decisions throughout the experience?

Probes: What did you do when they did not?
Did it work? Why/why not?
A. Collaboration (2)
B. Leadership, Responsibility, and Accountability (LRA)

5. What do you think were your strengths?

Probe: Why do you think this was something you were particularly good at?
A. Leadership, Responsibility, and Accountability (LRA)
B. Flexibility and Adaptability (FA)

122
 C. Collaboration (2)

6. Which of your strategies or decisions were especially inventive or creative?

Probe: Did you come up with a way of doing something that was different than
what others did?
A. Creativity and Innovation (CA)

7. What did you learn or figure out as you went?

A. Flexibility and Adaptability (FA)
B. Critical Thinking (CT)

8. What would you do differently if you were to do this same course again?
A. Flexibility and Adaptability (FA)
B. Leadership, Responsibility, and Accountability (LRA)

9. Tell me the process you used to problem solve when you had difficulties.
A. Problem Solving (PS)
B. Critical Thinking (CT)
C. Creativity and Innovation (CA)
D. Flexibility and Adaptability (FA)

10. How could this experience help you solve a different kind of problem at home or at
school?
A. Initiative, Self-Direction, and Productivity (ISP)

11. Is there anything else you want to tell me about the experience?

123
 APPENDIX O
Finalized Coding Scheme with Example Utterances

124
Coding Scheme 1
CATEGORY CODE DEFINITION Example Utterances
Communication/ 1  Articulate thoughts and ideas effectively using Just saying “we”
Social Skills oral, written and nonverbal communication doesn’t mean
skills in a variety of forms and contexts collaboration- it’s
 Listen effectively to decipher meaning, communication unless
including knowledge, values, attitudes and there is some back
intentions and forth agreement
 Use communication for a range of purposes
(e.g. to inform, instruct, motivate and
persuade)
 Communicate effectively in diverse
environments (including multi-lingual)
 Know when it is appropriate to listen and when
to speak
 Conduct themselves in a respectable,
professional manner
 Present oneself professionally and with proper
etiquette
 communicate new ideas to others effectively
Collaboration 2  Be open and responsive to new and diverse Question and answer
perspectives; incorporate group input and related to the problem
feedback into the work
 Assume shared responsibility for collaborative “I’ll check the trees,
work, and value the individual contributions you check the door”
made by each team member
 Work effectively in diverse teams Exchanging
 Respect cultural differences and work information to come to
effectively with people from a range of social a decision
and cultural backgrounds
 Respond open-mindedly to different ideas and Talking about ideas
values
 Collaborate and cooperate effectively with
teams

125
Coding Scheme 2
CATEGORY CODE DEFINITION Example Utterances
Learning and
Innovation Skills
Creativity and CI  Use a wide range of creation techniques Out of the ordinary
Innovation  Demonstrate originality and inventiveness in ideas
work and understand the real world limits to
adopting new ideas “follow my voice”
 Create new and worthwhile ideas (both
incremental and radical concepts) Making a new idea
Act on creative ideas to make a tangible and based on observations
useful contribution to the field in which the
innovation will occur
Critical Thinking CT  Use various types of reasoning (inductive, Analyzing what we’re
deductive, etc.) as appropriate to the situation doing
 Effectively analyze and evaluate evidence,
arguments, claims and beliefs and major “…because it’s closer”
alternative points of view
 Analyze how parts of a whole interact with The thinking that
each other to produce overall outcomes in a informs the plan
complex system (systems thinking)
 Synthesize and make connections between
information and arguments
 Reflect critically on learning experiences and
processes
Problem Solving PS  Interpret information and draw conclusions When they actually
based on the best analysis solve a problem,
 Solve different kinds of non-familiar problems usually CT first
in both conventional and innovative ways
 Identify and ask significant questions that
clarify various points of view and lead to better
solutions
Life and Career 
Skills
Flexibility and FA  Adapt to varied roles, jobs responsibilities,
Adaptability schedules and contexts “(since time is almost
 Work effectively in a climate of ambiguity and up)…so let’s go to #3
changing priorities instead of #5”
 Deal positively with praise, setbacks and
criticism; Incorporate feedback effectively When they change
 Exercise flexibility and willingness to be helpful plan because of
in making necessary compromises to something unexpected
accomplish a common goal
 Reflect critically on past experiences in order
to inform future progress
View failure as an opportunity to learn;
understand that creativity and innovation is a
long-term, cyclical process of small success
and frequent mistakes

126
Initiative, Self- ISP  Set goals with tangible and intangible success Thinking about what to
Direction and criteria, even in the face of obstacles and do
Productivity competing pressures
 Balance tactical (short-term) and strategic A plan
(long-term) goals
 Prioritize, plan and manage work to achieve Monitoring- “We’re
intended results running out of time…”
 Monitor, define, prioritize and complete tasks
without direct oversight
 Multi-task
 Manage time and projects effectively
 Go beyond basic mastery of skills to explore
and expand one’s own learning and
opportunities to gain expertise
Leadership and LRA  Use interpersonal and problem-solving skills to Trying to influence
Responsibility and influence and guide others toward a goal
Accountability  Leverage strengths of others to accomplish a Being aware of
common goal teammate’s needs
 Inspire others to reach their very best via
example and selflessness Taking initiative
 Demonstrate integrity and ethical behavior in
using influence and power Compromising
 Act responsibly with the interests of the larger
community in mind Concern for team
 Be accountable for results member

127
 APPENDIX P
Coded Pilot Study Transcript

128
 (Walking out of the school) Alright, you ready to move this
V1- 10:15A 2 2 2 2 1 1 1 100% ISP ISP ISP ISP 1 1 1 100%
through pretty quick? goal
B Yeah, as soon as we get our map I have a good strategy. 1 2 2 2 0 1 1 67% ISP PS ISP PS 0 1 0 33% plan
A Alright 1 2 2 2 0 1 1 67% 0 1 1 1 100%
B But I don't know beacause I don't know how far out we'll be 1 1 1 1 1 1 1 100% CT CT CT CT 1 1 1 100% analyzing potential plan
Well from what he said they're planted on every single side of
A 1 1 1 1 1 1 1 100% CT ISP ISP CT 1 0 0 33%
the property except the front parking lot. the thinking behind the plan
B This parking lot's off limits 1 1 1 1 1 1 1 100% ^ ISP ISP ISP 0 1 1 67% all these are the same idea
A Yeah, except for the front parking lot 1 2 1 1 1 0 1 67% ^ ISP ISP ISP 0 1 1 67%
B Nobody's gonna be allowed there 1 1 1 1 1 1 1 100% ^ ISP ISP ISP 0 1 1 67%
A Right. And no hopping fences either, so 1 2 1 1 1 0 1 67% ^ ISP ISP ISP 0 1 1 67%
No getting in neighbors either. No getting in any houses.
B A:/Yeah (laughs) no burglary. Burglary is off limits for this 1 1 1 1 1 1 1 100% ^ ISP ISP ISP 0 1 1 67%
game. B:/It's off limits for everything cause it's illegal
Ha, yep. (walking around the school) I'm pretty sure I saw
A 1 1 1 1 1 1 1 100% CT ISP CT CT 1 0 1 67%
number zero was somewhere on here. Actually. analyzing map
OK, this is our start/finish line right here. Ok, the opening in
the fence. Ok so when you come back when you're done you
come back to here. Remember if you're planning and want to
use the bleachers, we've got bees on this side, we didn't see
any bees over here, so you can use the track, you can sit down,
you can use the wall, whatever. OK, remember a couple of
important things. You're looking where the circles are. This is a
map, and a map is like you're flying overhead like a bird. Well I
should say this is a little bit, this is an aerial photo. It's not
quite exactly a map. The good thing is things are really
recognizable like you can see the track and the football field.
The bad thing is this was taken when leaves were on the trees
and there's shadows. Sometimes it's hard to see what's a
F1 0 1 1 1 0 1 1 1
shadow and what's a treeline. So kind of keep that in mind.
Remember, there's a target circle where that control marker is.
It may be even a little bit outside that in some cases. So start
there and kind of work your way out. Don't work your way out
halfway across the map because that's too far but go a little bit
further out from that circle if you think it makes sense. If you
start toward the middle, see what makes sense, think about it.
Try and think like me, if you were hiding a control marker
where would you hide it. A really really important thing too is
do not disturb the control markers. All you do is punch. You
don't pull on them, you don't untie them, you don't move
them.You just take them and "pt" (makes punching motion)
and move on. (Instructions given by facilitator)
F2 It's just like a hole punch, like in school 0 1 1 1 0 1 1 1
F1 OK, uh, everbody know what they're looking for? 0 1 1 1 0 1 1 1
students
yeah 0 1 1 1 0 1 1 1
Some are this big (shows with fingers). Some are this big
(shows with hands). Nothing is hidden hidden. Sometimes you
F1 may have to kinda look behind a tree or underneath the leaves 0 0 0 0 0 1 1 1
but it's not like look under, turn over rocks or anything like
that. Does it make sense?
B Yep 0 1 1 1 0 1 1 1
OK. So as soon as you get a map, which I'm gonna start to pass
F1 0 1 1 1 0 1 1 1
out
Alright, (passing out maps) so the first thing, you're probably
F2 0 1 1 1 0 1 1 1
gonna go come up with a strategy
A Let's see. 1 1 1 1 1 1 1 100% 0 1 1 1
F1 Time has started 0 1 1 1 0 1 1 1
A Time has started! 1 1 1 1 1 1 1 100% ISP ISP ISP ISP 1 1 1 100% aware of time limitation
OK. We are here ok? (pointing on map) 2 1 2 0 1 0 33% CT CT 1 0 0 33% CODE1 CODE2 analyzed current location
B OK A:/Oh, wait, there's one right over there I say we, 2 2 2 2 1 1 1 100% CT CT CT CT 1 1 1 100% 86% 69% analysis
oh no, wait, let's flip it upside down so say we were right here
2 2 2 2 1 1 1 100% CT CT CT CT 1 1 1 100%
instead, analysis
it would actually be over there (both point). Come on! 2 2 2 0 1 1 67% PS PS 1 0 0 33% drew a conclusion
Number 7, I think it was by the treeline right? (running) 2 2 2 0 1 1 67% CT CT CT CT 1 1 1 100% analysis
B Yes (running) 1 2 2 2 0 1 1 67% 0 1 1 1 100%
OK, so remember, there was 45 minutes, 9 markers, therefor
A 1 1 2 1 1 1 0 67% ISP ISP ISP ISP 1 1 1 100%
we've got about five minutes to get each one. the plan
Since we got off right here, we got a pretty good start. OK 1 2 2 0 0 1 33% ISP ISP 1 0 0 33% evaluation of how the plan is progressing
B Stop I want to make sure we did it right 2 2 2 2 1 1 1 100% CT CT CT CT 1 1 1 100% analysis of past decision
A See where we are. 1 2 2 2 0 1 1 67% CT CT CT CT 1 1 1 100%
B It's upside down completely 1 1 2 1 1 1 0 67% CT PS CT CT 1 0 1 67%
No, that's. I turned it upside down because we were heading
the, right there (pointing to the map) so we were here since
A 2 2 2 2 1 1 1 100% CI PS PS PS 0 1 1 67%
we were heading that way. We turn it upside down so we can,
in a sense, have the right direction B:/ Oh! A:/ Because creative approach?
B Let's look 2 2 2 2 1 1 1 100% ISP CT ISP 1 0 0 33% MANAGING
A It's kinda, it seemd like an opening in the trees. 1 1 1 1 1 1 1 100% CT CT CT CT 1 1 1 100% analysis
Remember it'd be this way (pointing) considering we're facing
2 2 2 0 1 1 67% PS PS PS 1 0 1 67%
the opposite direction. evaluation?
Oh wait, did we get an orange card? 2 1 1 1 0 1 1 67% ISP ISP 1 0 0 33% remembered the objective
B (looks back) No! Go back 1 2 2 2 0 1 1 67% ISP ISP FA ISP 1 1 0 67%
Yeah (starts running). Oh wait, we can't split up.! Oh thank
A 2 1 2 2 1 0 1 67% ISP LRA FA LRA 0 1 0 33%
goodness! a requirement?
F2 (Handing him score card) You left without a score card. 0 1 1 1 0 1 1 1
Oh good. (sees other students) Oh, what do you know, I guess
A 1 1 1 1 1 1 1 100% FA PS PS 0 0 1 33%
that might be it. thinking about changing plan?

B
(Seeing crowd) No, let's go and try and find another one and
2 1 2 2
129
1 0 1 67% ISP PS ISP ISP 1 0 1 67%
get this one on the way back. one plan
A No let's get it now, we're right here. 2 1 1 1 0 1 1 67% ISP LRA LRA LRA 0 1 1 67% a different plan
B No number 2 is over there 1 1 1 1 1 1 1 100% CT LRA LRA LRA 0 1 1 67% Code1 Code2 thinking behind doing 1st plan
A Number 2 is over where? Let's see. 2 2 2 2 1 1 1 100% CT CT CT CT 1 1 1 100% 76% 68%
 (Looking at the map and pointing) Way over there. Number 2
B 2 1 2 2 1 0 1 67% PS LRA PS 0 1 0 33%
(starts running) Let's go over there, the plan
we can get this one on the way back. 1 2 2 0 0 1 33% ISP ISP ISP 1 0 1 67%
Alright (running) we'll get number 2 and then we'll come back
A 1 2 2 2 0 1 1 67% ^ LRA FA FA 0 0 1 33%
for this one B:/Yep
Like I said, we should have 5 minutes for each one, we have
A 1 1 1 1 1 1 1 100% ISP ISP ISP ISP 1 1 1 100%
plenty of time.
B May I see the map? (Gets the map) OK, so, we're right here. 1 2 2 2 0 1 1 67% CT CT CT 1 0 1 67% location
Two should be…right back there (pointing right and continuing
A 1 1 1 0 1 1 67% PS PS PS 1 0 1 67%
to run) location of 2
B You alright Iain? B:/Yep, getting in shape for soccer 2 2 2 2 1 1 1 100% LRA LRA 0 0 1 33%
(Laughs) Alright, (stops running) so we have the opening right
A 1 1 2 1 1 1 0 67% CT CT CT CT 1 1 1 100%
here (points).
I think it might be just behind that tree (points to the map)
1 2 2 0 0 1 33% PS PS PS PS 1 1 1 100%
B:/Alright
Alright (looking at map) The problem is I don't know where
the… see the property line's right there. Let's see, so I guess
V2- 0 A start looking for one of those orange markers (looking) so it's 1 1 1 1 1 1 1 100% CT CT CT CT 1 1 1 100%
anywhere around these. Just around this treeline (pointing to
map.)
B Alright. (taking map) A/:So is this 2 2 1 0 0 33% CT CT CT CT 1 1 1 100%
B It's right on the property line (pointing) It's probably up here. 1 1 1 1 1 1 1 100% PS PS PS PS 1 1 1 100%
Yeah, and it's orange and white, right? B/: yep Putting these two together
A 1 2 2 0 0 0 0% CT CT CT CT 1 1 1 100% shows collaboration better
AD- (DONE)
A Some are small, some are bigger than the others. Ok. 1 1 1 1 1 1 1 100% CT CT CT CT 1 1 1 100%
B Where is it? 1 1 1 1 1 1 1 100% PS CT CT 0 0 1 33%
A So it can be up here? 1 1 1 1 1 1 1 100% CT PS CT CT 1 0 1 67%
It's the op… (pausing) think it's right over here. The map says
B 1 1 1 1 1 1 1 100% PS CT CT CT 0 1 1 67%
anyway.
Here, they're all gathered over there, let's see what they
A 1 1 1 1 1 1 1 100% CI LRA CI CI 1 0 1 67%
found.
B No. That's number 1 (pointing) out on the baseball field. 1 1 1 1 1 1 1 100% CT LRA LRA CT 1 0 0 33%
A No (pointing) there's a group of people right down there. 1 1 1 1 1 1 1 100% CI CT CI CI 1 0 1 67%
Wait a minute, I could've sworn I just saw a flash of orange.
B 1 1 1 1 1 1 1 100% FA CT FA FA 1 0 1 67%
Somewhere
Hey Iain. B:/ Yeah A:/(Pointing) Remember you said they could
be outside the circle. You have a group of people hiding right Remember, you
A 2 1 2 2 1 0 1 67% CT CI CI 0 0 1 33%
there and they're just leaving the area. I say we search it said…referencing what other
(running that way). person said
A There it is, right there B:/Aw! 1 1 1 1 PS PS PS PS 1 1 1 100%
Great, alright this is number 2 right? B:/Yep A:/ 2 (punches Putting these two together
A 1 2 2 2 0 1 1 67% CT CT CT CT 1 1 1 100%
scorecard) shows collaboration better
B Aw it's worth some points, awesome! 1 2 1 1 0 0 33% LRA LRA LRA LRA 1 1 1 100%
I say we head for 5, or do we go that way? Asking for other person's
A 2 2 2 2 1 1 1 100% ISP CT ISP ISP 1 0 1 67%
opinion
B (Pointing) Let's go this way for 7. 1 2 2 1 1 0 0 33% ISP ISP ISP ISP 1 1 1 100%
Oh, right, 7 not 5. (Running) alright, we know it's right over
there (pointing). (Stops running) I'd say we have time, let's
A 2 1 2 2 1 0 1 67% ISP ISP ISP ISP 1 1 1 100%
look at the map on where we'd be heading next. Alright Iain,
can I see the map? Should we separate this? LM
That's the other thing A:/Right, so remember, 7's over there.
(Pointing to map) Number 1, yes, is over by the baseball field.
Right over there. And number 6, further up the property line
from there. Hmmm. Alright, it was along over there. Number 5,
B 1 2 2 2 0 1 1 67% ISP ISP ISP ISP 1 1 1 100%
it's on the other side of the property, on the treeline that's
heading to the school. After that we have to run all the way
around the building . We can stop at 13 and 4 then. Get zero get
three and run back
Which ones are worth double points, which are the ones that
B 2 2 2 2 1 1 1 100% CT ISP CT CT 1 0 1 67% Code1 Code2
are worth double?
A Uh which ones are worth double, the ones with the circles? 2 2 2 2 1 1 1 100% CT ISP CT CT 1 0 1 67% 77% 79%
B 6 1 2 1 2 0 1 0 33% CT ISP CT CT 1 0 1 67%
A 6 and 2 are the circled ones 1 2 1 2 0 1 0 33% CT ISP CT CT 1 0 1 67%
B Yes, we've already punched A:We already got 2 1 2 2 2 0 1 1 67% CT ISP CT CT 1 0 1 67%
Oh man! Awww 6 is over by the teamline A:Man! Alright, we'll
B 1 2 1 1 1 0 1 67% CT ISP CT CT 1 0 1 67%
have to
We can head down to the baseball field and then we can head
B 2 1 2 2 1 0 1 67% ISP ISP ISP ISP 1 1 1 100%
there from here
Yeah, actually I think we head there because then we'd be able
A to head down to the baseball field from there. Alright, anyway 2 2 2 2 1 1 1 100% ISP ISP ISP ISP 1 1 1 100%
we've got to find this, come on! (Runs)
We should've punched it earlier. 1 1 1 1 1 1 1 100% LRA LRA LRA LRA 1 1 1 100%
B Let me check the map. 1 2 1 1 1 0 1 67% ISP CT ISP ISP 1 0 1 67%
A Aw, we walked right past it, it's back there 1 2 1 1 1 0 1 67% PS ISP PS PS 1 0 1 67%
B I'll get it! (running) Punch it real good! 1 2 2 2 0 1 1 67% FA ISP ISP ISP 0 1 1 67%
A (Laughs) alright (punches card)Number 7, alright. 1 1 1 1 1 1 1 100% ISP ISP ISP ISP 1 1 1 100%
B Let's head over to the baseball field. Right over here 2 1 2 2 1 0 1 67% ISP LRA ISP LRA 0 1 0 33%
Right (running) we probably should've punched 7 the first
A 2 2 2 2 1 1 1 100% LRA CT LRA LRA 1 0 1 67%
thing. You good Iain? B:/Yeah
Alright. So what number's over there? By the baseball field?
A 2 2 2 2 1 1 1 100% CT CT ISP CT 1 1 0 67%
B:/ 1
1. Ok, (pointing) and then number 6, right over there by the Anita- Might want to put
treeline. these together since they are
A 1 2 2 2 0 1 1 67% PS CT CT CT 0 1 1 67%
essentially representing the
same code DONE

130
 5's over there too. Somewhere. A:/Well B:/Somewhere in
B 1 2 2 2 0 1 1 67% CT CT CT CT 1 1 1 100%
there. In that treeline right there.
Alright, I say we head up here first and so then we can just
head down and hit to these two. Instead of going here then up
A 2 2 2 2 1 1 1 100% ISP ISP ISP ISP 1 1 1 100%
and then back down. (Runs) Let's go around the ditch right
here. Good.
Well I think they're ahead of us. In fact I think they all are. Only
2 2 2 2 1 1 1 100% LRA LRA LRA LRA 1 1 1 100%
got two. Ooh. Which treeline is it?
B 6 is over there 1 2 1 2 0 1 0 33% PS CT CT CT 0 1 1 67%
A Over there? (pointing) 2 2 2 2 1 1 1 100% CT CT CT CT 1 1 1 100%
B 5 is over there 1 2 1 2 0 1 0 33% PS CT CT CT 0 1 1 67% ASKING AND ANSWERING
(Pointing) 5 is over there. OK. Let's see the map. Here, can I see A-Might want to separate this
A the map for a second? It'll free up some space for your hands. 2 2 2 2 1 1 1 100% LRA LRA 0 0 1 33% b/c there seems to be a lot
going on in this chunk
Alright. (looking at map) We've got 6 over there by that first
house. It's just gonna be right up here. (Turning map) yeah, 1 1 1 1 1 1 1 100% CT CT CT CT 1 1 1 100%
telephone's right there, right there (pointing to map) alright.
Come on Iain you can do it! 2 1 2 2 1 0 1 67% LRA LRA LRA LRA 1 1 1 100%
Let's see. Alright so it'll be right up here along these trees
1 1 1 0 1 1 67% CT CT CT 0 1 1 67%
(looking around) Probably behind them.
You search these three, I'll search these three. B:/ Alright 1 2 2 0 0 1 33% ISP ISP ISP 0 1 1 67% A-Could also be CI
Technically we won't be splitting up, we're still in the same
6:00 A 1 2 2 2 0 1 1 67% CI CT CI CI 1 0 1 67%
area.
Orange, orange, orange. Man. Come on! Alright. We should've
gotten that 7 earlier. It would've saved us the trip back. That 2 2 2 2 1 1 1 100% ISP LRA ISP 1 0 0 33%
cost us right there. Some valuable time.
Iain did you find anything? Come on, where is it? (turns map)
It's gotta be right along here. Hey Iain, come back a little bit 2 2 0 0 1 33% LRA LRA 0 0 1 33%
this way. Unless you found it. Did you find it?
No. Oh, here it is. It's a small one. Oh, but this is 5, this is 5.
B 1 2 1 1 1 0 1 67% CT PS PS PS 0 1 1 67%
A:/Alright. (Punches card) Oop, puncture it. B:/Here we go
A Alright, so wait, we're heading back this way now 1 1 1 1 1 1 1 100% CT ISP CT CT 1 0 1 67%
No. That was 5, 6 is over here. Oh, wait no. Wait, what the
B 2 1 2 2 1 0 1 67% CT CT CT CT 1 1 1 100%
heck? Is this 6? Or 5?
That's 6. Man, I punched the wrong one. (Punching the other
A 1 1 1 1 1 1 1 100% CT ISP ISP ISP 0 1 1 67%
number)
B Doesn't matter, doesn't matter 1 2 1 1 1 0 1 67% ISP FA LRA FA 0 1 0 33%
A It does, I'll have to…I'll repunch this one. Come on 1 2 1 1 1 0 1 67% ISP LRA LRA LRA 0 1 1 67%
B We have videotaped proof 1 2 1 1 1 0 1 67% CI ISP CI CI 1 0 1 67%
Yeah we have proof. Alright, we'll head down that way, we can
hit the one on the baseball field and then we have to go
A 2 2 2 2 1 1 1 100% ISP ISP ISP ISP 1 1 1 100%
around the building. Come on, we're the only ones still over
here! (running)
Yeah they're all over there. So I think we're the only ones going
B 1 1 1 1 1 1 1 100% CT CT ISP CT 1 1 0 67%
around the building
Yeah we're the only ones on this side of the building, which
A 2 1 1 1 0 1 1 67% CT CT CT CT 1 1 1 100%
means we're behind!
It's on the..woops, it's on this edge, this round, right here
(pointing). Right along here. Alright! Found it! Aw man, last
time I went like down this I was hiking on the trail by my house 1 1 1 0 1 1 67% PS PS PS 0 1 1 67%
and I fell over and almost broke my ankle. Huh. Wooh.
(punching card) Alright, I got 5
B 1 is over here, see that bush (unintelligable) 1 1 1 1 1 1 1 100% CT ISP CT CT 1 0 1 67%
A I'll staple it right next to it. 2 1 1 1 0 1 1 67% FA ISP FA FA 1 0 1 67%
Alright, yeah, let's go to the baseball field. Man, we shoulda
2 1 2 2 1 0 1 67% ISP ISP ISP ISP 1 1 1 100%
got…
B I have an idea where that is. (unintelligable) 1 1 1 1 1 1 1 100% CT CI CT CT 1 0 1 67%
Is that it right there? More trees. Alright, so where are we right
now. There's that. No it's on that side. It's on that side of the
A 2 2 2 2 1 1 1 100% CT CT CT CT 1 1 1 100%
field. Or at least that's what the map says. Still, does that
orange thing right there look like anything to you?
B Let me check. Gatorade bottle. 1 2 2 2 0 1 1 67% PS PS PS 0 1 1 67%
Gatorade bottle. Ok. (Looking around field) There's the
A pitcher's mound, kind of bleacher. Yeah keeping looking 1 2 2 2 0 1 1 67% CT ISP CT CT 1 0 1 67%
around right here (pointing)
There it is on the trash can! A:/On the trash can? Alright! Woa.
B 1 2 1 1 1 0 1 67% PS ISP PS PS 1 0 1 67%

(Standing behind fence) Pass me the scorecard through the

B 1 2 2 2 0 1 1 67% CI ISP CI CI 1 0 1 67%
thing so I can punch it.
We can't go throught the fence. Even if it is a scorecard I don't
A 1 2 2 2 0 1 1 67% LRA LRA LRA LRA 1 1 1 100%
wanna risk it
Alright. Normally I would, but we're on video here (laughts)
##### B 2 2 1 2 1 1 0 67% CT LRA LRA LRA 0 1 1 67%
Alright what number is this, 1?
1, alright 1 right there (holds it while Iain punches it) Alright,
A there's 1 B:/Oh God, no. (Drops control marker in trash can) 2 2 2 2 1 1 1 100% CT LRA LRA LRA 0 1 1 67%
A:/Oh, that's not good
(Takes it out, lays it over edge of trash can) There we go, sorry.
B That's sneaky. Putting it in the trash A:/Alright B:/I bet 1 1 1 1 1 1 1 100% CT CT LRA LRA 0 0 1 33%
somebody moved it
A We got everything on this side of the route 1 1 1 1 1 1 1 100% PS ISP ISP PS 1 0 0 33% INTERPRET INFO AND DRAW
Let me see the map. They're rounding the school so we should
B 2 1 2 2 1 0 1 67% CI ISP CI CI 1 0 1 67%
go up there (Takes off running)

131
 Hold on, here look at the map, we gotta go, if we go for 4, we
A 2 1 2 2 1 0 1 67% ISP ISP ISP ISP 1 1 1 100%
can get 4, 0, 3, come back and get 13.
B Alright, so where are we going first? 2 2 2 2 1 1 1 100% ISP CT ISP ISP 1 0 1 67%
Alright, we've gotta head all the way around the school right
A 2 1 1 1 0 1 1 67% ISP ISP ISP ISP 1 1 1 100%
now.
Front parking lot's off limits. I don't know if they would count
2 2 2 1 0 1 67% ISP ISP ISP 1 0 1 67%
the sidewalk. Alright, come on. (running) Have we got time?
F1 You've got 30 minutes 0 1 1 1 0 1 1 1
30 minutes, alright (looks at scorecard) 1, 2, 3, 4, alright, we're
12:00 A 1 2 1 1 1 0 1 67% ISP ISP ISP ISP 1 1 1 100%
ahead of schedule.
Come on! Let's head behind him. Sorry sir! (runs around man
weed eating). Alright. (walks and looks at map) Actually,
alright, we can go for 13 here (pointing to map) which is right
2 2 0 0 1 33% LRA LRA 0 0 1 33%
around here. Get 0, 3, come back for 4 and then a final sprint
back. (Talking to himself) Just ignore it, ignore the pain.
Alright. 13 is around here.
B I think I see it. Over here! (Running) 1 1 1 1 1 1 1 100% CT CT CT CT 1 1 1 100%
A Good eye, good eye! 1 2 2 2 0 1 1 67% LRA LRA LRA LRA 1 1 1 100%
No! It's an orange cone! A: Aww, come on! Curse you triangle
B 1 1 1 1 1 1 1 100% CT FA CT CT 1 0 1 67%
thingama whatsit!
B I think it's a square. 1 1 1 1 1 1 1 100% CT CT CT CT 1 1 1 100%
A It's a square? It says it's right around here. B:/Yep 1 1 1 1 1 1 1 100% CT CT CT CT 1 1 1 100% CODE 1 CODE 2
A Can we… anywhere around this lunch area. 1 1 1 1 1 1 1 100% CT CT CT CT 1 1 1 100% 77% 76%
Hey you don't think they'd be sneaky enough put it right next
B 2 1 2 2 1 0 1 67% CI CT CI CI 1 0 1 67%
to the 13 numbered door
Alright, they wouldn't do that. Let's head around back. Or do
A 2 1 2 2 1 0 1 67% CI ISP ISP ISP 0 1 1 67%
you want to check the door? B:/(unintelligable)
Alright, I'll check the trees. We're still in the same area. (hears
A yelling) Did you find it? B:/No (unintelligable) Oh man 1 1 2 2 0 0 1 33% CT ISP ISP ISP 0 1 1 67%
(unintelligable)
A Oh! Found it, found it! 2 1 2 2 1 0 1 67% PS LRA CI PS 1 0 0 33%
Follow my voice. It's right here in the trees B/: Follow your
^ ^ ^ ^ 1 1 1 PS LRA CI CI 0 0 1 33%
voice?
A It's right here in the trees 1 1 1 1 1 1 1 100% PS PS PS PS 1 1 1 100%
B Oh sneaky it's behind the bush. A/: Alright (punching card) 1 1 1 1 1 1 1 100% CT PS CT CT 1 0 1 67%
B Where are we going next? 2 2 2 2 1 1 1 100% ISP ISP ISP ISP 1 1 1 100%
We head around the building to 0 and then to 3 and then the
A final sprint to 4 and back before anyone else. Let's go! (runs) 2 1 2 1 0 1 0 33% ISP ISP ISP ISP 1 1 1 100%
Because everyone else I think has already been around.
B No, I saw a team heading around. There was one team at 5 1 1 1 1 1 1 1 100% CT ISP CT CT 1 0 1 67%
A Alright (pointing at map) 0 is, alright, 4 is actually right there, 2 1 1 1 0 1 1 67% ISP CT ISP CT 0 1 0 33%
but we need to get that when we come back. That way we're
^ ^ ^ ^ 1 1 1 ISP CT ISP ISP 1 0 1 67%
not… going too far
B Yeah, then we can have a downhill sprint 2 2 2 2 1 1 1 100% ISP ISP ISP ISP 1 1 1 100%
A Alright, number 0's gonna be between the treeline 1 1 1 1 1 1 1 100% PS CT CT CT 0 1 1 67%
B Right here? 2 2 2 2 1 1 1 100% CT CT CT CT 1 1 1 100%
A Yeah I guess…so? Man! 2 2 2 2 1 1 1 100% CT CT CT CT 1 1 1 100%
This stupid fence. And it's locked too so I guess we're not
B 1 1 1 1 1 1 1 100% CT PS CT CT 1 0 1 67%
supposed to go there
It's right, it's right, according to the map, look around, maybe
like a tree or something. Other than that, it's right there on the
other side of the fence. We'll go back to… B/:I had a feeling
15:00 A 1 1 1 1 1 1 1 100% CT CT CT CT 1 1 1 100%
they were going to do something like that A:/Uhhh B/: Be right
around the fence where we can see it. Tease us. A:/Man B:/
Look at them!
Alright Iain it is over there and not on this side. Let's get 4 and
A 2 2 2 2 1 1 1 100% FA ISP FA FA 1 0 1 67%
we'll head back around.
B Yeah, we can go to 0 2 2 2 2 1 1 1 100% ISP ISP ISP ISP 1 1 1 100%
A We'll head to 3 and then to 0 2 2 2 2 1 1 1 100% ISP ISP ISP ISP 1 1 1 100%
B and back 1 2 2 2 0 1 1 67% ISP ISP ISP ISP 1 1 1 100%
A and then back around. 1 2 2 2 0 1 1 67% ISP ISP ISP ISP 1 1 1 100%
You know at first, I though 13 and 0 were inside the building.
B 1 1 1 1 1 1 1 100% PS CT PS PS 1 0 1 67%
There it is, right there.
B Punch it. I'll start to go around 1 2 2 1 1 0 0 33% LRA ISP ISP LRA 1 0 0 33%
A Ah, of course it's right in a pine tree (punches card) 2 1 2 1 0 1 0 33% CT PS PS PS 0 1 1 67%
Alright, 3 and 0 that's all we gotta get. Come on! (Runs) 1 1 1 1 1 1 1 100% CT PS ISP ISP 0 0 1 33%
We can't head in the building and we can't go in the parking
lot. The building is the solid thing so that means the sidewalk 1 1 1 1 1 1 1 100% CT CT CT CT 1 1 1 100%
must be on limits
Head down the hill. (unintelligable) A:/Yeah. Hurry they're
behind us! It's open, come on! Were you saying something or
B was that the other team? B:/What? A:/ Did you say anything? 1 1 1 1 1 1 1 100% ISP ISP ISP ISP 1 1 1 100%
B:/No A:/ Ok that was the other team. B:/ (unintelligable)

Oh my legs feel like I'm gonna fall off. He said the side parking
A 2 1 2 1 0 1 0 33% CT CT CT CT 1 1 1 100%
lot right here? GRIT
B Let's walk to the sidewalk 1 1 1 1 1 1 1 100% FA LRA LRA LRA 0 1 1 67% Making a decision best for
He said the side parking lot right here is on limits, therefor we
A must be allowed to hit the sidewalk. We just won't be able to 1 1 1 1 1 1 1 100% CT PS CT CT 1 0 1 67%
touch the pavement.
B From here on. 1 1 1 1 1 1 1 100% CT PS CT CT 1 0 1 67%
Yep, from here on. We can't touch any of the pavement past
A 1 1 1 1 1 1 1 100% PS PS PS PS 1 1 1 100%
that fire hydrant.
Alright, number 3, number 3 is over there, we'll pass number 3
^ 2 ^ ^ 1 0 1 ^ ISP ISP ISP 0 1 1 67%
on the first turn

132
 B We'll hit 3 on the way back 2 1 2 1 0 1 0 33% ISP ISP ISP ISP 1 1 1 100%
V3 A 3 is on the way, 3' s on the way. We've got 3 and then 0. 2 1 2 1 0 1 0 33% ISP ISP LRA ISP 1 1 0 67%
Alright, you ready? Got a second sprint coming? B:/ No 2 1 2 2 1 0 1 67% ISP ISP LRA LRA 0 0 1 33%
A Alright then 1 2 2 2 0 1 1 67% ^ LRA LRA ^ 1 0 0 GRIT
B But not a long one 1 1 1 1 1 1 1 100% ^ ^ ISP ^ 1 1 0
Alright. I hear them behind us. The rules for… B/:Think he's
already finished A/:Yeah, probably. I think he yelled done.
A 1 1 1 1 1 1 1 100% ^ ISP ISP ISP 0 1 1 67%
Anyway B/: We can come in third though cause someone's
probably heading back with them A:/Yeah
B Unless they have no idea how to get back to 0 1 1 1 1 1 1 1 100% CT CT CT CT 1 1 1 100%
(Laughs) Let's hope that stumbles others as much as it did us,
cause, and hopefully the…oh nope they're coming, let's speed
A up a little bit. We can just jog a little bit and number 3 will be 2 1 2 1 0 1 0 33% ISP ISP ISP ISP 1 1 1 100%
around this first corner. Let's hope we can find it quick.

Alright 3, found it. 2 1 1 1 0 1 1 67% PS PS PS PS 1 1 1 100%

Shhhh, shhh come this way, quiet, quiet. 2 1 2 1 0 1 0 33% LRA LRA LRA LRA 1 1 1 100%
B But where is 0? 1 1 2 2 0 0 1 33% CT CT CT CT 1 1 1 100%
(Whispering) It's this way, by the gate, remember? Quickly!
(Turns the corner) See? These guys were finding it too, right?
A 1 1 1 2 0 0 0 0% PS CT CT CT 0 1 1 67%
Alright, hello! Alright it's right around here. Can be behind the
tree, could be right under the treeline right there.
B You look that way, I'll look this way 2 1 2 1 0 1 0 33% LRA ISP LRA LRA 1 0 1 67%
Got it (looking at map) alright, 0, 0… (looking around) If
anything, I think it'd be on this tree (pointing). That tree right
A 1 1 1 1 1 1 1 100% CT CT CT CT 1 1 1 100%
there, or under it, right here. Oh lord, please don't let us
missed it on the other side of the fence here.
Found it! 1 1 1 1 1 1 1 100% PS PS PS PS 1 1 1 100%
Alright, you ready for the sprint back? (punches card) 2 2 0 0 1 33% LRA LRA 0 0 1 33%
B Hurry up! 1 1 1 1 1 1 1 100% ISP LRA LRA ISP 1 0 0 33%
Was that a whistle? B:/It's a train. A:/Is that what it was? We've
A 1 1 2 2 0 0 1 33% ISP ISP ISP ISP 1 1 1 100%
got 15 minutes. Come on! Don't touch the pavement
Hope I don't throw up all over the place. A:/Hey Ethan. Don't
worry, we'll sprint once we get to the other side of the
B building. I wouldn't drink much else or else you stomach will 2 1 2 2 1 0 1 67% ISP LRA LRA LRA 0 1 1 67%
fill up. Save that for the exhausting barf afterwards B:/Oh,
yeah, barf. A:/ (Laughs)
I was running in the Fincastle run and I was drinking a whole
load of water A:/Ooooh B/: And I threw up in a trashcan and
B then collapsed and just sat down A/: Huh, well this is school 1 1 1 1 1 1 1 100% ^ FA ^ FA 0 1 0 33%
property so don't throw up anywhere besides the trashcan
B:/The Fincastle run for me turned into a Fincastle walk past experience
(Laughs) Yeah. Alright, you ready for a sprint back? B:/(starts
A 2 2 2 2 1 1 1 100% LRA ISP LRA LRA 1 0 1 67%
running)
Right here on the sidewalk. Alright, push, right past them. You
can do it! Cut corners if you have to. We're allowed back on the
sidewalk- I mean the pavement. Let's cut through this right
here. Come on Iain, we're almost there, we're almost done!
A 2 2 2 2 1 1 1 100% LRA LRA LRA LRA 1 1 1 100%
Gee this feels like a mile run more then anything else. Hey,
right back up there, right back up there! Ah! Fourth place, we
just missed it. Fourth? Man! Cross the finish line. Come on Iain,
aw right there.
F1 34 minutes and 37 seconds! Awesome. You get them all? 1 1 1 1 1 1 CODE 1 CODE 2 40%
Yep, we got them all, we accidentally got the wrong stamp on a
A 1 1 1 1 1 1 1 100% LRA LRA LRA LRA 1 1 1 100%
different one. So we restamped 5. Alright! 76% 77%
77% 1 1 75%
CI 9 5%
CT 66 35%
PS 23 12%
FA 6 3%
ISP 54 29%
LRA 29 16%
viable nonfrm are characterized by programs that..

COMMUNICATION TO 77
33

COLLABORATION TO 77
31

ALL: Comm Collab

95 92
51% 49%

133
APPENDIX Q

Pilot Study

134
 Pilot Study

To ensure the research design, data collection, and data analysis procedures were both

appropriate and effective for answering the proposed research questions a pilot study was

conducted. The following research questions guided the pilot study.

Research Questions

RQ#1: What is the relationship between a non-formal learning experience and acquisition of

the following 21st century skills:

1. Learning and Innovation

A. Creativity and innovation

B. Critical thinking

C. Problem solving

2. Life and Career

A. Flexibility and adaptability

B. Initiative, self-direction and productivity

C. Leadership, responsibility and accountability

3. Socio-Cultural

A. Communication

B. Collaboration

RQ#2: What characterizes a viable non-formal learning experience for student acquisition of

21st century skills?

135
 Methods

Participants

Participants in the pilot study consisted of seven students, ranging in age from 11 to 14,

who participated in a STEM summer camp at a suburban middle school in southwest Virginia.

These students were divided into dyads (teams of two) while engaged in a 45-minute

orienteering experience on the school grounds. One dyad was randomly assigned to wear a

GoPro® camera and be video and audio recorded during the experience.

Procedures

The course, consisting of nine control markers of various sizes (Appendix K), was set up

the morning of orienteering experience. Control markers of three different sizes (small, medium,

and large) were placed based on terrain and visibility. Control markers in open areas were

smaller, so they would be more difficult to spot from a distance, while those hung within trees or

brush were larger. This placement necessitated student use of the map, rather than merely visual

location of control markers. Each control marker was assigned a point value based on difficulty.

Five of the control markers were assigned a value of ten points each, two control markers that

were more difficult and/or farther from the start/finish were valued at twenty points each, while

the two most difficult control markers were valued at thirty points each. Students were tasked

with accumulating as many points as possible within a 45-minute time limit. If teams did not

return to the starting point within 45 minutes, 5 points were lost for every minute they were late.

Points were accumulated by finding a numbered control marker and using the attached hole

punch to punch a unique hole pattern into the box on the scorecard with the corresponding

number.

136
 Students participated in a brief lesson on reading the map, were given a scorecard, and

told rules for the orienteering course. This pre-experience lesson took approximately 20 minutes

to complete. Students worked very hard to complete the course as quickly as possible, and all

teams arrived back at the starting point prior to the 45-minute time limit. Following the

experience, all seven students participated in a post-interview consisting of twelve questions

(Appendix D). A stationary camera was set up in a classroom to capture the post-interview. The

researcher transcribed the experience and subsequent interview.

Results

The camera captured the dyad’s entire orienteering experience, and they naturally

articulated their thoughts and experiences throughout. The post-interview was conducted with all

seven participants. This occurred after the students waited for everyone to complete the

orienteering experience. Students were excited about the orienteering, and they shared their

experiences with one another while waiting for teams to finish and while walking back to the

classroom. As a result, valuable data relevant to answering the RQs were not captured. Also,

while feedback from all students was interesting during the post-interview, it also served to

distract discussion away from the phenomena observed within the actual data collection.

Recognizing these shortcomings, the research design was modified to conduct the post-interview

in the field directly after completion of the experience rather than waiting to walk to a classroom.

The pilot study also demonstrated that the post-interview should be conducted with only the

video and audio recorded dyads. Not only would this assist in conducting the post-interview

directly following the dyads’ experience, without waiting for all students to complete the course,

it would help focus the discussion directly toward the experiences of those students observed in

the video.

137
 Using the preliminary coding scheme based on the Framework for 21st Century Learning

developed by Partnership for 21st Century Skills (P21, 2015), student engagement in 21st century

skills was evident throughout the transcripts. However, this initial coding scheme proved to have

copious overlap between skills and some vague definitions that would cause difficulty in

assigning codes, thus leading to low inter-coder reliability. This indicated that both the

categorization of 21st century skills and the codes used for those skills needed to be revised.

Specifically, the critical 21st century skills, that must be included were divided into three distinct

categories: Learning & Innovation (creativity and innovation, critical thinking, and problem

solving), Life & Career (flexibility and adaptability, initiative, self-direction and productivity,

and leadership, responsibility and accountability), and Socio-Cultural (communication and

collaboration). Given these categorical revisions, a newly revised and validated coding scheme

was needed to analyze data collected.

Transcription and Coding Process

Transcription. Video recordings of dyad orienteering experiences and post-interviews

were transcribed manually with individual utterances from each dyad member entered verbatim

into alternating rows of a spreadsheet. Timestamps were inserted every ten minutes to establish

reference points throughout each video. This transcription approach resulted in a typed version of

the verbalizations between participants and post-interview verbalizations with time-stamps

throughout.

Segmentation and coding of text-based verbalizations. The method used to segment

the text-based version of dyad verbalizations was conducted utilizing the coding categories

previously described. This method involved concurrent analysis of a given transcript by three

independent coders. Coders segmented and coded, simultaneously dividing the utterances until

138
each individual segment contained a single code that reflected only one of the eight 21st century

skills. The use of three independent coders ensured rigor and robustness of data analysis.

Arbitration. Once independent coders completed the segmentation and coding of a given

transcript, they met to arbitrate. During arbitration, the independent coders compared, discussed,

and justified the 21st century codes they assigned to each segment. A final code was assigned

where agreement of independently coded segments occurred. Segments that differed in assigned

codes required coders to engage in arbitration to dispute the assigned coding and reach

agreement on the 21st century skill addressed. Independent coders arbitrated until they reached an

agreement on the final code. This coding and arbitration process developed heuristics throughout

the coding process of the pilot transcript. It resulted in a final data set that was readied for use in

statistical analysis. Frequency counts were extracted from the final arbitrated data by means of

Microsoft Excel (2014). Percentage frequency distribution charts were created manually based

on these frequency counts. This quantitative data was then compared to qualitative data in the

orienteering experience and post-interviews to provide a comprehensive picture for data analysis.

Establishing a Valid Coding Scheme

Collaborating and arbitrating throughout the coding process creates a systematic

approach that reinforces clarity and transparency (Hall, et al., 2005). Content analysis through

coding and arbitration established heuristics that were utilized throughout the process (see

“Example Utterances,” Appendix O). Assessment of inter-coder reliability indicates agreement

between two or more coders and provides evidence of rigor of analysis (Ryan, 1999; Lu and

Schulman, 2008). The measure of inter-coder reliability validates the interpretation of content

and quality of research. It requires independent coders to agree on interpretation of the content

based on the coding scheme (Cho, 2008). The percent agreement method, which computed the

139
average pairwise percent agreement (APPA) was used to measure the agreement between coders.

To compute this statistic, the average of cases of agreement between coders was calculated by all

possible codes in the final protocol. A percentage agreement above 75% was used as a

benchmark for coder consensus within the context of the study. This is deemed an acceptable

measure in social sciences (Klenke, 2008; Schloss & Smith, 1999; Stemler, 2004).

Developing codes. A preliminary coding scheme (Appendix C) was developed utilizing

the Framework for 21st Century Learning Definitions of Learning & Innovation Skills and Life &

Career Skills (P21, 2015). After comparing it to the pilot study transcript, it became obvious that

this coding scheme contained copious overlap, making it unsuitable for establishing inter-coder

reliability. The coding scheme was then paired down, and Communication & Cross-Cultural

Skills was separated as a third unique category, rather than combined within Learning &

Innovation Skills and Life & Career Skills (Appendix D). After comparing it once again to the

pilot study transcripts, this coding scheme was sent to a panel of two individuals selected

because of their extensive experience and background in the process of coding and arbitration.

The panelists were provided with the P21 Framework Definitions (P21, 2015) for Learning &

Innovation and Life & Career Skills (Appendix E), and a chart of the P21 skills cross-referenced

with Science and Engineering Practices in the NGSS and Common Core Standards for

Mathematical Practice MP1, MP2, MP3, MP6, and MP8 (Appendix H). The original coding

scheme (Appendix C), pilot study transcripts (Appendix P), and the coding scheme developed

for calibration (Appendix D) were also provided. According to Creswell (2014), the

development, implementation, assessment and continuous refinement of the coding scheme is

paramount to ensuring quality of analysis of the text. Data were coded using Weber’s (1990)

eight-step coding protocol (Table 1).

140
Table 2.

Weber’s Eight-Step Coding Protocol (1990)

____________________________________________________________________________
1. Definition of coding units (e.g., word, phrase, sentence, and paragraph)
2. Definition of coding categories
3. Test of coding on a sample
4. Assessment of the accuracy and reliability of the sample coding
5. Revision of the coding rules
6. Return to step 3 repeatedly until sufficient reliability
7. Coding of all text
8. Assess the achieved reliability or accuracy
______________________________________________________________________________

The panelists read the original definitions and the pared down coding rubric, indicating

whether they agreed or disagreed with each item. If they disagreed, they indicated what needed

to be changed (Appendix F). The panelists and lead researcher then met and reviewed all

comments, arbitrating solutions for each skill. Through this process, the lead researcher

developed a newly revised coding scheme (Appendix G).

Intercoder Reliability Process

Utilizing the newly revised coding scheme (Appendix G), co-coders and the lead

researcher individually coded 10% of the pilot study transcript. The student utterances were

segmented based on the meaning conveyed (Figure 4)

Alright. (looking at map) We've got 6 over there by that first

A house. It's just gonna be right up here. (Turning map) yeah, 1 1 1 1 1 1 1 100% CT CT CT CT 1 1 1 100%
telephone's right there, right there (pointing to map) alright.
Come on Iain you can do it! 2 1 2 2 1 0 1 67% LRA LRA LRA LRA 1 1 1 100%

Figure 4. Segmented transcript example

In the figure above, the utterance of student A is divided into two segments based on meaning.

The first segment conveys communication (1) and critical thinking (CT), while the second

segment conveys collaboration (2) and leadership, responsibility and accountability (LRA).

Individual coders recorded their codes in a Microsoft Excel (2014) document. Arbitration of the

141
generated codes produced 86% agreement in coding scheme one and 69% agreement in coding

scheme two. Based on this arbitration, the researcher and co-coders further refined the coding

rubric, then coded and arbitrated another 10% of the pilot transcript, resulting in 76% agreement

in coding scheme one and 68% agreement in coding scheme two. This led to further discussion,

refinement, and delineation of definitions. The process of coding and arbitrating 10% of the pilot

study transcript was then repeated for a third time. The third coding and arbitration resulted in

77% coder agreement for coding scheme one and 79% agreement for coding scheme two. The

researcher and co-coders again looked for discrepancies within the coding schemes and worked

to further clarify assignments of codes. An additional 30% of the pilot transcript was then coded

and arbitrated. This resulted in 77% agreement for coding scheme one and 76% agreement for

coding scheme two. The researcher and co-coders once again examined the coding schemes,

clarified and further developed heuristics for definitions, and then coded and arbitrated the

remaining 40% of the pilot study transcript. Inter-coder agreement of the last 40% was 76% for

coding scheme one and 77% for coding scheme two. The inter-coder agreement for the entire

pilot study transcript was 77% for coding scheme one and 75% for coding scheme two. This

iterative process resulted in operational definitions of codes, heuristics, and example utterances

that produced acceptable intercoder agreements (Appendix O).

Pilot Study Resulting Changes

Based on the results of the pilot study, the following changes were made:

 Reorganization of P21 skills into three main categories of skill

 Revised coding scheme based on 21st century skills
 Revised procedures for post-interview
o Conducted in the field directly after experience
o Conducted with only participating dyads instead of whole group
 Revised post-interview questions

142
Instruments and coding scheme resulting from the pilot study were then incorporated into the

final research design for the main study. The following sections will provide the details

regarding the method used to collect and analyze data in this study.

143
 Appendix R

Initial Post-Experience Interview Notes

144
Dyad Interview 1

TIME Sj UTTERANCE Notes

Did you begin with a strategy or
1:35 F develop one along the way?
apply knowledge- build skill,
Well we went to the first one that we strategic thinking in the
felt like it was, and then we went to field, self-correct, flexibility,
that one and we went to another one. learn through experience
And after that we decided that we
were gonna go around in a circle. First
we just went to the two closest ones
A and then we went around
The first 3 we decided randomly and strategy (ct)
then we decided to go in a loop that
was semi-close to the where the one
we were at was and went around
B and…
Ok, ok, was it real helpful to have a
F strategy?
self-evaluation, reflection
I think it did help because otherwise
we would've had to cover a lot more
distance. We would've been going over
A there and then coming back so
B Yeah I agree
Did your strategy change during the
course of your whole experience? You
mentioned that once you decided
what that strategy was, did it change
F at any point in time?
A Not really
Well we kind of went around in a loop
but then, I forgot exactly why we just
B kind of broke off from that loop
Because then we had to come across
A the road
B Yeah
A And then we just
B Yeah
Ok, were the control markers where
F you originally thought they were?
Um, most of them were in about a 5
foot radius of where I thought they
B would be

145
 So you didn't have any problem finding
F any of them?
B No
Ok, did you all agree on decisions that
were made throughout the whole
F entire thing?
A Yeah
Did you pretty much agree or did you
F have to negotiate some things?
A Not really
It was more of a one person, let's do cooperation
this, the other person, alrighty. Just
B simple stuff.
When that happened, was that like,
did you each, you know like you made
one decision and then you made a
decision, or was it pretty much one-
F sided, or did you both come up with it
Um, I think I did, I think I made a bit
B more deciding
A Yeah he was holding the map
F So you had the map
B Yeah
Is there anything that you would've
F done differently?
reflection
Not really, other than there was one
that we had to come back for in the
woods that um, that we, I think,
missed, but we would've planned that
into our circle. But other than that not
A really
B Yeah other than that
What do you think, while you all were
doing this, what do you think your
strengths were? What were your
strengths that you thought helped you
F do this?
A I think we communicated well, we um communication
B Yeah, cooperation

146
 collaboration
Yeah we didn't disagree on much. And
we, when we searched I think maybe
one of us would look there, one of us
would look in a slightly different
A direction, so that.
B yeah
So you cooperated well, is that what
F you're saying?
A Yeah
B Mhmm
Do you think you had any strategies or
decisions that were especially
F inventive or creative?
B Not that I can think of no CI
Not really. We just went to the closest
A ones
F Ok, that was your strategy
A Yeah
B Mhmm
Did, um, what do you think you might
have learned or figured out as you
F went
I think it's a good exercise for building collaboration
A teamwork skills, and cooperation
And your respect toward your collaboration
B partner's decisions
Tell me about maybe the process that
you went through whenever you had
to solve a problem that you
F encountered
problem solving, decision
Well, I mean for the most part things making process,
went pretty smoothly, but when we collaboration
did have a minor problem, just one of
us would suggest an idea and we'd
think about it for just a few seconds or
so and we'd agree on something to fix
it. Normally the first idea that was
A suggested. There wasn't much debate.

147
 How could this experience help you
solve a problem, um, a different kind
of problem maybe at home or at
school? In other words, is there
anything that you learned here today,
the skills that you learned here today
that you might could take with you
F somewhere else?
collaboration, strategic
Well as we said a couple times, thinking
teamwork and cooperation and even
though our strategy was pretty simple,
coming up with a strategy I think was
A important
F Ok
And those are all things that can relate
A to real life situations
B Yeah
F Anything else?
B (head nod)
Ok, is there anything else that you
might like to tell me about the
F experience?
8:00 A Not really, not anything from me.

7th period boys (6:30 total)

148
Dyad Interview 2

TIME Sj UTTERANCE Notes

First question, did your team begin with a
:52 F strategy?
A Kind of
B Kind of
We decided to find, we had no idea that we
were gonna find them all. So what we tried to
do is we tried to go, we picked a specific area,
A like toward the northwest corner of the field
B Over there
Yeah, the northwest corner of the field, and strategy
we picked the, got the highest points out of
A each area
Oh, ok. So did your strategy change at any
F point? Did you have to change it?
Not really. It worked really well for us. Except I
sort of wish we had been a little…we missed
A one, cause well we overlooked one that we…
continuous self-assessment
Cause we were, I don't know, I guess we just
had our sights set farther, like, but it was
actually way closer than we thought it was
B going to be
So you picked the places that had the highest
F points
Well we tried to. Eventually we were just like,
B there's one!
change in strategy
Well now we eventually decided to pick, after
a while we thought we were running a little bit
low on time so we decided to get the ones
closest to us, and we decided to work, go far
A out then work our way back to here.
Were all of the control markers where you
originally thought they were? Were they right
F where you…
B Uh, no
No. Number 4 was a whole lot farther down learning scale
A than we thought it was.
B Yeah
What did you do when you got to somewhere
where you thought it should be and it wasn't
F there?

149
B Well I was like Lee, look at the map again
decision-making in the field
Well we decided, she made me double look,
triple check and quadruple check. But then we
decided just to keep going because we knew
A we hadn't passed it yet. So
Did each of you agree on the decisions
throughout the experience? Did you always
want to do the same thing or did you ever
F have to...
Well we sort of were, I think we worked good collaboration
because, you know, whoever sort of came up
A with what we were gonna do first we did so…
B Yeah
So there was never a time when you wanted
F to do two different things?
Yeah. Not really. Sometimes he was like "I see
it, it's right there" and I was like "what,
B where?" But that's really it.

OK, now that you've done this once, if you had

F it to do again, what would you do differently?
A Ummm
B Probably not anything. I think we did good
A I think we got lucky
B Yeah we did
We got, we were lucky that we had a pretty
good idea, cause the map was pretty good,
cause it was good that it was aerial it didn't
A just have like a landmark
B Yeah. It was fun, I liked it
Alright, so what do you think, obviously you
did really, really well, so what do you think
were your strengths? What made you be able
F to do so well?
B Definitely teamwork
A Cooperation collaboration
B Yeah
Cause there was a couple times that she
probably wanted just to leave me in the dust,
A but…
F You stuck together pretty well?
A Except for one part but then I caught up
B He was like "I see it"

150
 Well the main thing is that my head set fell off,
so I had to pick it up. Well that happened
A down there and it like got twisted up here
OK, um, let's see, were any of your strategies
or any of the decisions you made especially
F inventive or creative?
B I don't know, not really, we kinda just…
Well, we decided, not exactly creative, but not, not CI
A I don't know
Innovation?
Once we saw groups farther down we were
like, hey let's go over there and then we
looked at the map and went there. (Looking at
another team) Ha! They ran past it too. That's
B hilarious.
They're gonna win, they're gonna win. They
A got the key
F Oh they found the…
Yeah, well one person ran toward the other
B one
Well yeah, they left each other behind. Cause
that's how they outran us, we would've been
A there first, but
B Maybe
A No we were there second
So what did you, I just have a few more. What
F did you learn or figure out as you went?
critical thinking, problem
That the thing, it was, we had to sort of accept solving
and understand that they are there. They're
not like mislabeled. Cause it got, sometimes it
got a little frustrating cause we'd be like "we're
A right here"
B It's not here, where is it. Yeah
And sometimes it would look like the circle
A was on this side and… but it actually was.
persevere, grit/resilience, safe
Oh yeah, down there, where there's the row of opportunity for failure
pine trees I was like "I think it's up here" so we
ran up and then I saw it like way down there
so I like slid down there. That was fun but it
was also frustrating because we were like
B "why is it down there?"

151
 Yeah, Ok. So when you did have difficulties,
when you got to places where you thought
they'd be and they weren't there, what did you
do to problem solve. Tell me your process, like
F what did you do next?
Like sort of let's go a little bit further and see if
A it's there
B Yeah mostly
problem solve
Like sort of problem solve. We would stop, we
would look at our map again and just like "hey,
maybe we're a little bit this way or we didn't
A quite get there."
Ok, and then if you went a little further you
found it each time, there was no time that it
F was like…
Mostly, yeah, um, well this one was really hard persevere
B but
we gave up on number 4 and we came running
cause we thought, well we'll get to 5, well we
ended up getting to 4 cause 4 was a lot farther
A down than we thought.
Ok, and can you tell me any times where
maybe this experience, having done this would
help you be better at solving problems, like in
F real life?
A Definitely cause
B Yeah
In group work, you need teamwork, and I
A mean the teachers can tell you to work
B Together
To work together but that's not gonna- they
A can't make you and when you're
B This is fun so…
When you're doing something that's fun, but collaboration
A also challenging
B You want to work
Yeah you want to work together and it
A teaches good teamwork
Oh, that's a really good way of saying that.
Anything else that you want to tell me about?
F Anything else that might help?
B We should do this more
A We should do this in gym

152
 B Yes!
F Yeah that would be fun
But you can be our PE teacher cause you're
7:48 A not yelling at us.

4th period mixed (7:00)

153
Dyad Interview 3

TIME Sj UTTERANCE Notes

So did your team begin with a strategy
1:23 F or just develop one along the way?
A More of a half strategy
strategy
Um, we kind of did a little bit of each.
We had our next move planned and
while we were going for that area we
B would figure out our next move
Yeah as we were stamping it we would
A figure out where to go next
Ok. So what do you think why do you
think it might be helpful to have a
F strategy when you started
Because you wouldn't be disoriented
and confused about where you were
B going, because there's only…
We didn't really know where to go create anxiety
constantly so it was really kind of a
A wager
Cause we thought it was gonna be
trouble that there was only a sliver of
road that we could go across so if you
don't realize that then you could really
B mess yourself up

How did the point values of the control

F markers influence the strategy you had?
Um, we kind of just ran to whatever was
nearest and I said "How much is this
A worth?"
We did run the…We didn't really mind
the 10 point ones but the 30 point ones
we did go for and then the 20 point
B ones we didn't really…
Yeah we definitely prioritized the 30 strategy
and the 20 and then the 10 if it was
A near or on the way we just kinda got it
B Mhhm

154
 Ok, did your strategy change during the
course, I mean as you started you said
you had the strategy whenever you
F reached the next one, did it change?
B Uh slightly
A A little bit
Sometimes on the way, you realize that continuous assessment, flexibility
you are not pinpoint where you think
B you are, so you'd have to
A Yeah we'd be running and he'd be like...
building skill
So if you're running one direction and
you thought you were running the other
direction to get one then it kinda
changes of how you do through out the
B map
A It really changes
Is there anything in particular that
changed during that time, you know,
your strategy, any particular strategy
F that changed?
Um, I think I remember running one
part to a 30 and he said "Hey there's a
10 nearby" and I said, "Yeah let's just
A get that one."
Cause there's this huge treeline down
B there, so
Yeah we kind of had to figure out where creative, critical thinking
to go into and out of, kind of like where
A people had been
Yeah, cause we had to swerve to get to
the last, the part that's nearest to the
B high school
Also we were one of the only people
that actually like stayed behind the
A treeline to get more of them
critical thinking- strategy
Yeah because people were walking
through it and we figured that would
take longer so we went around, and
since there were no branches or
anything in our way, we could run faster
B and go through faster
OK, were the control markers where
F you originally thought they were?

155
A Wait, what?
challenge, anxiety
Um, a little bit, I mean, the same thing
with the pinpoint thing, it was a little
disoriented, uh disorienting, because
you couldn't really tell exactly where
B some stuff are
A Yeah
difficulty level
Cause I was looking on the map, and I
was thinking if I could find the soccer
goal or the football goal then I could
find this easier but they weren't on the
B map, so it was a little more difficult
We'd really say it's in this general area
and then look for the things blowing in
A the wind or anything like that
OK, did you all always agree on the
F decisions throughout the experience?
A Um, mostly
B Yeah pretty much
F What did you do whenever you didn't?
B Um, we kept on walking to our close…
Yeah kinda went to the closest one collaboration
A while kinda negotiating
B Where to go
A Which one's better
B Yeah
F Did that work pretty good?
A Yeah it was ok
B Um, yes
Ok, is there anything that you would've
F done differently?

Um, I might have went for, knowing

that these are closer than we thought
they were. We thought that they were
really far apart and we thought that we
would only get like four or five of them.
Um, so I probably would've actually
stayed on the right side to get
everything and then went to the left
side. Because we kind of did a figure 8
B to get reflection

156
 Yeah, we kind of just, also I think we
shouldn't have waited in line as long as
A we did for some of the starting ones
B Yeah
A That took a lot of time
Cause everybody ran to that one right
B there and so
Yeah, they went to that one and then
A went to that one so it was just kind of
B Yeah
But then as it went throughout then
A everybody was spreading out
So you would've maybe started further
F in, and then came back
B Yeah
A Like do all those and then do these
Ok, so what do you think were your
strengths whenever you did this, maybe
something that you were particularly
F good at?
He was very good at figuring out the
A map, I'm just gonna say that
I do this kind of stuff a lot, um, it's,
B hmmm
6:43 A I think we ran a lot
B I think we did run a lot
A Yeah I kind of got tired toward the end
B Yeah
A He just kept running
And Tanner was like stop, stop, please
B stop
A Stop
Which of your strategies or decisions
F were especially inventive or creative?
Creativity & Innovation? Somewhat
I'm gonna say to stay behind the tree creative to stay behind the treeline,
line, because we could've hopped on critical thinking
the bandwagon just kind of gone
through and out the trees but we
decided it was getting a little scraped
up, we were getting a little scraped up
A so
And how was that an advantage for you
F to stay behind?

157
 Well we could A, run faster and B, not
have to worry about all the branches
A hitting us
Yeah, it cause it would slow us down so
B running around the treeline
A It was a lot safer too
B Yeah
Is there anything…what did you learn or
F figure out as you went?
B What?
What did you learn or figure out as you
F went?
B It's more difficult than it looks challenge
A Yeah it's a lot more difficult
self-reflection, positive reinforcement
Actually it's a little bit easier than it
looks because the map can be a little bit
confusing but once you get the hang of
where the bird's eye view is, like how
far apart things are from each other
then it gets a lot easier to move to the
B different points
A Yeah. Figuring out where to go
(Interrupted by students) Alright, I'm
gonn back up and ask you again, what
F you learned or figured out as you went
Yeah we figured out the distance
B between, um…
Yeah as we went we kinda figured out
A the map a lot better
evaluation throughout
Because as we were getting to different
points we were like it's at this point, but
then we realized that it's not around the
area we needed to move to the next
area, because it was a little bit, um off
B of where we perceived it to be
Tell me about the process that you used
whenever you ran into difficulty. Can
F you tell me, like what you did
Um, we really kept running while decision making in the field
A negotiating, really
Well, at those points, at some moments
B we did stop

158
 Yeah, sometimes we just kinda stopped
A and we
and we looked at the map, we slowed
B down
and we looked at the map and said this
A is here, this is here
B Yeah
A where are we gonna go?
We slowed down, took our time and
B then we went for it
Ok, great. How can this experience help
you solve a different kind of problem
either at home or here at school. Is
there anything, like take-aways from
F this experience?
B Finding my keys (laughs)
Really I think it helped a lot with
navigating, if you're ever camping or
A anything, that helped
B Yeah, true
And also team leadership and working collaboration, leadership
A together had a big role to play
And also learning a new area, so like if
you're visiting someone but you've
never been to the area, you can figure
B out better if you have a map nearby
I want to come back to what you said
about finding your keys, tell me a little
bit more about that, how do you think
F that would help?

Um, this actually kinda taught me to use

the landscape to find things. These were
orange and white but thos entire area is
greyish greenish, mostly green. So if you
were even to look very far away, unless
it's behind the treeline then you'd even
be able to find it from farther away. Like
we were around this area here and we
B looked and we saw number 1
Yeah we saw number 1 and we just kind
A of went for it

159
 So how does that translate to help you
F find your keys?
B Just uh
A They'll be lost forever
Not a test, just a question. You can say
F pass, it's ok
I don't really know, it was kinda more of
B a joke than it was actually
F Oh, ok I was just wondering
We kind of joked to help us get through
A this
Is there anything else that you might
want to talk about or tell about this
F experience?
I think it was a lot better than sitting in motivation
a classroom listening to someone
A talking

Yeah, I would really offer this up to

people if I could ever find a place to go
to to orienteer, because this was really
fun and I think a lot of my friends, most
of my friends would like to do this
because they like to adventure, they like
to find things, they love sports and
B games
A It's like recess with hidden learning motivation
I like that, that's good, recess with
F hidden learning. Good.
B Very mysterious
F OK, anything else guys
B You have anything?
A You have anything?
The map could sometimes, like again to
the map, it's very confusing because
figuring out
until about halfway through we kind of
figured it out and near the end we really
B understood the map
I think if we would've had a compass to
say like north, south, east and west it
A would've made it like a whole lot easier
B Uh huh

160
 When you figured out the map, what
are some things that helped you do
that? You know, for instance in the
beginning you said it was confusing,
F what things were confusing about it
active learning, figuring out, self-
So you would think it was in one specific reflection
spot and would think you see it or
something and you would go to it and
you look at your map and you think you
know exactly where you are and you
realize that the distance is different
than what you first thought. So then by
trial and error, you could figure out the
distance between the two points or
B whatever

Not to mention when we were out by

the trees, like when we were inside the
treeline we couldn't see the school, so
we couldn't use the school as a pinpoint
location anymore. We really had to
A improvise from there
Ok, and then you said about halfway
F through you started figuring things out
A Yeah
Is that the distance thing, is that what
F you figured out?
B Mostly
Yeah, mostly distance. Distance and
A kind of like
The map doesn't really show the
landscape, so I mean it shows the area
but it doesn't really show the hills, the
B size of the hills
F Topography
B Yeah
So we couldn't really tell, well we gotta self-regulation
A pace ourselves
So we could straight shot this forward
but then we saw the hill and we had to,
B uh
A Kinda like jog
B prioritize making decisions
A Yeah

161
14:45 F Ok guys, thanks a bunch. Great job

6th period boys (13:15)

162
Dyad 4 Interview

TIME Sj UTTERANCE Notes

Did your team start with a strategy or
did you kind of develop one along the
:00 F way?
Well first we were like oh there's one
right here, so we ran to this one. Oh it's
right there. And then we just kind of ran
A to the closest one.
OK, so you didn't stop with the map and
F kind of come up with a …
A Later in the activity
I was more of the map person, he was
B more of the one who went to go stamp different jobs-collab
Ok so tell me about that. You said later
in the experience maybe you stopped
F and came up with more of a plan
Cause we had most of them, we had a,
like, we have time to look where
A everything was and look on the map time management
Ok, and you said you were the map
person, so did you kind of have a plan,
F or…
I just kinda looked at it and as Brett said
just went to the closest one and he'd
run to there, stamp it and come back
B out

So you had kind of a strategy, you each

had your job, right? So, did your
strategy change at all did you, you
know, once you started doing things did
you realize anything that made you
F change what you were…
Closer to the end, like when she did the
5 minute whistle we kind of picked
A things up, and like ran the whole way Change strategies, flexible
Anything else that made you change
what you were doing? (head shakes)
Pretty smooth. Were all the control
markers where you originally thought
F they were?

163
 There was only, I think there was one,
cause I about stepped in the road.
Cause I thought it was right there but
then we had to cross the road and go,
cause they were like right near each
other but the road was in the middle, so
A I thought they were next to each other Challenge- self direction

So other than that did you get to a place

where you thought one would be but it
F wasn't where you thought it would be?
B Number 8
A Yeah
So what did you do when that
F happened?
We just followed down the trees until
B we found it.
We went back in the woods and ran
A behind it
B So people wouldn't see us Creative strategies?
Oh, good idea. Throughout the whole
experience, did you guys always agree
on what you were doing? Were the any
F issues teamwise?
A I think we agreed
B Not really
Not really agreed or not really
F problems?
B We agreed. There wasn't problems
There weren't, so I did notice, I think
F you'd run ahead and things like that
A Yeah
So did you have issues with that
F Kendall?
A Sometimes I ran ahead too much
But it wasn't an issue really? Were you
F reading the map at the time?
Yep, yeah. Like I'd point, he'd run over
there, stamp it and I'd still try to figure
B out where the map was going collaboration
F Ok, so that was part of your plan.
A Yes

164
 What could you have done differently?
Now that you've done it, if you had it to
do all over again knowing what you
know now, would you have done
F anything differently?

I think we would've gone like in a path,

so like over here we could've
followed…like, cause I remember we
were down there (pointing) and we ran
straight across and we could've stopped
A on our way, but Self-reflection
F So you saw a better path?
A Yeah
Now that you've done it, there's a
shorter way, maybe. What do think
were your strengths with this? What
F were you good at?
A It was fun.
Yeah, what were some things that were,
you know, that made you good at what
F you did?
A Uh, you can answer
He's fast and I can read maps really
B easily Leveraging strengths
That works. Did you have any strategies
or decisions that were especially
F inventive or creative?
I think once we like, I think to beat
another team we like slid down where it
wasn't even close and tried to race
them. So we like slid down in those
A trees, we slid down a path and… Competition spurring motivation
Ok, did you learn or figure out anything
F as you went?
A Orienteering is awesome motivation
Ok, and , so when things were not
exactly what you thought they were,
when they got difficult, can you tell me
F your problem solving process?
A Uh, Kendall
B Brett

165
 Uh, when we had a problem, we kinda
looked at the map closer and closer, to
see like where this road was like the
curve and stuff. So say there's like, I
guess all those trees and we'd look at
the curve just to know exactly where it Thinking critically, solving problems
A was. attention to detail
And, from doing this, could you see how
doing this experience can help you solve
different kinds of problems either at
home or at school? Was there anything
that you can think of that by doing this
F you could get better at?
A She'd get better at reading maps
Mapreading's a good skill. Anything
F else?
A Anything else (asking Kendall)
B No
Anything else you wanna tell me about
F this experience?
A I hope we can do it again motivation
Yeah, you guys should come back for
5:04 B just fun
D. 6th period mixed (5:00)

166
Dyad 5 Interview

TIME Sj UTTERANCE Notes

First question, did your team begin with a
:00 F strategy or develop one along the way?
We began with one, so what we thought was to
get the furthest one and move in and get the
A closer ones Strategy/ct
Ok, how long did you take to plan before you
F took off?
It didn't take us that long, it took us 10, 15
seconds just to figure out a plan and then we
A took off
And then you were off. You found the road
F crossing first, right?
A Yeah
F Was it helpful to have a strategy?
A Yeah it was
B (turns hand gesturing "sort of")
Yeah it was um helpful because if we didn't have
a strategy it'd be all mixed up and we wouldn't
know where to go and we'd be like, arguing over
A which one to go to strategy
F Did you guys agree on a strategy?
A Yeah
B We did
Ok, did the point values on the control markers,
F did they influence your strategy
No, we just tried to get them all before the time
A went up
And you guys were the ones who found the
F bonus
A Yeah
Was that just by accident or were you looking at
F the 10 point ones first to see if you could…
Um, we just went to the, we just started going to
A the further ones and we just found it
OK, so did your strategy change during the
F course?
A No

167
 Not really, um, we just really, um, one by one we
B got the closest one, and we just kept, doing that
Did you find like, did you do like a loop, or how
F did you…
Um, we went like this, we went back around and
like this (making circle with arm). But we
dropped the, I accidentally dropped the keychain
A so I had to go back to 13, then… perseverance
F Ok, but you found it
A Yeah, we found it
F So which one did you wind up doing first
We went to number 2 first I think, the one over
A here
F The on next to the road
A Yeah
Ok, were control markers where you originally
F thought they were?

Um, one of them wasn't, number 15. We

searched for it for like a long time. It wasn't long,
but we searched for it. Then we decided to move
on then we thought that we could probably
come back and find it and we did. So it wasn't
where we thought it was gonna be, it was further Flexible thinking/decision
A down the trail than we thought making
B yeah
At what point did you come back and get it, at
F the very end?
A No, toward the middle
F Ok, were you nearby
No, we were all the way down here then we
A stopped over here, then we just went up
Did the points on that one, that was a 30 point
one, did that influence your decision to go back
F and look for it?
Um, we didn't even check the points, we just
A decided to go get it
B Yeah
Ok, that was a 30 point one. Cool. Did you guys
both agree on your decision making process, like
F who decided to go back and get…

168
 I did because it would be helpful if we got them
all. Cause we didn't know if other teams
would've like got it or not, so… then we agreed
A on it competition
Were you good with that or were you like "oh
F let's keep going"
B Um, kind of, I thought it was kind of too late, but
F Ok, but you said ok, let's do it
B Mmmmhmm
You were like, you're game. Anything else where
F you maybe had different opinions
A No not really
B We were just,we were out looking
Ok, go with the flow. Have you guys worked
much together in the past on other labs or
F projects
B Not really
A Um, no, not really
F Ok, did it work, did you feel comfortable
A yeah
Of course now looking back you're like "we got
F them all"
A yeah
You did well but, had it turned out differently
F would you be like "ahhh"
Um, nah I still think it would've been like a fun
A experience for me and Yungjin motivation
B Mmmmhmm
A So we would've just had fun
F Cool. So what could you have done differently?
Um, not dropped the keychain out of my pocket.
A To bend over to tie my shoe
F Alright, so I noticed your pockets zip, so
A Yeah I forgot to zip it
Well that, I'll tell ya, that happens. Anything
F else?
A Um, no, not really
Do you think with the 15 that you abandoned
and came back to, any different opinions about
that, I mean, would it have been helpful to look
longer for it, or were you totally in the wrong
F spot

169
 We were way up the trail from where it was, so
A we were in the wrong spot
Ok, you guys did great, so what do you think
F your strengths were
A Working together for one collaboration
B right
A And like communicating with each other
F You talked the whole time?
B And our speed
A Yea, and our speed. That helped a lot
F How much did you run?
B We ran a lot
A We ran pretty much the whole course
Did you take any walk breaks, or what did you,
F did you walk to catch your breath?
we took a small walk break and then we tied our
B shoes
yeah the walk break was after when we came
over here and ran completely across the road
A and stuff
B Yeah
Over here from one side of the field to the other,
A so
How about your route, you think that was pretty
F efficient?
A Um, yeah
B Yeah
F Your strategy you thought was
A Yep
B Mmmmhmm
F What about your map reading?
Um, our map reading was pretty good except for
number 15 which I thought was further up the
A trail then it was
B Yeah, it was up the big field

Ok, did you have any, were any of your strategies

or the decisions you made, did you think they
were especially inventive or creative? So did you
come up with something that was different from
F what other teams were doing? No ci?
B I don't think so
A I don't think so, but, no

170
 Did you find any markers maybe a different way
F than you…
A yeah
B yeah
F Remember which ones? (hands them a map)
Um, hang on (looking at map) wasn't it, pretty
A sure, it was number 15 again
F OK
A The way we found it
F How'd you finally come into it

So we went up to number 2, then we came down

like halfway in between 2 and 15 and we were
like, alright, it should be in here but it turns out it
was further down the trail, so then we were like,
I think we should explore further down because
it might be down in there. Then Yungjin was like
yeah I think we should do that. So then we just
walked down the trail just a little bit and then we Perseverance, problem
A found it solving
Were you in the trees the whole time or were
F you on the cross country course
B We were on the cross country
F Ok
A We were on the cross country course
Yeah, not a lot of markers over there, I'm not
sure that I would've…that was different down
F that way
A Yeah
Not a lot of fixed course description on the map
so that would have, that definitely was different.
Um, ok, what did you learn or figure out as you
went? What got or became more obvious or
F easier for you as you went along
B Umm, I'm not sure
F Anything get easier?

Um, as we went farther along we thought, like

we should just, since we checked the time, each
time we got one we were like, alright, we can get
this one, we can get this one each time, so that's Learning from small
A what we did, like help us push on and get our… successes
So you figured out you could probably get them
F all

171
B Yeah
Now was that your plan originally or were you
F just gonna get as many as you could
No we planning on getting them all from the very
A beginning
B Yeah
So if you were to do the same course exactly
F again, anything different?
No, I think we'd keep the same strategy because
A it worked out fine reflection
Ok, so you had one problem when you couldn't
find that one and came back to it, uh, what was
the, can you talk about the process that you
went through to figure…problem solve that, you
know, we were going to get them all, we missed
F one, so now what

So after we got the ones that, after we got the

two that were down here, we cut up the field
and we were sitting in the middle of the field
talking and we were like, alright, I know we
missed number 15, but I think we should go back
and try to find that because it might help us out
with points, since, uh, more points the better, so
A that's what we, and we both agreed on it, so
At that point you were still pretty far away from
F it
A Yeah
B Yeah
It wasn't right next to you, so basically you made
the decision to take the time and go back and Decision making in the
F grab it field
A Yeah

Ok, with that in mind, kinda think about what

you did there, you had a problem, you fixed it.
How could this experience help you solve a
different kind of problem at home or at school?
Nothing to do with orienteering or finding
something in the woods or the fields. You've just
got some issue, with a friend or with parents, or
F brother, sister...
We can like communicate with each other or
work together on some things, then we'd get our Communication,
A relationship goal back up, so collaboration

172
 F Ok, anything else
A Um, no

Uh, is there anything else you wanna tell me

F about this experience, like it was fun, was it fun?
A Yeah, it was really fun
F Anything else? motivation
A No, not really
B shakes head
F Would you do it again?
A Yeah
B Yeah
A Definitely
If it wasn't at school would you go out and do it
F somewhere
A Yeah
8:47 B Totally
E: 4th period boys (8:47)

173
 Appendix S
Student Utterances by Theme

174
Theme Student Utterances

Thinking critically to 1. “when we did have a minor problem, just one of us would
solve problems suggest an idea and we'd think about it for just a few seconds
or so and we'd agree on something to fix it.” (Dyad 1, line
50)
2. “Yeah, the northwest corner of the field, and we picked the,
got the highest points out of each area” (Dyad 2, line 7)
3. “Like sort of problem solve. We would stop, we would look
at our map again and just like "hey, maybe we're a little bit
this way or we didn't quite get there."(Dyad 2, line 60)
4. “…we had to sort of accept and understand that they are
there. They're not like mislabeled. Cause it got, sometimes it
got a little frustrating cause we'd be like ‘we're right here’”
(Dyad 2, line 53)
5. “Yeah we definitely prioritized the 30 and the 20 and then
the 10 if it was near or on the way we just kinda got it”
(Dyad 3, line 13)
6. “Uh, when we had a problem, we kinda looked at the map
closer and closer, to see like where this road was like the
curve and stuff. So say there's like, I guess all those trees and
we'd look at the curve just to know exactly where it was.”
(Dyad 4, p. 4)
7. “Yeah we kind of had to figure out where to go into and out
of, kind of like where people had been” (Dyad 3, line 25)
8. “Yeah because people were walking through it and we
figured that would take longer so we went around, and since
there were no branches or anything in our way, we could run
faster and go through faster” (Dyad 3, line 28)
9. “Uh, when we had a problem, we kinda looked at the map
closer and closer, to see like where this road was like the
curve and stuff. So say there's like, I guess all those trees and
we'd look at the curve just to know exactly where it was.”
(Dyad 4, line 51)
10. “So we went up to number 2, then we came down like
halfway in between 2 and 15 and we were like, alright, it
should be in here but it turns out it was further down the trail,
so then we were like, I think we should explore further down
because it might be down in there. Then Yungjin was like
yeah I think we should do that. So then we just walked down
the trail just a little bit and then we found it” (Dyad 5, line
105)
11. “It wasn't right next to you, so basically you made the
decision to take the time and go back and grab it” (Dyad 5,
line 128)

175
Scaffolded successes 1. “The first three we decided randomly and then we decided to
encouraging go in a loop that was semi-close to where the one was…”
learning/solving (Dyad 1, line 4)
problems 2. “Well now we eventually decided to pick, after a while we
thought we were running a little bit low on time so we
decided to get the ones closest to us, and we decided to work,
go far out then work our way back to here.” (Dyad 2, line13)
3. “We had our next move planned and while we were going for
that area we would figure out our next move” (Dyad 3, line
4)
4. “to stay behind the tree line, because we could've hopped on
the bandwagon just kind of gone through and out the trees
but we decided it was getting a little scraped up, we were
getting a little scraped up so” (Dyad 3, line 70)
5. “…each time we got one we were like, alright, we can get
this one, we can get this one each time, so that's what we did,
like help us push on and get our…” (Dyad 5, line 115)
6. “So after we got the ones that, after we got the two that were
down here, we cut up the field and we were sitting in the
middle of the field talking and we were like, alright, I know
we missed number 15, but I think we should go back and try
to find that because it might help us out with points since, uh,
more points the better, and we both agreed on it so” (Dyad 5,
line 3 124)
7. “Um, as we went farther along we thought, like we should
just, since we checked the time, each time we got one we
were like, alright, we can get this one, we can get this one
each time, so that's what we did, like help us push on and get
our…” (Dyad 5, line115)
8. “So after we got the ones that, after we got the two that were
down here, we cut up the field and we were sitting in the
middle of the field talking and we were like, alright, I know
we missed number 15, but I think we should go back and try
to find that because it might help us out with points since, uh,
more points the better, and we both agreed on it so” (Dyad 5,
line 124)

Authentic collaboration 1. “It was more of a one person, let's do this, the other person,
alrighty. Just simple stuff.” (Dyad 1, line 23)
2. “A:/Yeah, cooperation B:/ Yeah we didn't disagree on much.
And we, when we searched I think maybe one of us would
look there, one of us would look in a slightly different
direction, so that.” (Dyad 1, lines 33-34)

176
 3. “A:/ I think it's a good exercise for building teamwork skills,
and cooperation. B:/ And your respect toward your partner's
decisions” (Dyad 1, lines 47-48)
4. “…teamwork and cooperation…” (Dyad 1, line 52)
5. “I think we worked good because, you know, whoever sort of
came up with what we were gonna do first we did so…”
(Dyad 2, line 22)
6. “A:/Definitely teamwork” B:/Cooperation” Dyad 2, line 34-
35)
7. A:/ “In group work, you need teamwork, and I mean the
teachers can tell you to work” B:/ “Together” A:/ “To work
together but that's not gonna- they can't make you and when
you're” B:/ “This is fun so…” A:/ “When you're doing
something that's fun, but also challenging” B:/ “You want to
work” A:/ “Yeah you want to work together and it teaches
good teamwork” (Dyad 2, lines 67-73)
8. “…I’d point, he’d run over there, stamp it and I’d still try to
figure out where the map was going.” (Dyad 4, p.2)
9. A:/ “Yeah kinda went to the closest one while kinda
negotiating B:/ “Where to go” (Dyad 3, lines 40-41)
10. “Um, we really kept running while negotiating, really” (Dyad
3, line 88)
11. “And also team leadership and working together had a big
role to play” (Dyad 3, line 100)
12. “I was more of the map person, he was more of the one who
went to go stamp” (Dayad 4, line 6)
13. “Like I'd point, he'd run over there, stamp it and I'd still try to
figure out where the map was going” (Dyad 4, line 32)
14. “He's fast and I can read maps really easily” (Dyad 4, line
43)
15. (strengths) “Working together for one” (Dyad 5, line 71)
16. “And like communicating with each other” (Dyad 5, line 73)
17. “We can like communicate with each other or work together
on some things…” (Dyad 5, line 131)

Motivation 1. A:/ “When you're doing something that's fun, but also
challenging” B:/ “You want to work” (Dyad 2, p.4)
2. “It's like recess with hidden learning” (Dyad 3, lines 115)
3. “I think it was a lot better than sitting in a classroom listening
to someone talking” (Dyad 3, line 113)
4. “I think once we like, I think to beat another team we like
slid down where it wasn't even close and tried to race them.
So we like slid down in those trees, we slid down a path
and…” (Dyad 4, line 45)
5. “Orienteering is awesome” (Dyad 4, line 47)
6. “I hope we can do it again” (Dyad 4, line 58)

177
 7. “Um, nah I still think it would've been like a fun experience
for me and Yungjin” (Dyad 5, line 59)
8. “Yeah, it was really fun” (Dyad 5, line 135)

Self-evaluation, 1. “I think it did help because otherwise we would've had to

flexibility cover a lot more distance.” (Dyad 1, line 6)
2. “Not really, other than there was one that we had to come
back for in the woods that um, that we, I think, missed, but
we would've planned that into our circle.” (Dyad 1, line 30)
3. “Well we decided, she made me double look, triple check
and quadruple check. But then we decided just to keep going
because we knew we hadn't passed it yet.” (Dyad 2, line 20)
4. “B:/Mostly, yeah, um, well this one was really hard but A:/
we gave up on number 4 and we came running cause we
thought, well we'll get to 5, well we ended up getting to 4
cause 4 was a lot farther down than we thought.” (Dyad 2,
line 62)
5. “Sometimes on the way, you realize that you are not pinpoint
where you think you are, so you'd have to” A:/ “Yeah we'd
be running and he'd be like...” B:/ “So if you're running one
direction and you thought you were running the other
direction to get one then it kinda changes of how you do
throughout the map” A:/ “It really changes” (Dyad 3, lines
18-20)
6. “We didn't really know where to go constantly” (Dyad 3, line
4)
7. “I mean, the same thing with the pinpoint thing, it was a little
disoriented, uh disorienting, because you couldn't really tell
exactly where some stuff are…Cause I was looking on the
map, and I was thinking if I could find the soccer goal or the
football goal then I could find this easier but they weren't on
the map, so it was a little more difficult” (Dyad 3, lines 31-
33)
8. B:/ “so I probably would've actually stayed on the right side
to get everything and then went to the left side. Because we
kind of did a figure 8 to get A:/ “Yeah, we kind of just, also I
think we shouldn't have waited in line as long as we did for
some of the starting ones” (Dyad 3, lines 48-49)
9. “Because as we were getting to different points we were like
it's at this point, but then we realized that it's not around the
area we needed to move to the next area, because it was a
little bit, um off of where we perceived it to be” (Dyad 3, line
86)
10. “Closer to the end, like when she did the 5 minute whistle we
kind of picked things up, and like ran the whole way” (Dyad
4, line 12)

178
 11. “There was only, I think there was one, cause I about stepped
in the road. Cause I thought it was right there but then we
had to cross the road and go, cause they were like right near
each other but the road was in the middle, so I thought they
were next to each other” (Dyad 4, line 14)
12. “I think we would've gone like in a path, so like over here we
could've followed…like, cause I remember we were down
there (pointing) and we ran straight across and we could've
stopped on our way, but” (Dyad 4, line 36)
13. “Um, one of them wasn't, number 15. We searched for it for
like a long time. It wasn't long, but we searched for it. Then
we decided to move on then we thought that we could
probably come back and find it and we did. So it wasn't
where we thought it was gonna be, it was further down the
trail than we thought” (Dyad 5, line 33)

Reflection, intrinsic 1. “…even though our strategy was pretty simple, coming up
positive reinforcement with a strategy I think was important.” (Dyad 1, line 52)
building skill/knowledge 2. “Cause we were, I don't know, I guess we just had our sights
set farther, like, but it was actually way closer than we
thought it was going to be” (Dyad 2, line 10)
3. “Actually it's a little bit easier than it looks because the map
can be a little bit confusing but once you get the hang of
where the bird's eye view is, like how far apart things are
from each other then it gets a lot easier to move to the
different points” (Dyad 3, line 81)
4. “So you would think it was in one specific spot and would
think you see it or something and you would go to it and you
look at your map and you think you know exactly where you
are and you realize that the distance is different than what
you first thought. So then by trial and error, you could figure
out the distance between the two points or whatever” (Dyad
3, line 125)
5. “it's very confusing because until about halfway through we
kind of figured it out and near the end we really understood
the map” (Dyad 3, line 121)
6. “So you would think it was in one specific spot and would
think you see it or something and you would go to it and you
look at your map and you think you know exactly where you
are and you realize that the distance is different than what
you first thought. So then by trial and error, you could figure
out the distance between the two points or whatever” (Dyad
3, line 125)

179
 Theme Frequency Percentage
Thinking critically to 11 17
solve problems
Scaffolded successes 8 13
encouraging
learning/solving
problems
Authentic collaboration 17 27
Motivation 8 13
Self-evaluation, 13 21
flexibility
Reflection, intrinsic 6 9
positive reinforcement
building
skill/knowledge

Total 63 100

180