Analysis of Sample

PISA Items: Gateway to

Understanding the Philippine
Performance in ILSAs

Rosario M. Belmi, PhD

Faculty of Science Technology and Mathematics
Philippine Normal University
 Reflection Activity Sheet
• Due to the limitation of our online platform
facilitation of this topic (designed for
asynchronous study) and considering the
number of participants, may I invite you to
please answer the Reflection Activity Sheet I
prepared.

• It covers 5-point questions and can be accessed

through this LINK: https://bit.ly/3EeAYoE
Science Pre-survey Test
 Partial Result
Number of Distribution of teachers by score
Scores
Score 0
Pre-training Survey Part 2
Teachers
186 1%
1%
2%
1% 1% 0% 0%

1% 1%0%
Score 6 1 2%
1%
Score 31 2 1% Score 0

Score 6
Score 33 4
Score 31
Score 34 1 Score 33
Score 35 2 Score 34

Score 37 1 Score 35

Score 38 3 Score 37

Score 38
Score 39 4
Score 39
Score 40 2 Score 40
Score 41 2 Score 41

Score 42 2 Score 42

Score 46 1 Score 46
88%
Score 47 1 Score 47

Grand total 212

 INTERPRETATIONS
• 88% (186 teachers) did not take the test
seriously or they simply failed
• Of the 12% (26 teachers) who took the test there
can be 2 interpretations:

• At 50% passing rate = 1 FAILED and

25 PASSED

• At 75% passing rate = 24 FAILED

and 2 PASSED
 Goals of the Topic 2 in Module 3
• Make use of the learnings from Module 3 Topic 1: Understanding the
PISA Framework in analyzing sample PISA items.

• Relate how the different aspects of scientific literacy competencies,

levels of cognitive demand and science and technology contexts are
put together to come up with a challenging assessment item.

• Reflect and compare the quality and complexity of the PISA items with
the assessment items we design and implement for our students.
 What can a Filipino Science Learner do?
1. What is the human body’s normal body temperature?

2. How can the body regulate its temperature given a hot weather?

3. Given a heat index of 49 degrees Celsius, what series of physiological activities in the body will
contribute to the occurrence of heat stroke?

I . Body gains heat from the environment exceeding normal internal body temperature to 40 degrees
Celsius
and up (hyperthermia)
II. Overwhelmed vasodilatation and elevated blood flow to the skin
III. Heart rate, cardiac output, and minute ventilation increase

a. I, III, II b. I, II, III c. I then II only d. III then II only

MEDIUM LEVEL OF COGNITIVE DEMAND
LOW LEVEL OF COGNITIVE DEMAND
What about PISA questions?
• How do they look like?

• How are they made?

• Why are Filipino students having

difficulty answering them?
 Quick Review of the PISA Framework
PISA 2018 Science
Framework
• Looks into the following aspects:

• Contexts
• Domains of Knowledge
• Level of Cognitive Demand
• Scientific Literacy Competencies
 Contexts for Assessment Items
• PISA 2018 assesses scientific knowledge using contexts that raised pertinent
issues that were often relevant to the science education curricula of
participating countries.

• The context may involve technology or, in some cases, a historical element
that may be used to assess students’ understanding of the processes and
practices involved in advancing scientific knowledge.

• These contexts have been chosen in light of their relevance to students’

interests and lives and because they are the areas in which scientific literacy
has particular value in enhancing and sustaining quality of life and in the
development of public policy.
Source: https://www.oecd-ilibrary.org
Sample PISA Items and their CONTEXTS

•Personal

•Local /National

•Global
Quick Review of the PISA Context
Personal – Procedural –High – Evaluate and Design
Loc/National – Epistemic –High – Interpret Data..
Global – Content –Low / Medium– Explain/ Interpret
Sample PISA Items and their
KNOWLEDGE DOMAIN Knowledge Domain

•Content
•Procedural
•Epistemic
•
 Knowledge in Assessment Items
PISA 2018 believes that in the process of acquiring specific scientific literacy
competencies, scientific knowledge is inevitable. In this assessment framework
3 forms of knowledge is considered:

1. Content Knowledge

• The content knowledge that PISA assesses is selected from the major fields of physics,
chemistry, biology, and earth and space sciences
• Is relevant to real-life situations
• Represents an important scientific concept or major explanatory theory that has enduring
utility;
• Is appropriate to the developmental level of 15-year-olds. Source: https://www.oecd-ilibrary.org
Quick Review of the Content Knowledge
• Usually, PISA items are thematic
where under a specific topic,
several questions may arise.

• However, the questions variably

address complexity of items in
terms of cognitive demand,
knowledge domain and scientific
literacy.

• It can also be presented in either

multiple choice, interactive, or
constructed response format
Local/ National
Local/ National – Content –Medium –Interpret Data

Drop down choices:

gravitational,
potential, kinetic,
electrical
 Knowledge in Assessment Items
In this assessment framework 3 forms of knowledge is considered:

2. Procedural Knowledge
• knowledge needed to both to undertake scientific enquiry and engagement
• knowledge of the standard concepts and procedures essential to scientific enquiry
that underpins the collection, analysis and interpretation of scientific data and the
critical review of the evidence that might be used to support particular claims
• knowledge that help produce well-established concepts and methods such as the
notion of dependent and independent variables, the control of variables, various types
of measurement and forms of error, methods for minimizing error, a recognition of
common patterns observed in data, and methods of presenting data
Source: https://www.oecd-ilibrary.org
Quick Review of the Procedural Knowledge
Personal - Procedural
 Knowledge in Assessment Items
In this assessment framework 3 forms of knowledge is considered:

3. Epistemic Knowledge

• knowledge of the constructs and defining features essential to the process of

knowledge building in science (e.g. hypotheses, theories and observations) and
their role in justifying the knowledge produced by science (Duschl, 2008)
• knowledge used to explain, with examples, the difference between a scientific
theory and a hypothesis or between a scientific fact and an observation
• knowledge required to explain (provides a rationale) why the use of the control
of variables strategy is central to establishing scientific knowledge
Source: https://www.oecd-ilibrary.org
Quick Review of the Epistemic Knowledge
Local / National Epistemic
LEVEL OF COGNITIVE DEMAND

•Low
•Medium
•High
•
 Quick Review of the Levels of Cognitive Demand
Level of Cognitive Demand of Assessment Items
• The definition of levels of cognitive demand within the assessment of scientific
literacy and across all three competencies of the framework is a key feature where
each PISA item can be mapped based on the depth of knowledge on the following
categories /level:

1. Low (L) Carrying out a one-step procedure, such as recalling a fact, term, principle or concept
or locating a single point of information from a graph or table.
2. Medium (M) Using and applying conceptual knowledge to describe or explain phenomena;
selecting appropriate procedures involving two or more steps; organising or displaying data; or
interpreting or using simple data sets or graphs.
3. High (H) Analysing complex information or data; synthesising or evaluating evidence; justifying;
reasoning given various sources; developing a plan or sequence of steps to approach a
problem.
Source: https://www.oecd-ilibrary.org
Global– Procedural
Global– Procedural –Low (L1b) –Interpret Data and Evidence

Procedural:
Ways of abstracting data
using graphs and charts
Low (L)

• Carrying out a one-

step procedure,
such as recalling a
fact, term,
principle or
concept or
locating a single
point of
information from a
graph or table.
 Global–
Global– Procedural –Low (L 1a)–Interpret Data and Evidence
Procedural

Procedural:
Ways of
abstracting data
using graphs and Low (L) Carrying out a one-step procedure,
charts
such as recalling a fact, term, principle or
concept or locating a single point of
information from a graph or table.
Refer to “Atmospheric Carbon Dioxide” on the right. Click
on a choice to answer the questions.

Based on the information provided, what effect do

volcanic eruptions have on the concentration of
carbon dioxide in the atmosphere?

A major effect, because there have been many

eruptions.
A major effect, because each eruption ejects large
amounts of material.

A minor effect, because volcanoes release little CO2

compared to other sources.

A minor effect, because CO2 levels in the atmosphere

decrease during eruptions.

Low (L) Carrying out a one-step procedure,

such as recalling a fact, term, principle or
concept or locating a single point of
information from a graph or table.
Global – Procedural
Global – Procedural –Medium (L2) - Explain

Procedural: Medium (M) Using and applying

Ways of conceptual knowledge to describe or
abstracting data explain phenomena; selecting
using graphs and
appropriate procedures involving
charts
two or more steps; organising or
displaying data; or interpreting or
using simple data sets or graphs.
 Global
Global – Content –Medium L3– Explain
– Content

Medium (M) Using and applying conceptual

knowledge to describe or explain
phenomena; selecting appropriate
procedures involving two or more steps;
organising or displaying data; or interpreting
or using simple data sets or graphs.
 Personal––Procedural
Personal Procedural –High (L4) – Evaluate and Design

Prepared by: Rosario M. Belmi, PhD

Faculty of Science Technology and Mathematics
Philippine Normal University
At Level 4, students can
• use more complex or more
abstract content knowledge,
which is either provided or
recalled, to construct
explanations of more complex
or less familiar events and
processes.
• conduct experiments involving
two or more independent
variables in a constrained
context.
• justify an experimental design,
drawing on elements of
procedural and epistemic
knowledge.
• interpret data drawn from a
moderately complex data set or
less familiar context, draw
appropriate conclusions that go
beyond the data, and provide
justifications for their choices.
Local – Content –High (L5) – Interpret Data and Evidence
At Level 5, students can
• use abstract scientific ideas or
concepts to explain unfamiliar
and more complex phenomena,
events, and processes involving
multiple causal links.
• apply more sophisticated
epistemic knowledge to evaluate
alternative experimental designs
and justify their choices and use
theoretical knowledge to interpret
information or make predictions.
• evaluate ways of exploring a
given question scientifically and
identify limitations in
interpretations of data sets
including sources and the effects
of uncertainty in scientific data.
Local – Content
Local – Content –High (L6) –Explaining phenomenon

High (H) Analysing complex

information or data; synthesising or
evaluating evidence; justifying;
reasoning given various sources;
developing a plan or sequence of
steps to approach a problem.
At Level 6, students can

• draw on a range of interrelated

scientific ideas and concepts from the
physical, life, and earth and space
sciences and use content, procedural,
and epistemic knowledge in order to
offer explanatory hypotheses of novel
scientific phenomena, events, and
processes or to make predictions.
• discriminate between relevant and
irrelevant information and can draw
on knowledge external to the normal
school curriculum.
• can distinguish between arguments
that are based on scientific evidence
and theory and those based on other
considerations.
• evaluate competing designs of
complex experiments, field studies, or
simulations and justify their choices.
SCIENTIFIC LITERACY COMPETENCY

• Explaining phenomena
• Evaluating and designing
scientific enquiry
• Interpreting data and
evidence scientifically
•
Quick Review of the Scientific Literacy Competencies

Scientific Literacy Competencies in Assessment Items

• All these competencies are framed as actions, conveying what the
scientifically literate person both understands and is capable of doing.

Competency 1: Explaining phenomena scientifically

-Recognising, offering and evaluating explanations for a range of natural and technological phenomena.
Competency 2: Evaluating and designing scientific enquiry
-Describing and appraising scientific investigations and proposing ways of addressing questions
scientifically.
Competency 3: Interpreting data and evidence scientifically
-Analysing and evaluating data, claims and arguments in a variety of representations and drawing
appropriate scientific conclusions. Source: https://www.oecd-ilibrary.org
Local– Content –Medium(L3) - Explaining

Specific competency :
Offering explanatory
hypothesis
Global – Procedural–High –Interpret Data and Evidence

Prepared by: Rosario M. Belmi, PhD

Faculty of Science Technology and Mathematics
Philippine Normal University
Personal– Procedural –Medium (L3) –Evaluate Scientific Enquiry

Specific competency :
Distinguishing questions that are possible to
investigate scientifically.
Let us have some PISA Item Writing Workshop!
 Guidelines
1. You will be given a template form
(see Figure on the left) to create 2
PISA items using a mix of all the
elements in the table. Choose
lesson topics that your current
students took a year ago.

2. You have one week to work on

these items and have them checked
by your Science Coordinator or any
colleague.
 Guidelines

3. Once your items are ready, you will upload them in the
provided google form with this link:
https://tinyurl.com/SubmitScienceItems.

4. On December 18, 2021, get ready to present your work to the

coach during 2nd synchronous sessions.

** From the submission folder, six items will be drawn for coaching
exercises. You will be notified if your item was chosen.
This table will come in very handy for you when you decide what context you will consider for your item.
These tables will come in very handy for you when you decide what competency you will consider for your item.
These tables will come in very handy for you when you decide what knowledge domain you will consider for your item.
 Level of Proficiency vis a vis Level of Cognitive Demand

HIGH
Evaluating
Interpreting

MEDIUM
Evaluating
Interpreting

LOW
Explaining
 Complexity of PISA Item Format

LOW
Explaining
Evaluating
Interpreting

MEDIUM TO HIGH
Explaining
Evaluating
Interpreting
Complexity and Range of Knowledge

Prepared by: Rosario M. Belmi, PhD

Faculty of Science Technology and Mathematics
Philippine Normal University
 Let us examine sample constructed PISA-like items!

More Sample Items for you to Study and Benchmark on!

• L1b (Content)Explaining Phenomenon Scientifically Physical-Physics:
Recalling and applying appropriate scientific knowledge
• L1a (Procedural) Explaining Phenomenon Scientifically Physical-Chemistry: Identify
explanatory models and representations / Offering explanatory hypothesis

L1a EASY (Procedural) Physical: Identify explanatory

models and representations / Offering explanatory
hypothesis

A. ocean

B. storage tank

C. Delivery pipeline

D. water distribution system

L2 (Content) Explaining Phenomenon Scientifically Earth and Space: Explaining potential
implication of scientific knowledge to society

Prepared by: Rosario M. Belmi, PhD

Faculty of Science Technology and Mathematics
Philippine Normal University
L3 (Procedural) Interpreting data and evidence scientifically Physical Systems: Analyze and interpret data and
draw appropriate conclusions

Prepared by: Rosario M. Belmi, PhD

Faculty of Science Technology and Mathematics
Philippine Normal University
L4 (Epistemic) Interpreting data and evidence scientifically

Living Systems: Distinguishing arguments that are based on evidence and theory and those from other
considerations
L5 (Procedural) Interpreting data and evidence scientifically Physical-Chemistry: Transform data from one
representation to another

May be turned into a multiple choice

item since the sample is of CR type

Prepared by: Rosario M. Belmi, PhD

Faculty of Science Technology and Mathematics
Philippine Normal University
 Item Writing

•Good luck and happy writing!

This is the end of my
presentation

THANK YOU!!!

You may request a copy of this

material through this email:
belmi.rm@pnu.edu.ph
 References
REFERENCES
• OECD website, http://www.oecd.org/pisa/35070367.pdf.
• OECD (2019), “PISA 2018 Science Framework”, in PISA 2018 Assessment and Analytical
Framework,
• OECD Publishing, Paris. DOI: https://doi.org/10.1787/f30da688-en
• OECD (2018), PISA Results in focus 2015, PISA, OECD Publishing, Paris,
https://www.oecd.org/pisa/pisa-2015- results-in-focus.pdf
• PISA 2018 National Report of the Philippines December 2019 Department of Education
• Rychen, D. and L. Salganik (eds.) (2001), The Definition and Selection of Key Competencies
• Sample PISA Items: http://www.oecd.org/pisa/pisa-2015-science-test-questions.htm