Studies in Educational Evaluation 59 (2018) 278–287

Contents lists available at ScienceDirect

Studies in Educational Evaluation

journal homepage: www.elsevier.com/locate/stueduc

Are the tests scores of the Programme for International Student Assessment T
(PISA) and the National Educational Panel Study (NEPS) science tests
comparable? An assessment of test equivalence in German Schools
Helene Wagner , Inga Hahn, Katrin Schöps, Jan Marten Ihme, Olaf Köller
⁎

IPN - Leibniz Institute for Science and Mathematics Education at Kiel University, Olshausenstraße 62, D-24118 Kiel, Germany

ARTICLE INFO ABSTRACT

Keywords: The aim of this study is to link the science scale of the German National Educational Panel Study (NEPS) with the
Scientific literacy science scale of the Programme for International Student Assessment (PISA). One requirement for a strong linking of
Missing data test scores from different studies is a sufficient similarity of the tests regarding their constructs. The present study
Linking study aims to assess the similarity of the operationalized constructs of the NEPS and PISA scientific literacy tests with
Equipercentile equating
the aim to link the scales of the two tests. A linking study was carried out for this purpose in which 1079 students
PISA
NEPS
worked on the tasks of both studies. The results of the comparison between NEPS and PISA indicated a high
overlap regarding their constructs. However, both studies deal with missing responses differently. The linking
via equipercentile equating showed a high classification consistency which was highest when missing responses
were ignored in both studies.

1. Introduction
In 1997, the Standing Conference of the Ministers of Education and
Cultural Affairs of the Länder in the Federal Republic of Germany
decided on Germany’s regular participation in international large-scale-
assessments. Germany participates in the Trends in International
Mathematics and Science Study (TIMSS) every four years at the end of
primary school as well as in the Programm for International Student
Assessment (PISA) every three years at the end of lower secondary
education.
However, these studies only allow cross-sectional analyses and only
address specific age groups. Until recently, no large-scale study mea-
suring the development of competencies over the lifespan had been
carried out in Germany. The National Educational Panel Study (NEPS;
Blossfeld, 2008) which started in 2009 is the first German attempt to
close this gap by assessing the development of skills and competencies
over the lifespan (Hahn et al., 2013). NEPS strives to connect with
national and international large-scale assessment studies to achieve a
common interpretation of scores (Blossfeld, 2008). However, com-
paring test results from different studies is a challenge because they are
based on different frameworks and their results are not reported on the
same scale. Therefore, the test instruments have to be linked to a
common scale.
This study examines the comparability of the Grade 9 NEPS science
test with the PISA science test. Connecting both tests could extend the
interpretation of their test scores. Until now, no proficiency levels have
been defined in NEPS. Hence, the NEPS results cannot be interpreted
and reported in a criterion-based manner. The link between the NEPS
and PISA tests can allow for classification of the NEPS test scores in the
criterion-based international reference framework of PISA which is well
established in the public educational debate in Germany. The long-
itudinal design in NEPS could help to identify the determinants of
competence acquisition which can predict the performance in the PISA
test. Furthermore, the link between NEPS and PISA could be used to
investigate in NEPS samples to what extent the performance on the
international PISA scale can predict success in upper secondary edu-
cation and the further professional career.
Due to the fact that the NEPS and PISA studies deal differently with
missing responses, we also investigated how the different treatments of
missing values affect the comparability of the test scores and the quality
of the linking. To link the NEPS and PISA tests to a common scale, 1079
9th grade students took both tests in a linking study.
According to Kolen and Brennan (2004) the linking of test scores
from different studies requires sufficient similarities of the tests with
regard to:

⁎
Corresponding author.
E-mail addresses: h.wagner@ipn.uni-kiel.de (H. Wagner), hahn@ipn.uni-kiel.de (I. Hahn), schoeps@ipn.uni-kiel.de (K. Schöps),
ihme@ipn.uni-kiel.de (J.M. Ihme), koeller@ipn.uni-kiel.de (O. Köller).

https://doi.org/10.1016/j.stueduc.2018.09.002
Received 6 October 2017; Received in revised form 2 August 2018; Accepted 5 September 2018
Available online 11 October 2018
0191-491X/ © 2018 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/BY-NC-ND/4.0/).
H. Wagner et al.
• Inferences: To what extent are the scores for the two tests used to
draw similar types of inferences? In other words, to what extent do
the two tests share common measurement goals?
• Populations: To what extent are the two tests designed for testing
similar populations?
• Characteristics and conditions of the measurement: To what extent
do the two tests share common measurement conditions, for ex-
ample, with regard to test format, administration conditions, test
length, etc.?
• Constructs: To what extent do the two tests measure the same
construct?
To closely examine these aspects, the next section will look into the
similarities and differences of the NEPS and PISA frameworks.
2. Comparing the scientific literacy tests of NEPS and PISA
2.1. Inferences
NEPS and PISA have different objectives. The aim of PISA is to
monitor educational systems at the end of lower secondary school in
terms of student performance (OECD, 2013). This goal is realized every
three years by a cross-sectional overview of the educational level of 15-
year-old students. The aim of NEPS is to provide longitudinal data of
the competence development from early childhood to late adulthood in
Germany. In order to achieve this goal the data collection in NEPS is
embedded in a multicohort sequence design (von Maurice, Sixt, &
Blossfeld, 2011) which makes it possible to compare the educational
level of 9th grade students from different cohorts. In other words, de-
spite the different objectives of NEPS and PISA the measurements of
these studies allow to assess the educational level of students at the end
of lower secondary school.
2.2. Target populations
The target population of the NEPS test are 9th grade students (von
Maurice et al., 2011). PISA examines the competence of 15-year-old
students (15 years and 3 months to 16 years and 2 months of age). In
Germany the target population of 15-year-old students for PISA 2012
was defined as the persons born in 1996. The analysis of the compo-
sition of the PISA sample in Germany in 2012 showed that 48% of the
selected students attended Grade 9, 33% of them attended Grade 10
and 11, and 19% of them attended Grade 7 and 8 (Sälzer & Prenzel,
2013). Hence, the target populations in NEPS and PISA are not iden-
tical, but the overlap of both selected samples is high.
2.3. Characteristics and conditions of the measurement
PISA is a cross-sectional study which in 2012 assessed mathematics,
reading, science and financial literacy of 15-year-old students (OECD,
2014b). The 53 items of the science test were split into three clusters
and presented to students with seven mathematics clusters and three
reading clusters in thirteen test booklets. Each booklet consisted of four
clusters with each cluster representing 30 min of test time. Each student
worked on one to two science clusters so that each item was processed
by a sufficient number of students.
NEPS provides longitudinal data on educational processes and
competence development in information and communication technol-
ogies, mathematics, reading and science (von Maurice et al., 2011). The
28 items of the NEPS science test were presented in 2010 in 28 min and
each person got the same items in a fixed sequence (Schöps & Saß,
2013).
In Germany data collection and processing for PISA 2012 and NEPS
2010 were coordinated by the IEA Hamburg. Both tests examined in
this linking study were administered as a paper pencil test. The majority
of the items in NEPS 2010 and PISA 2012 had a closed-constructed
279
Studies in Educational Evaluation 59 (2018) 278–287
response format (OECD, 2014a; Schöps & Saß, 2013). However, PISA
2012 also used an open-constructed response format (32% of the total
number of items).
NEPS and PISA deal differently with missing responses. PISA 2012
used a two-stage procedure for handling missing responses (OECD,
2014b): in the first step, not–reached and not valid items were ignored
and omitted items were scored as incorrect when estimating the item
parameters. In the next step, the estimated item parameters were used
for the estimation of person parameters where missing responses were
scored as incorrect. In contrast, NEPS 2010 ignored all missing re-
sponses for the estimation of item and person parameters (Pohl &
Carstensen, 2012).
A number of studies (De Ayala, Plake, & Impara, 2001; Pohl, Gräfe,
& Rose, 2014; Rose, von Davier, & Xu, 2010) showed that scoring
missing responses as incorrect leads to a bias in the estimation of
parameters and to the overestimation of the reliability. Based on these
results we assume that ignoring the missing responses in NEPS and PISA
will increase the comparability of their test scores (hypothesis two) and
their scales (hypothesis four), and hence the quality of linking (hy-
pothesis six).
2.4. Operationalized constructs: comparing the contents of the science tests
of NEPS and PISA
The definition of scientific literacy used by NEPS includes aspects of
the concept of competence as defined by Weinert (2001), and of the
concepts of scientific literacy developed by the American Association for
the Advancement of Science (American Association for the
Advancement of Science, 2009) and by PISA (OECD, 2006). Therefore,
the NEPS scientific literacy framework has a substantial overlap with
the scientific literacy framework from PISA 2012 (Fig. 1).
Fig. 1 shows that the frameworks of both studies differ in the
number of components used for assessing scientific literacy: The fra-
mework of NEPS only considers the content-related components which
are related to the knowledge of science (KOS) in PISA, and process-related
components which are related to the knowledge about science (KAS) in
PISA. The PISA framework differentiates further and also distinguishes
between the competencies identifying scientific issues, explaining phe-
nomena scientifically and using scientific evidence. At this point, it can be
concluded that the frameworks of the two studies differ in their con-
ceptual scope. But how different are they on the task level?
This question can be examined using the theory of bias and
equivalence of van de Vijver (1998), which was modified for the pur-
poses of equivalence research in the linking studies by Pietsch, Böhme,
Robitzsch, and Stubbe (2009). Pietsch et al. (2009) suggested assessing
the similarity of the operationalized constructs of two tests by regarding
their conception, their dimensional structures and their scales.
According to Pietsch et al. (2009) two tests are equivalent regarding
their conception when their constructs have equivalent frameworks. In
order to analyze the conceptual equivalence of the scientific literacy in
NEPS and PISA (Wagner, Schöps, Hahn, Pietsch, & Köller, 2014) seven
experts in the field of science didactics familiar with large-scale as-
sessments classified the NEPS items according to the categories of KOS
and KAS and to the competencies in PISA. The results showed that 79
percent of the NEPS items could be assigned to the contents of the PISA
framework. However, according to five of the seven raters some of the
KOS components in PISA (earth and space systems and technology
systems) were not covered by NEPS items.
3. Linking methods and linking studies
Different methods of linking can be applied depending on the level
of equivalence of the two tests. Mislevy (1992) and Linn (1993) dif-
ferentiate between five types of linking: equating, vertical scaling, con-
cordance, projection and moderation. Fig. 2 illustrates the different
linking methods which, in terms of their applicability depend on the
H. Wagner et al.
Fig. 1. Framework of the scientific literacy by NEPS and PISA. The corresponding equivalents were underlined.
degree of similarity of the test instruments to be linked. In addition,
Fig. 2 shows the linking continuum (Ryan & Brockmann, 2009), which
reflects the above-mentioned methods’ strength of linking. The linking
continuum can be interpreted as follows: the more similar the tests are
in terms of inferences, target populations, measurement characteristics,
and operationalized constructs, the stronger a type of linking method
can be applied, and thus the closer these tests can be connected.
According to Ryan and Brockmann (2009) equating is the strongest
type of linking. Equating can only be applied, if the tests are similar in
all features named in the Kolen and Brennan (2004) approach. The
linking via equating makes it possible to predict the scores from one test
to another and vice versa. However, this method also has one dis-
advantage: It is sensitive to irregularities in the distribution of the test
scores. For example, the equating relationship cannot be determined for
score ranges that exceed the highest observed score and scores that fall
below the lowest observed score. One possibility of approaching the
problem is by pre- or post-smoothing the equipercentile equivalents
(Livingston, 2004).
Worldwide there have been different approaches to link
Fig. 2. Dependence of the linking methods on the degree of similarity of the test instruments.
280
Studies in Educational Evaluation 59 (2018) 278–287
international data with national data since the first PISA survey in
2000. Depending on the goal of the linking studies, they can be clas-
sified into three groups (Nissen, Ehmke, Köller, & Duchhardt, 2015):
studies with the goal of explaining differences in outcome (Wu, 2010),
studies with the goal of explaining differences in proficiency levels
(Hambleton, Sireci, & Smith, 2009), and lastly, studies to locate the
outcome of the national study in an international reference (Cartwright,
Lalancette, Mussio, & Xing, 2003; Pietsch et al., 2009).
In their study, Nissen et al. (2015) also wanted to locate mathe-
matical scores of NEPS on the TIMSS scale. This study is relevant to our
study insofar that it aimed to link tests developed for similar but not the
same populations. Nissen et al. (2015) compared different linking
methods by applying them to the mathematics tests from TIMSS 2011
(students at the end of Grade 4) and NEPS 2010 (beginning of Grade 5).
The comparison of the equipercentile equating with the IRT linking
method (based on a sample of 733 4th graders) showed a higher clas-
sification consistency between the NEPS and TIMSS results on the
TIMSS proficiency scale when the equipercentile equating was used
(percentage matching of 44%). Furthermore the refered study can be
H. Wagner et al.
used as a comparison for our linking results as the target populations of
NEPS and PISA showed a high overlap but are not identical.
Worldwide there have been different approaches to link the tests in
mathematics and reading. Hanushek and Wößmann (2015) showed
that the scientific literacy measure in PISA is a far better predictor for
economic growth of a country than the results for mathematics and
reading. The linking of the scientific scales of NEPS and PISA would
allow to examine the development of scientific competence in Germany
in the context of its economic growth. Therefore, the linking of scien-
tific literacy in NEPS and PISA is important not only with regard to
educational research but also with regard to economic research ques-
tions.
4. Research questions
A basic requirement for linking two tests by equating is a high si-
milarity of their test constructs. According to Pietsch et al. (2009) the
similarity of the operationalized constructs can by assessed by re-
garding their conception, their dimensional structures and their scales.
The assesment of the first aspect – the conceptual equivalence of the
NEPS and PISA tests – was part of another study and showed a high
overlap of the scientific literacy frameworks (Wagner et al., 2014).
Analysis of the two other aspects – the equivalence of the dimensional
structures and scales – are part of the present study. The comparison of
the NEPS and PISA frameworks in the present paper showed that the
studies differ in the response formats they use and in the treating of
missing data. In particular, the influence of this last aspect has to be
investigated regarding the dimensional and the scalar equivalence of
NEPS and PISA and the linking of their scales.
According to Pietsch et al. (2009) the dimensional equivalence
means that two tests have the same factor structure, whereas the scalar
equivalence means that the scores of two tests have the same meaning.
To address our questions concerning the dimensional equivalence, we
examine (1) whether the items of the NEPS and PISA tests measure the
same construct of scientific literacy and (2) to what extent a different
handling of missing data in NEPS and PISA influences the comparability
of their science scores.
To analyze the scalar equivalence of both tests, we investigate (3) to
what extent the scales in NEPS and PISA are equivalent regarding their
descriptive statistics such as means, standard deviations, skewness, and
kurtosis. Furthermore, we inspect (4) to what extent a different hand-
ling of missing data in NEPS and PISA influences the comparability of
their scales.
Subsequently, the scales of the two studies are linked and compared
regarding their descriptive statistics. (5) We also gauge the classification
consistency between the linked test scores in NEPS and PISA according
to the PISA 2012 international benchmarks and examine (6) to what
extent a different handling of missing data in NEPS and PISA influences
the linking of the science scores.
5. Method
5.1. Data collection
Although the assessments of NEPS and PISA showed an overlap
concerning their framework conceptions, they have no common items.
Accordingly, the science scores of these studies cannot be reported on
the same scale. Therefore, a linking study was conducted in four federal
states in Germany using a single group design (Fig. 3) to assess the
construct equivalence in NEPS and PISA and to link their scientific
literacy scales.
The study was carried out on two consecutive days parallel to the
PISA survey in the spring of 2012. On the first day, every student
completed one of the five PISA booklets with science and mathematics
items (two booklets with two science clusters and three booklets with
one science cluster) within two hours. In total, students worked on 53
281
Studies in Educational Evaluation 59 (2018) 278–287
Fig. 3. Single group design in the linking study.
science items split into three clusters. On the second day, the students
took the NEPS science test which consisted of 28 items. The test
booklets used in the presented study were the same as in the main
studies of PISA 2012 and NEPS 2010.
The data collection and processing in the main NEPS and PISA
studies as well as in the linking study was coordinated by the IEA
Hamburg. The sample consisted of 1528 9th grade students from 65
schools who participated in the secondary school program Increasing the
Efficiency of Teaching in Mathematics and Science Education in Secondary
School (SINUS; Prenzel & Ostermeier, 2006).
Overall N = 1079 ninth grade students (50% female) took both the
NEPS and the PISA science tests. The age mean of the sample was
M = 15.5 years and the standard deviation was SD = 0.55. The analysis
of the age composition of the sample showed that 46% of the tested
students were born in 1996 and thus corresponded to the selection
criterion for PISA 2012 in Germany. Eight percent of students were
born bevor 1996 and 46% of students were born after 1996.
5.2. Scoring and data processing
Germany has participated in PISA (which takes place every three
years) since the year 2000. Each PISA cycle examines a major domain in
depth, so two-thirds of the testing time is devoted to this domain; the
other domains provide a summary profile of skills. Major domains were
reading literacy in 2000 and 2009, mathematical literacy in 2003 and
2012 and scientific literacy in 2006 and 2015. PISA 2012 reused some
of the items developed in 2006, therefore the scoring of the PISA data in
this study was conducted by applying the coding rules of PISA 2006
(OECD, 2009). The scoring of the NEPS data was carried out using the
NEPS 2010 coding rules (Schöps & Saß, 2013).
5.3. Analysis of the dimensional equivalence
Our first research question addressed the extent to which the items
of NEPS and PISA measure the same construct of scientific literacy. In
order to assess this question, the software ConQuest (Wu, Adams,
Wilson, & Haldane, 2007) was used to compute the item parameters of
both studies by assigning the items to the corresponding tests as two
dimensions. In a first step, the two dimensions were correlated to
analyze the dimensional equivalence of NEPS and PISA. Afterwards the
two dimensional model (NEPS and PISA) was compared with the one
dimensional model (scientific literacy) with regard to the information
criteria.
The information criteria (AIC, BIC, SABIC) are measures of the
goodness of fit of an estimated statistical model. The Akaike informa-
tion criterion (AIC) takes into account the number of estimated para-
meters in this model in addition to the likelihood of the model to be
analyzed. Compared to the AIC, the Bayesian information criterion
(BIC) strongly penalizes the number of estimated parameters. The
H. Wagner et al.
sample-size adjusted BIC (SABIC) places a penalty for adding para-
meters based on sample size but less strongly than the BIC. Given a set
of candidate models for the data, the preferred model is the one with
the minimum information criteria value. Raftery (1995) proposes to
interpret the difference of ΔBIC≥10 as very strong evidence for a better
model fit.
In the next step the dimensional equivalence of NEPS and PISA was
analyzed by examining the assumption of local item independence (LII),
one of the cornerstones of the item response theory model (Embretson
& Reise, 2000). The LII means that the observed items are conditionally
independent of each other given an individual score on the latent
variable (Henning, 1989). If the items of different tests measure dif-
ferent constructs, the LII is violated. Any statistical analysis based on
the Rasch model is unjustified if the assumption of local item in-
dependence is violated. Moreover, the violation of the local item in-
dependence is the evidence that the tests measure different constructs
and are not equal.
One of the statistical procedures to identify the local item de-
pendency is the computation of the partial correlation index (PRT) from
Huynh, Michaels, and Ferrara (1995): the predicted value from the
linear regression on the raw score of the total test (in our study the
weighted likelihood estimates (WLE)) is subtracted from the item raw
score. The correlations between the residuals of items are partial cor-
relations and the mean of the partial correlations is the index of the
local item dependency. A PRT index above the critical value of 0.2
(Chen & Thissen, 1997) indicates a violation of the local independence.
In order to examine the comparability of scores in the main studies
of NEPS and PISA, the dimensional equivalence in the linking study was
analyzed by handling missing data in a study-specific way when esti-
mating item parameters. This means that not-reached and not valid
items were ignored in both studies, whereas the omitted items were
ignored in NEPS and were scored as incorrect in PISA. To examine the
research question two addressing the influence of handling missing data
on the comparability of the NEPS and PISA scores, the dimensional
analyses described earlier were carried out in two additional ways. In
the first case the missing responses in NEPS and PISA were ignored
because we assumed that this would increase the comparability. To
ensure that this increase cannot be explained only by the equal hand-
ling of missing data, the analysis of the dimensional equivalence was
performed when missing responses in both studies were scored as in-
correct.
5.4. Analysis of the scalar equivalence
To assess the third research question regarding the scalar equiva-
lence in NEPS and PISA, their raw scores were analyzed based on the
1 PL Rasch model with fixed item parameters taken from NEPS 2010
(Schöps & Saß, 2013) and PISA 2012 (OECD, 2014a). Due to the aim of
this study to classify the NEPS scores into the international reference
framework of PISA, PISA items were fixed to the international item
parameters. To estimate the scientific literacy in NEPS and PISA five
plausible values were drawn per student and test and linearly trans-
formed to the mean of 500 and a standard deviation of 100. The mean,
skewness and kurtosis were compared to assess the scalar equivalence
of the NEPS and PISA science tests.
As for the testing of dimensional equivalence, the scalar equivalence
in the presented study was analyzed by handling missing responses in a
study-specific way when estimating person parameters. This means that
the missing responses in NEPS were ignored and the missing responses
in PISA were scored as incorrect. However, the fourth research question
in our work addressed the influence of handling missing data on the
comparability of the NEPS and PISA scales. Consequently, when ana-
lyzing the scalar equivalence the missing responses in NEPS and PISA
were first ignored and, in a second step, scored as incorrect.
282
Studies in Educational Evaluation 59 (2018) 278–287
5.5. Linking procedures
The linking of the NEPS and PISA scores in the present study was
conducted via equipercentile equating (e.g. Cartwright et al., 2003;
Nissen et al., 2015; van den Ham, Ehmke, Nissen, & Roppelt, 2016).
This procedure is based on the idea that the scores of two tests with the
same percentile rank are declared as equivalent (Kolen & Brennan,
2004). For example, if a score of 425 is equal to a percentile rank of 10
on the NEPS scale, and a score of 496 is equal to percentile rank of 10
on the PISA scale, then the scores 425 and 496 are considered
equivalent. Due to the sensitivity of equipercentile equating to irregu-
larities in the distribution of the test scores (Livingston, 2004) the NEPS
equivalents were post smoothed with a value of 0.3. Post smoothing
means that equipercentile equating is performed on the basis of ob-
served distributions and the equating relation is smoothed. This step
produces a smoothed distribution with nonzero probability at the
highest and lowest score levels (Kolen & Brennan, 2004). In our study
equipercentile equating was carried out for each plausible value by
using the computer software LEGS (Brennan, 2003). Afterwards the
linking results were averaged.
5.6. Analysis of the classification consistency
In the last step, the students were classified according to the inter-
national benchmarks of PISA 2012 (OECD, 2013) based on the PISA
scores and the NEPS equivalents on the PISA metric. To analyze re-
search question five regarding the classification consistency between
the linked test scores in NEPS and PISA, the percentage matching (the
percentage of the students assigned to the same benchmark) was cal-
culated. Another measure of the classification consistency in our study
was the Cohen’s coefficient Kappa (Cohen, 1960). Landis and Koch
(1977) propose the following labels assigned to the corresponding
ranges of kappa: “.21-.40″ fair, “.41-.60″ moderate, “.61-.80″ sub-
stantial and “.81-1.00″ almost perfect agreement.
As for the testing of equivalence of the NEPS and PISA results the
missing data were handled study-specific by ignoring the missing re-
sponses in NEPS and scoring the missing responses in PISA as incorrect
when linking the scales of both studies. In order to examine research
question six addressing the influence of handling missing data on the
classification consistency between NEPS and PISA, the missing re-
sponses in both studies were first ignored and than scored as incorrect.
6. Results
6.1. Assessing the dimensional equivalence
The aim of this section is to examine the first research question
regarding the dimensional equivalence of the NEPS and PISA scientific
literacy tests. To achieve this objective, the comparability of the NEPS
and PISA scores were analyzed in a first step by correlating the two
dimensions. The correlation between NEPS and PISA was r = .85, in-
dicating a high relation between the scores of both tests. Furthermore
the estimated correlation showed that the majority (72%) of the var-
iance in each test can be explained by the other test.
In the next step, the dimensional equivalence of the NEPS and PISA
tests was analyzed by comparing the fit indices of the two dimensional
model with the one dimensional model, which considered all items as
indicators of the scientific literacy and did not differentiate between the
tests. Table 1 shows the results of the factor analyses.
The information criteria (AIC, BIC and SABIC) of the one dimen-
sional model were larger than the information criteria of the two di-
mensional model. This result indicates that the factor of scientific lit-
eracy cannot explain the systematic item variance completely. This
might be caused by the difference of the measurement conditions be-
tween NEPS and PISA. In order to examine this assumption, we address
the influence of the different handling of missing responses on the
H. Wagner et al. Studies in Educational Evaluation 59 (2018) 278–287

Table 1
Summary of factor analysis of the NEPS and PISA data.
N of Log likelihood AIC BIC SABIC
parameters

2 dimensions 109 65572 65790 66283 65937

(NEPS & PISA)
1 dimension 107 65680 65894 66378 66038
(scientific literacy)

Note. AIC = Akaike information criterion; BIC = Bayesian information cri-

terion; SABIC = Sample-Size Adjusted BIC; * = p < .05.

Fig. 5. Average percentage distribution of missing responses in NEPS and PISA.

Fig. 4. The partial correlations of item pairs in NEPS and PISA.

comparability of the NEPS and PISA scores in the next section.

The factor analysis showed that the NEPS and PISA scores cannot be
mapped onto one common dimension. But does this result indicate that
the NEPS and PISA tests measure different constructs? This question
was analyzed by examining the assumption of local item independence
for the NEPS and PISA items. Fig. 4 provides information about the Fig. 6. Average percentage distribution of invalid responses in NEPS and PISA.
correlations between the item residuals in NEPS and PISA after the
individual score on the latent variable (scientific literacy) was sub- Our analyses showed that the relation between the two dimensions
tracted from the item raw score. seems to depend on handling missing data: when missing responses
The diagram shows that only few correlations exceeded the critical were ignored in both tests the correlation between NEPS and PISA in-
value of .2. The mean of the partial correlations was −.02 and in- creased to r = .90. Thus, the NEPS and PISA tests shared 9% more of
dicated the one-dimensionality of the NEPS and PISA items. Thus, the the variance when missing responses were ignored. This increase
analysis of local item independence in our study showed no evidence cannot be explained only by the equal handling of missing data. When
that the NEPS and PISA tests measure different constructs. missing responses in both tests were scored as incorrect the correlation
between NEPS and PISA decreased slightly (r = .83).
The analyses of the factor structure also showed a relation between
6.2. Assessing the influence of handling missing data on the dimensional
the handling of missing data and the comparability of the NEPS and
equivalence
PISA scores. If the missing responses in both tests were ignored
(Table 2), the two dimensional model fitted the observed data better
Our second research question addressed the influence of handling
missing data on the dimensional equivalence of the scientific literacy
scores in NEPS and PISA. Figs. 5 and 6 provide the average percentage
Table 2
distribution of missing and invalid responses in NEPS and PISA. They
Summary of factor analysis of the NEPS and PISA data (missing responses are
show that the PISA test has significantly (p < .05) more missing data ignored).
than the NEPS test. In PISA 2012 missing responses were scored as
incorrect when estimating the person parameters (OECD, 2009). N of Log likelihood AIC BIC SABIC
parameters
However, NEPS 2010 ignored missing responses when estimating the
item and person parameters (Pohl & Carstensen, 2012). 2 dimensions 109 63806 64024 64517 64171
In order to assess the influence of handling missing data on the (NEPS & PISA)
comparability of the NEPS and PISA scores (research question two), the 1 dimension 107 66865 64079 64563 64223
(scientific literacy)
analyses of the dimensional equivalence presented in the previous
section were conducted by handling missing data in two ways. In the Note. AIC = Akaike information criterion; BIC = Bayesian information cri-
first case the missing responses in both tests were ignored and in the terion; SABIC = Sample-Size Adjusted BIC; * = p < .05.
second case they were scored as incorrect.

283
H. Wagner et al. Studies in Educational Evaluation 59 (2018) 278–287

Table 3 Table 5
Summary of factor analysis of the NEPS and PISA data (missing responses are Descriptive statistics for NEPS and PISA science scores and results of the
scored as incorrect). equipercentile linking (missing responses are ignored).
N of Log likelihood AIC BIC SABIC Mean (SD) SE Skewness SE Kurtosis SE
parameters
NEPS scores 539.45 (86.64) 2.64 −0.01 0.07 −0.14 0.15
2 dimensions 109 69702 69920 70413 70067 PISA scores 600.87 (75.12) 2.29 −0.06 0.07 −0.13 0.15
(NEPS & PISA) NEPSPISA 600.86 (75.05) 2.28 −0.06 0.07 −0.13 0.15
1 dimension 107 69843 70057 70541 70201
(scientific literacy) Note. SD = standard deviation; SE = standard error.

Note. AIC = Akaike information criterion; BIC = Bayesian information cri-

Table 6
terion; SABIC = Sample-Size Adjusted BIC; * = p < .05.
Descriptive statistics for NEPS and PISA science scores and results of the
equipercentile linking (missing responses are scored as incorrect).

than the one dimensional model, similar to the analysis in the previous Mean (SD) SE Skewness SE Kurtosis SE
section. But the difference between the information criteria of models
NEPS scores 521.92 (88.21) 2.69 −0.05 0.07 0.02 0.15
decreased when missing responses were ignored. PISA scores 580.62 (81.64) 2.49 −0.07 0.07 −0.10 0.15
Then again, scoring missing responses as incorrect led to the highest NEPSPISA 580.91 (81.94) 2.49 −0.07 0.07 −0.01 0.15
difference of information criteria and thus to the lowest comparability
of the NEPS and PISA scores (Table 3). Compared to the results in the Note. SD = standard deviation; SE = standard error.
previous section (when missing responses were handled study-specific)
the results of this section showed that ignoring missing responses in missing data study-specific, the difference between the means of the
both tests increased the relation between the NEPS and PISA scores. At person parameters in NEPS and PISA increased.
the same time, scoring missing responses in both tests as incorrect de- The average correlation of the competency scales in NEPS and PISA
creased the relation between the NEPS and PISA scores. decreased to .82, when missing responses in both tests were scored as
incorrect. The descriptive statistics in Table 6 (upper two lines) show
6.3. Assessing the scalar equivalence the same difference between the means of person parameters in NEPS
and PISA when ignoring missing responses in both tests.
In order to examine the construct equivalence of the NEPS and PISA
tests, the third research question regarding the comparability of their
6.5. Assessing the classification consistency between NEPS and PISA
scales is analyzed in this section. The average correlation of the NEPS
and PISA scales was .85. The descriptive statistics for the two dimen-
The aim of this study was to link the scientific literacy scales of
sions are provided in the upper part of Table 4. The statistics showed
NEPS and PISA. The results of the linking are provided in the last line of
significant differences (p < .05) between the means of person para-
Table 4. The descriptive statistics showed that the mean as well as the
meters in NEPS and PISA. The students achieved higher proficiency
skewness and kurtosis of the NEPS equivalents on the PISA metric
values in the PISA test than in the NEPS test. The data of both tests were
strongly resembled the PISA scale statistics.
normally distributed and did not differ in skewness and kurtosis.
In the next step the students were classified according to the PISA
Furthermore, the analyses showed that the mean of person para-
international benchmarks in science. The last line (for NEPS) and the
meters in PISA was higher than the ability mean of the German sample
last column (for PISA) in Table 7 show that both tests assigned ap-
in 2012 (524; OECD, 2014a). This might be caused by the participation
proximately the same number of students to the PISA 2012 interna-
of selected schools in the primary school program SINUS (Dalehefte
tional benchmarks. The distribution of students on the proficiency le-
et al., 2014), which might have led to a higher average ability than the
vels based on the linked NEPS and PISA scores was equivalent
ability of the reference sample.
(χ² = 0.72; df = 5, p > .05).
The matching of the students according to the PISA 2012 interna-
6.4. Assessing the influence of handling missing data on the scalar
tional benchmarks was calculated in order to analyze the classification
equivalence consistency between NEPS and PISA addressed in research question
five. This was done based on the linked NEPS and PISA scores. In
In order to assess the influence of handling missing data on the
contrast to the distribution of students on the proficiency levels, the
comparability of the NEPS and PISA scales (research question four), the
individual assignment of students differed between the tests. The per-
descriptive statistics reported in the previous section were computed by
centage matching of NEPS and PISA lay between 42% on the
firstly ignoring missing responses in both tests and by secondly scoring
them as incorrect.
Table 7
The average correlation of the competency scales in NEPS and PISA Percentage of students classified according to the PISA 2012 international
increased to .91, when missing responses in both tests were ignored. benchmarks in science.
Table 5 (upper two lines) provides the descriptive statistics for the
NEPSPISA
NEPS and PISA scores. Compared to the results computed by handling
I II III IV V VI Total number
Table 4 of persons
Descriptive statistics for NEPS and PISA science scores and results of the
equipercentile linking. PISA I 42 11 23
II 42 50 13 1 105
Mean (SD) SE Skewness SE Kurtosis SE III 16 36 55 22 5 293
IV 3 30 55 28 2 361
NEPS scores 539.45 (86.64) 2.64 −0.01 0.07 −0.14 0.15 V 1 21 57 44 240
PISA scores 580.62 (81.64) 2.49 −0.07 0.07 −0.10 0.15 VI 10 54 58
NEPSPISA 580.69 (81.64) 2.49 −0.06 0.07 −0.09 0.15 Total 100 100 100 100 100 100
Total number of persons 24 105 290 362 240 59
Note. SD = standard deviation; SE = standard error.

284
H. Wagner et al.
proficiency Level I and 57% on the proficiency Level V. The mean
percentage matching was 52%. The Cohen’s coefficient Kappa (a mea-
sure of consistency) was k = .40, so the matching between the tests can
be rated as fair (Landis & Koch, 1977).
6.6. Assessing the influence of handling missing data on the classification
consistency
Research question six addressed the influence of handling missing
data on the classification consistency. The missing data were handled in
two ways to examine this question: in the first case the missing re-
sponses in both tests were ignored and in the second case they were
scored as incorrect.
The distribution of students on the proficiency levels of PISA based
on the linked scores was also equivalent (χ² = 0.95; df = 5, p > .05)
when missing responses were ignored in both tests. However, the mean
of the percentage matching increased up to 60% (Min. 29%, Max. 70%)
when missing responses were ignored. Also, the Cohen’s coefficient
Kappa increased to k = .55 and showed a higher classification con-
sistency compared to the study-specific handling of missing data.
According to Landis and Koch (1977) the matching between the tests in
this case can be rated as moderate.
Similar to ignoring and to handling missing data study-specifically,
scoring missing responses as incorrect led to an equivalent distribution
of students on the proficiency levels of PISA (χ² = 3.25; df = 5,
p > .05). Compared to the previous results, the average of the percen-
tage matching between NEPS and PISA decreased to 48% (Min. 33%,
Max. 53%). The analysis of the classification consistency with Cohen’s
Kappa (k = .36) showed a fair matching between NEPS and PISA.
7. Summary of results and discussion
The aim of this study was to link the NEPS 2010 science test with
the PISA 2012 science test. As a basis for this analysis, the compar-
ability of the scientific literacy scores was examined for both tests,
taking into account the study-specific way of handling missing data.
The dimensional equivalence and the scalar equivalence of the test
scores were assessed for this purpose.
7.1. Are the NEPS and PISA tests equivalent?
In order to examine the possibility of linking the scientific literacy
scales of NEPS and PISA, their tests were compared with regard to in-
ferences, target populations, measurement conditions, and oper-
ationalized constructs. The comparison of the inferences in NEPS and
PISA showed an overlap between the studies concerning the assessment
of the students’ educational level at the end of secondary school.
Furthermore, the target populations in these studies were not equiva-
lent, but our analyses showed that their overlap was high.
The next aspect in assessing the equivalence of NEPS and PISA was
the comparison of the measurement’s characteristics showing that the
tests differ with regard to the response format and the handling of
missing data. A number of studies (De Ayala et al., 2001; Pohl et al.,
2014; Rose et al., 2010) showed that scoring missing responses as in-
correct can lead to a bias in the estimation of parameters. Our analysis
of the dimensional equivalence indicated that ignoring the missing re-
sponses in NEPS and PISA led to the highest comparability of their test
scores. Furthermore, the presented results showed that the equal
handling of missing data cannot explain the increase of comparability
between the tests. From our point of view these results can be inter-
preted as evidence that ignoring missing responses leads to more ac-
curate estimates of the person parameters. Thus, the findings in this
study support the conclusions of the studies mentioned in this para-
graph.
Also, the difference in the response formats may influence the
comparability of the tests. The comparison of the average percentage
285
Studies in Educational Evaluation 59 (2018) 278–287
distribution of missing and invalid responses in NEPS and PISA showed
that the PISA test had significantly more missing data than the NEPS
test. This difference in the distribution of missing responses between the
tests might partly be explained by the difference in response formats
used in NEPS and PISA, because test persons often tend to omit items
with open-constructed response formats (De Leeuw, Hox, & Huisman,
2003). In connection with scoring missing responses as incorrect in
PISA, this tendency could lead to an overestimation of the difficulty of
open-response items. Since the linking study focuses on the competence
of students and not on the characteristics of the individual items, the
difference in the response formats should not have played a major role
in linking the results of NEPS and PISA.
The last aspect in assessing the equivalence of NEPS and PISA was
the investigation of the construct equivalence of the tests. The analyses
in our study showed that the constructs in the NEPS and PISA tests are
not equivalent but that they are comparable. This is indicated by a high
relation and a high amount of shared variance between the tests, as well
as by one common construct of scientific literacy in both tests. The
investigation of the scalar equivalence of NEPS and PISA also indicated
a high comparability of their scales.
Our analyses showed that although the tests of NEPS and PISA are
not equivalent, they are nevertheless comparable with regard to their
inferences, their target populations, their measurement conditions, and
their operationalized constructs. In this regard, the comparability of the
NEPS and PISA tests provides a basis for linking their scientific literacy
scores by equipercentile equating.
7.2. The linking and classification consistency
The analyses of the classification consistency addressed in research
question five showed equivalent distribution of students on the profi-
ciency levels of PISA on the basis of the linked NEPS and PISA scores.
Also, the Cohen’s Kappa k = .40 and the percentage matching of 52%
showed a good approximation between the tests.
The classification consistency is influenced by the correlation be-
tween the tests, their reliabilities and the number of proficiency levels
(Ercikan & Julian, 2002; Pietsch et al., 2009). Pietsch et al. (2009)
showed that the expected consistency between two tests with five
proficiency levels is 40% if the tests have reliabilities of Rel₁ = 1 and
Rel₂ = .8 and the correlation between the tests is r = .85. Furthermore
the classification accuracy decreases by 10% for an increase by one
proficiency level (Ercikan & Julian, 2002). PISA has six proficiency
levels. Therefore, the percentage matching of 52% exceeded the ex-
pected value and can be rated as very satisfactory. This value also ex-
ceeded the percentage matching between NEPS and TIMSS found in the
Nissen et al. (2015) study. Thus, the assessed classification consistency
in our study indicates that linking the NEPS and PISA scientific literacy
scales leads to a high concordance regarding the competence of stu-
dents. However, similar to the results of Cartwright et al. (2003) and
Nissen et al. (2015) there are some minor differences between NEPS
and PISA regarding the individual classification into the benchmarks.
Thus, inferences from the NEPS and PISA results can be drawn for
groups of students but not for individual students.
The classification consistency between NEPS and PISA addressed in
research question six was analyzed under different conditions of
handling missing data. The analyses showed that the percentage
matching between NEPS and PISA was higher when missing responses
were ignored. This way of handling missing data led to an increase in
the average consistency by 8% compared to the study-specific handling
of missing data. According to the study of Pietsch et al. (2009) a growth
of 8% in the classification consistency can be expected for the NEPS and
PISA tests by increasing the correlation between the two tests to 1. Thus
the results of linking in this study indicate that ignoring missing re-
sponses not only increased the comparability of the tests but also the
consistency of the classification. The last PISA assessment (OECD,
2016) partially eliminated the differences between PISA and NEPS in
H. Wagner et al.
handling missing data by ignoring not-reached items when estimating
person parameters. This change in the framework of PISA 2015 in-
creased the comparability of the NEPS and PISA results.
7.3. Limitations
This study examined the equivalence of the NEPS test from 2010
and the PISA test from 2012. The NEPS and PISA tests have changed in
many ways since 2012: new items were developed for the NEPS as-
sessment in 2014 in addition to the items of 2010 so that a three-stage
(easy, average and difficult) test could be administered and could still
be linked to the test from 2010 by using link items. PISA 2015 in-
troduced the following changes in the test administration and scaling
(OECD, 2016): the assessment mode (computer-based instead of paper-
pencil), the scaling model (two-parameter model instead of one-para-
meter model), the handling of differential item functioning across
countries (calibration for a number of country-by-cycle-specific devia-
tions from the international item parameters instead of ignoring the
“dodgy” items for some countries), the handling of not-reached items
(dealing as not administered instead of as wrong answers when esti-
mating the person parameters) and finally the changes in the frame-
work of scientific literacy (e.g., the KOS component technology systems
was excluded from the framework). These changes are not relevant for
the linking carried out in this work, but they must be taken into account
in the future implementation of linking NEPS and PISA.
According to the PISA 2015 report (OECD, 2016) the scientific lit-
eracy tests from 2006, 2009, 2012 and 2015 use the same science
performance scale and therefore the comparison of the scores across
time is possible. Robitzsch et al. (2016), however, showed in their
work, that the change from paper pencil to computer tests could have
biased the trend estimation of the German data in science. They sug-
gested using the field test data for the trend estimation. We would
follow this recommendation and suggest including field test data for the
future linking of NEPS and PISA.
Robitzsch et al. (2016) also investigated the question of how far the
new interactive PISA 2015 tasks could have biased the trend estimation
of the German data in science. For this purpose, the authors examined
the German trend in science based only on the old (less interactive)
items from PISA 2012. The analyses showed no change in the trend
estimation compared to the trend estimation based on all items. Fur-
thermore, the dimension formed by the old items correlates closely to
one with the dimension consisting of new items. It can be concluded
that the interactive format of the new tasks did not change the science
performance scale in PISA 2015.
A second limitation of the study concerns the selectivity of the
tested sample. The choice of the sample can be justified by the second
goal of this study, namely the investigation of the long-term effect of
the SINUS program. The aim of this school program is to more effi-
ciently support teachers in teaching mathematics and science. This
could lead to a higher performance in science by students at these
schools. At the same time, the choice of the sample offers the oppor-
tunity to examine which contents of this program have an influence on
the students’ performance. Another limitation concerns the test design,
namely that the NEPS and PISA tests were administered over the course
of two days. A training effect or a decrease in motivation might be a
possible consequence.
7.4. Practical implications
Linking the NEPS and PISA scientific literacy scores allows classi-
fying the NEPS test scores in the criterion-based reference framework of
PISA. This and the longitudinal assessment in NEPS could help to learn
more about the development of scientific literacy. For example, the
linking function presented in our study could be used to identify the
factors of competence development measured longitudinally in the
NEPS main studies, which can predict performance in the PISA test.
286
Studies in Educational Evaluation 59 (2018) 278–287
Furthermore, the presented linking function could be used to in-
vestigate, to what extent the performance in PISA can predict the suc-
cess in upper secondary education and in the further professional career
measured by NEPS.
The linking function between the NEPS and PISA science scores can
be used to classify the NEPS scores within the international benchmarks
of PISA. Until now, proficiency levels have not been defined in NEPS.
Haschke, Kampa, Hahn, and Köller (2017) developed proficiency
standards for adults based on the NEPS scientific literacy test using the
Item Descriptor Matching method. The linking function presented here
provides a basis for the proficiency standards in NEPS with regard to
the outcome that students should have achieved in secondary school. In
connection with the longitudinal design of NEPS, the proficiency
standards can be used to analyze the influence of the educational de-
velopment in Germany on the competencies internationally considered
crucial for secondary school.
Our study showed the influence of handling missing data on the
comparability of two tests and their linking. The influence of this aspect
on the comparability of two tests has not yet been investigated.
Therefore, the conducted study is highly relevant with regard to future
comparisons of the results from different studies.
Acknowledgements
The study was funded by the Centre for International Student
Assessment (ZIB) and the Federal Ministry of Education and Research
(BMBF) in Germany. The authors would like to thank the ZIB and BMBF
for their support.
References
American Association for the Advancement of Science (2009). Benchmarks for science
literacy. Project 2061. New York: Oxford University Press.
Blossfeld, H.-P. (2008). Education as a lifelong process. A proposal for a national educational
panel study (NEPS) in Germany. Part B: Theories, operationalization and piloting strate-
gies for the proposed measurements. Bamberg: Universität Bamberg.
Brennan, R. L. (2003). LEGS: A computer program for linking with the randomly equivalent
groups or single- group design. Version 2.0. Iowa City: University of Iowa: Center of
Advanced Studies in Measurement and Assessment.
Cartwright, F., Lalancette, D., Mussio, J., & Xing, D. (2003). Linking provincial student
assessments with national and international assessments. Education, skills and learning,
research papers, Bd. 005. Ottawa: Statistics Canada.
Chen, W.-H., & Thissen, D. (1997). Local dependence indexes for item pairs using item
response theory. Journal of Educational and Behavioral Statistics, 22, 265–289.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and
Psychological Measurement, 20, 37–46.
Dalehefte, I. M., Wendt, H., Köller, O., Wagner, H., Pietsch, M., Döring, B., ... Bos, W.
(2014). Bilanz von neun Jahren SINUS in deutschen Grundschulen: Evaluation im
Rahmen der TIMSS 2011-Erhebung. Zeitschrift für Pädagogik, 60, 245–263.
De Ayala, R. J., Plake, B. S., & Impara, J. C. (2001). The impact of omitted responses on
the accuracy of ability estimation in item response theory. Journal of Educational
Measurement, 38, 213–234.
De Leeuw, E. D., Hox, J., & Huisman, M. (2003). Prevention and treatment of item
nonresponse. Journal of Official Statistics, 19(2), 153–176.
Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Multivariate
applications books series. Mahwah, NJ, US: Lawrence Erlbaum Associates
Publishers371.
Ercikan, K., & Julian, M. (2002). Classification accuracy of assigning student performance
to proficiency levels: Guidelines for assessment design. Applied Measurement in
Education, 15(3), 269–294.
Hahn, I., Schöps, K., Rönnebeck, S., Martensen, M., Hansen, S., Saß, S., ... Prenzel, M.
(2013). Assessing scientific literacy over the lifespan – A description of the NEPS
science framework and the test development. Journal for Educational Research Online,
5(2), 110–138.
Hambleton, R. K., Sireci, S. G., & Smith, Z. R. (2009). How do other countries measure up
to the mathematics achievement levels on the National Assessment of Educational
Progress? Applied Measurement in Education, 22(4), 376–393.
Hanushek, E. A., & Wößmann, L. (2015). The knowledge capital of nations: Education and
the economics of growth. Cambridge, MA: MIT Press.
Haschke, L. I., Kampa, N., Hahn, I., & Köller, O. (2017). ). Setting standards to a scientific
literacy test for adults using the item-descriptor (ID) matching method. In B. P.
Veldkamp, & M. von Davier (Eds.). Methodology of educational measurement and as-
sessment. Berlin: Springer-Verlag.
Henning, G. (1989). Meanings and implications of the principle of local independence.
Language Testing, 6(1), 95–108.
Huynh, H., Michaels, H. R., & Ferrara, S. (1995). Comparison of three statistical
H. Wagner et al.
procedures to identify clusters of items with local dependency. Annual Meeting of
National Council on Measurement in Education.
Kolen, M. J., & Brennan, R. L. (2004). Test equating, scaling, and linking: Methods and
practices (2nd ed.). New York: Springer.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for cate-
gorical data. Biometrics, 33, 159–174.
Linn, R. L. (1993). Linking results of distinct assessments. Applied Measurement in
Education, 4, 185–207.
Livingston, S. A. (2004). Equating test scores (without IRT). Princeton, NJ: ETS Educational
Testing Service.
Mislevy, R. J. (1992). Linking educational assessments: Concepts, issues, methods, and pro-
spects. Princeton, NJ: ETS Policy Information Center.
Nissen, A., Ehmke, T., Köller, O., & Duchhardt, C. (2015). Comparing apples with or-
anges? An approach to link TIMSS and the National Educational Panel Study in
Germany via equipercentile and IRT methods. Studies in Educational Evaluation, 47,
58–67. https://doi.org/10.1016/j.stueduc.2015.07.003.
OECD (2006). Assessing scientific, reading and mathematical literacy: A framework for PISA
2006. Paris: OECD.
OECD (2009). PISA 2006 technical reportParis: OECD.
OECD (2013). PISA 2012 assessment and analytical framework: Mathematics, reading, sci-
ence, problem solving and financial literacy. Paris: OECD Publishing.
OECD (2016). PISA 2015 results (Volume I): Excellence and equity in education. Paris: OECD
Publishing.
OECD (2014a). PISA 2012 results. Paris: OECD Publishing.
OECD (2014b). PISA 2012 technical reportParis: OECD Publishing.
Pietsch, M., Böhme, K., Robitzsch, A., & Stubbe, T. C. (2009). Das stufenmodell zur le-
sekompetenz der längerübergreifenden bildungsstandards im vergleich zu IGLU
2006. In D. Granzer, O. Köller, A. Bremerich-Vos, M. van den Heuvel-Panhuizen, K.
Reiss, & G. Walther (Eds.). Bildungsstandards deutsch und mathematik. leistungsmessung
in der grundschule (pp. 393–416). Weinheim und Basel: Beltz Verlag.
Pohl, S., & Carstensen, C. H. (2012). ). NEPS technical report – Scaling the data of the
competence tests. NEPS Working Paper No. 14Bamberg: Otto-Friedrich-Universität.
Pohl, S., Gräfe, L., & Rose, N. (2014). Dealing with omitted and not reached items in
competence tests: Evaluating approaches accounting for missing responses in IRT
models. Educational and Psychological Measurement, 74, 423–452.
Prenzel, M., & Ostermeier, C. (2006). Improving mathematics and science instruction: A
program for the professional development of teachers. In F. K. Oser, F. Achtenhagen,
287
Studies in Educational Evaluation 59 (2018) 278–287
& U. Reynolds (Eds.). Competence oriented teacher training. Old research demands and
new pathways (pp. 79–96). Rotterdam: Sense Publisher.
Raftery, A. E. (1995). Bayesian model selection in social research. Sociological
Methodology, 25, 111–163.
Robitzsch, A., Lüdtke, O., Köller, O., Kröhne, U., Goldhammer, F., & Heine, J.-H. (2016).
Herausforderungen bei der Schätzung von Trends in Schulleistungsstudien.
Diagnostica. https://doi.org/10.1026/0012-1924/a000177.
Rose, N., von Davier, M., & Xu, X. (2010). Modeling nonignorable missing data with item
response theory (IRT). ETS Research Rep. No. RR-10-11. Princeton, NJ: Educational
Testing Service.
Ryan, J., & Brockmann, F. (2009). A practitioner’s introduction to equating with primers on
classical test theory and item response theory. Washington, DC: CCSSO.
Sälzer, C., & Prenzel, M. (2013). PISA 2012-eine Einführung in die katuelle studie. In M.
Prenzel, C. Sälzer, E. Klieme, & O. Köller (Eds.). PISA 2012. Fortschritte und
Herausforderungen in Deutschland (pp. 11–46). Münster: Waxmann.
Schöps, K., & Saß, S. (2013). NEPS technical report for science. Scaling results of starting
cohort 4 in ninth grade (NEPS working paper No. 23)Bamberg: University of Bamberg,
National Educational Panel Study.
van de Vijver, F. J. R. (1998). Towards a theory of bias and equivalence. In J. Harkness
(Vol. Ed.), ZUMA-nachrichten spezial: 3, (pp. 41–65). Mannheim: ZUMA.
van den Ham, A.-K., Ehmke, T., Nissen, A., & Roppelt, A. (2016). Assessments verbinden,
Interpretationen erweitern? Zeitschrift für Erziehungswissenschaft, 20(1), 89–111.
https://doi.org/10.1007/s11618-016-0686-2.
von Maurice, J., Sixt, M., & Blossfeld, H.-P. (2011). The german national educational panel
study: Surveying a cohort of 9th graders in Germany (NEPS working paper No. 3).
Bamberg: Otto-Friedrich-Universität, Nationales Bildungspanel.
Wagner, H., Schöps, K., Hahn, I., Pietsch, M., & Köller, O. (2014). Konzeptionelle
Äquivalenz von Kompetenzmessungen in den Naturwissenschaften zwischen NEPS,
IQB-Ländervergleich und PISA. Unterrichtswissenschaft, 42(4), 301–320.
Weinert, F. E. (2001). Concept of competence: A conceptual clarification. In D. S. Rychen,
& L. H. Salganik (Eds.). Defining and selecting key competencies. Göttingen: Hogrefe and
Huber Publishers.
Wu, M. (2010). ). Comparing the similarities and differences of PISA 2003 and TIMSS. OECD
education working papers, No. 32. Paris: OECD Publishing.
Wu, M. L., Adams, R. J., Wilson, M. R., & Haldane, S. A. (2007). ACERConQuest Version 2:
Generalised item response modelling software. Camberwell: Australian Council for
Educational Research.