Research Master Review 2011

Peaks in sight

Peaks in sight - Research Master Review 2011 - NVAO

NVAO
Accreditation Organisation of the Netherlands and Flanders

Parkstraat 28 / 2514 JK Den Haag

P.O. Box 85498 / 2508 CD The Hague
The Netherlands

T +31 (0)70 312 23 00

F +31 (0)70 312 23 01
info@nvao.net
www.nvao.net
Research Master Review 2011

Peaks in sight
Preface

4 Pe aks i n s i g ht
Dutch universities are renowned the world over for their good
educational and research achievements. In all rankings,
universities in the Netherlands score high on the list. It can be
said, with good reason, that Dutch university education stands
on a high plateau. The Master’s programmes, almost without
exception, have received high marks and are attractive to
foreign students. In addition, Dutch universities are highly
cherished partners in cooperative ventures – understandably
so, in view of their level of achievement.
Despite this excellent track record, it is often claimed that the
high plateau occupied by Dutch universities has no peaks of
excellence. In the field of research, it has been proven that
this claim is unsubstantiated, but in the field of education this
view continues to dog the universities. Not that consistently
staying on a high plateau is not an achievement in itself,
yet obviously the existence of peaks (and perhaps also an
occasional valley) is necessary to put the high plateau into
relief. It is, in part, for this reason that some are arguing for
greater differentiation and diversity between study
programmes.
A number of years ago, the so-called Research Master’s
programmes were introduced in the Dutch university sector.
These two-year programmes are aimed at students looking
for a research programme, or at students who want to pursue
a profession in which research skills are essential.
The Research Master’s programmes are taught by extremely
proficient, experienced researchers. In these programmes,
emphasis is placed on learning research skills.
The combination of accomplished researchers and motivated
students, in particular, should lead to a study programme on
the cutting edge.
This report focuses on the quality of Dutch Research Master’s
programmes. Accreditation assessments, the assessments
by foreign experts who have evaluated a wide selection of
Master’s theses, student surveys, and the positions held by
alumni, are all used to sketch a picture of the quality of the
Research Master’s programmes. The findings in this report all
point to one conclusion: Dutch Master’s programmes are of
a good quality, and the Research Master’s programmes are
among the best in their fields. So the high plateau of Dutch
university education does appear to have real peaks of
excellence.
NVAO hopes this report will be welcomed. It is good for the
universities to know that Research Master’s programmes are
held in high esteem internationally. It is good for students to
know that this type of Master’s programme is a challenge for
anyone interested in research. It is good for society to know
that these programmes are performing at the forefront of
their respective fields. Research Master’s programmes
offer the possibility of increasing the number of doctorates
pursued; they offer the possibility of discovering and
educating talented people; and they provide a firm foundation
to further strengthen already leading research achievements
in many disciplines and themes.
Karl Dittrich
Chairman, NVAO
NVAO Research M aster Review 2 0 1 1 5
Contents

6 Pe aks i n s i g ht
Summary 8
1 The Research Master Review 2011:
A bird’s eye view 14
1.1 Background and aims 15
1.2 Approach 17
2 Student enrolment, attrition,
and graduation 20
2.1 Student enrolment 21
2.2 Attrition rates 23
2.3 Graduation rates 24
2.4 Time to degree completion 24
2.5 Discussion 25
3 Thesis Assessment Study 28
3.1 Introduction 29
3.2 Preliminary analyses 30
3.3 Primary analyses 31
3.4 Discussion 32
4 Student satisfaction in MA and RMA programmes 36
4.1 Overall ratings 37
4.2 Content of the programme of study 39
4.3 Acquisition of general skills 39
4.6 Teachers & lecturers in the programme 39
4.7 Assessment and grading 39
4.8 Study load 39
4.9 Academic guidance, support and supervision 39
4.10 Other aspects 41
4.11 Discussion 42
5 The alumni perspective 46
5.1 Selectiveness of the study programme 47
5.2 Satisfaction with research related aspects
of the programme 47
5.3 Satisfaction with teachers and thesis
supervisors 49
5.4 Instruction time, workload and study results 49
5.5 Current employment 49
5.6 Satisfaction with preparation for professional
practice 49
5.7 Discussion 50
6 The accreditation of research master’s
programmes 54
6.1 The accreditation of RMA programmes 55
6.2 KNAW committees’ final conclusions 55
6.3 KNAW committees’ judgments
of themes 56
6.4 KNAW committees’ judgments
of standards 57
6.5 Discussion 61
7 Research master’s: Crossing the borders
of science 64
7.1 The fitful climate for science 65
7.3 The future of research master’s programmes 68
7.4 In closing 69
8 Synthesis 72
8.1 What did we find? 73
8.2 What are the implications? 73
References 78
RMR 2011 ⁄ A collective effort 80
Interviews
Lucy van de Wiel ⁄ Cultural Analysis 12
Jantine Boselie ⁄ Psychopathology 18
Werner Raub ⁄ Sociology and Social Research 26
Helena Cousijn ⁄ Cognitive Neuroscience 34
Alexander Sack ⁄ Cognitive and Clinical Neuroscience 44
Mirjam de Bruijn ⁄ African Studies 52
Bojou Neecke ⁄ Health Sciences 62
Thijs Porck ⁄ History: Society and Institutions 70
Anne-France Pinget ⁄ Linguistics 76
NVAO Research M aster Review 2 0 1 1 7
Summary

8 Pe aks i n s i g ht
What do we want to know?
The higher education system of the Netherlands is often
characterised as a ‘plateau without peaks’: exhibiting high
quality across the board, yet virtually devoid of true excel-
lence. NVAO wishes to challenge this rather deflated image,
being convinced that universities in the Netherlands certainly
do offer a reasonable amount of excellent study programmes.
More specifically, we suppose that the research master’s
programmes (RMAs) just might constitute such ‘peaks on
the plateau’.
NVAO initiated the Research Master Review (RMR 2011) with
one paramount research question in mind: What is the level of
educational quality realised in RMA programmes? A series of
studies was performed to answer this question. We analysed
accreditation decisions, key statistics about enrolment,
attrition and graduation, and looked into the quality of RMA
theses, student satisfaction and the alumni perspective.
What did we find?
Each of these studies provides valuable insights about
the quality and strengths of RMA programmes. They also
provide insights that go beyond the research question and
give an indication about how we can improve quality in
higher education.
Enrolment, attrition and graduation
Our analyses show that the average number of entrants has
increased in recent years. However, the student intake of
many RMAs seems to be well below programmes’ capacity.
The average attrition rate increases from 11% after one year
to 17% after four years. The average time to degree is 23.7
months. About 78% of all entrants graduate within four years
after entering the programme. Enrolling on a PhD programme
prior to graduation features among the reasons reported to
explain students’ delay in finishing the programme.
International students make up a large proportion of the RMA
entrants: 36% in 2009.
Thesis assessment study
This study involved a blind assessment of MA and RMA theses
by panels composed of foreign experts in four disciplines.
Our analyses indicated that all the theses assessed meet
international quality standards. However, RMA theses are
given significantly higher ratings than MA theses. Moreover,
the differences in quality between MA and RMA theses
appear to be considerably large. The assessment panels
considered the (vast) majority of theses assessed to be of
top international level.
Student satisfaction in MA and RMA programmes
Although MA and RMA students both are generally quite
satisfied with their programme, satisfaction is higher in RMA
students. RMA students are especially more positive about
the academic skills acquired through the programme, about
their teachers and lecturers, and about the academic
guidance, support and supervision offered by the programme.
On the other hand, RMA students are less positive about their
study load, the programme timetables, the extent to which
they learn how to work together with others, the availability
and affordability of accommodation in the city where they are
studying, and about several facilities in their institution and
student-city environment.
The alumni perspective
A large-scale survey among three cohorts of MA and RMA
alumni shows that RMAs are more selective than MA
programmes. Also, compared to MA alumni, RMA alumni
are more satisfied with research related aspects of their
study programme and with the quality and availability of
the teaching staff and thesis supervisors. They have more
instruction time, a higher workload, and higher study results.
And finally, they are more satisfied with the extent to which
their programme prepared them for professional practice.
More than half of all RMA alumni in the sample have enrolled
on a PhD programme.
Analysis of accreditation decisions for
RMA programmes
The norms which RMA programmes have to satisfy in order
to be accredited by NVAO clearly surpass those which have to
be met by regular MA programmes. Therefore by definition,
accredited RMAs demonstrate higher levels of quality than
regular MA programmes. Although the KNAW committees
who assess the RMAs maintain stern demands, they are quite
positive about the quality demonstrated by RMA. Between a
fifth and a third of all applications are judged as either good
or excellent on standards regarding (amongst others) the
intended learning outcomes, the curriculum and the quality
of the teaching staff.
What are the implications?
Educational policy
The RMA programmes fit in perfectly with the strategic
agenda initiated by the urgent call of the Veerman Committee.
RMAs main features include selection, sharper profiling,
investments in research and higher qualified personnel – the
same features which are highlighted in the recommendations
of the Veerman Committee. In doing so, the RMA contributes
NVAO Research M aster Review 2 0 1 1 9
to making the Dutch higher education system more future-
proof. Therefore, the Dutch government and policy makers
would do well to cherish the RMA and assure the continuity,
accessibility and quality of RMA programmes.

Didactical approach
RMA programmes derive part of their success from their
characteristic didactical approach: the master/apprentice
model of instruction and academic socialisation.
Several distinctive attributes of RMAs are among the most
powerful influences on achievement:
> highly motivated students who are ready to put a
considerable effort into their studies;
> challenging learning goals;
> a captivating delivery by highly qualified and passionate
teachers;
> teachers and students forming a cohesive community
of learners.

Enhancing continuity through increasing student

enrolment
For many RMAs, increasing student enrolment remains the
biggest challenge. RMR 2011 suggests a number of ways to
achieve this:
> Educational institutions could do more to infuse more
positive attitudes towards science and research in their
students. Challenging research seminars as part of the
bachelor’s programmes could be a good starting point to
foster a deeper and lasting interest in research and science.
> Scouting is a promising strategy: approach talented
students at an early stage and give them a taste of what
an RMA might bring them.
> The co-operation between universities could be
strengthened. Invigorating the inter-university
research schools seems to be a logical starting point.
> To recruit more foreign research talent, Nuffic could
feature the RMA more explicitly as a unique selling point
of the Dutch higher education system.

10 Pe aks i n s i g ht
NVAO Research M aster Review 20 1 1 11
Interview

12 Pe aks i n s i g ht
Lucy van de Wiel
Cultural Analysis
Lucy van de Wiel started her PhD in June 2011, after finishing
the research master’s programme Cultural Analysis at the
University of Amsterdam: “It is an interdisciplinary programme,
focused on analysing cultural objects. That could be anything,
for example, a radio programme, a policy text, or a literary
work. The programme teaches students how to analyse those
objects, using theories about culture stemming from different
philosophical traditions. These analyses provide insight in
the meaning of cultural symbols and objects within a given
social context, while at the same time contributing to theory
building.
The course of my studies is a bit atypical. After finishing
my bachelor’s programme, I had the opportunity to study
abroad on a Fulbright scholarship. In that year, I participated
in a PhD programme at the University of California in Berkeley.
I was able to count this experience as an elective course for
the research master’s programme, so I could finish in one year
instead of two. I wanted to do a research master’s programme
anyhow, because it provides more challenge and because I
have always been interested in research.
I only did the compulsory courses, plus one optional course.
The general courses were interesting, although they were a bit
too broad in scope. Sometimes I would have liked more depth.
In general, the courses in the second year of the curriculum
have more depth, but I already did the second year in
Berkeley. However, I did like the degree of freedom we had in
choosing the topics for our projects. The thesis supervision
was very good, so in that respect I am absolutely satisfied.
The research for my PhD involves freezing egg-cells, making it
possible for women to have children at a later age. This has
been a hotly debated issue in recent years, for one, because
the Amsterdam Medical Center wanted to start a clinic where
women could have their egg-cells frozen in. At first, that was
prohibited in the Netherlands, although it was allowed in
Belgium and the UK. I study the societal debate about this
issue. Which kinds of arguments are used, which images and
conceptions about men, women, and parenting form the basis
for these arguments? How do these ideas originate, and how
are they translated in different cultural expressions, such as
parliamentary debates, Victorian medical texts, and biblical
stories? What do they tell us about identity formation, ideas
about the body, and temporality?
My primary aim is to profit as much as possible from the next
four years. I want to prepare myself fully for a career in
science, by working on my dissertation, giving lectures, and
writing articles. After completing my doctoral studies, I would
like to publish my dissertation and rewrite it into a book that
is more accessible to a broader audience.”
NVAO Research M aster Review 20 1 1 13
Chapter 1 ⁄
The Research
Master
Review 2011:
A bird’s eye
view

14 Pe aks i n s i g ht
The higher education system of the
Netherlands is often characterised
as a ‘plateau without peaks’: exhibiting
high quality across the board, yet
virtually devoid of true excellence.
NVAO is convinced that universities in
the Netherlands certainly do offer a
number of excellent study programmes.
We suppose that the research master’s
programmes (RMAs) just might
constitute such ‘peaks on the plateau’.
To support that assumption, NVAO took the initiative for the
Research Master Review 2011. The project consisted of
several studies, combining both quantitative and qualitative
research methods. This chapter discusses the backgrounds
and aims of the project, and provides an overview of the
chapters that follow.
1.1 ⁄ Background and aims
1.1.1 The introduction of the research
master’s programme
In 2003, the RMA was introduced as a new type of study
programme in the Dutch higher education system. RMAs are
specifically aimed at the acquisition of advanced research
abilities. Enrolment in an RMA is restricted to students with
above-average academic achievements and motivation.
RMAs are embedded in an academic environment of good to
outstanding quality, and the core teaching staff consists of
highly qualified researchers. For many, an RMA is the natural
steppingstone towards a PhD trajectory.
The RMA was introduced for several reasons. For a long time,
Dutch policy makers have argued that the Netherlands can
only compete in a global perspective as a knowledge economy.
In the global competition between knowledge societies,
innovation and creativity are considered to be the most
important drivers of economic growth. In this context, the
RMA would answer to the nation’s demand for more and
better researchers.
As a beacon of high-level quality, the RMA would help attract
young talent from outside the Netherlands. Also, RMAs were
supposed to increase the disappointing graduation rates of
Dutch PhD-trajectories and decrease the average time that
doctoral students take to finish their dissertations.
For most universities, there also was a modest financial
incentive for starting up RMA programmes. When the
bachelor’s-master’s degree structure was implemented,
regular master’s programmes received public funding for
one year (60 ECs). Only a number of disciplines, such as the
natural sciences, life sciences, agriculture, and engineering,
were allowed master’s programmes with a duration of and
funding for two years (120 ECs). Also, the students’ financial
support by the government depends on the duration of
the programme. The RMA provided universities with the
opportunity to offer master’s programmes with a two-year
duration in other disciplines, such as the humanities, social
and behavioural sciences. The institutions do not receive
extra public funding for these programmes, but students
enrolled on an RMA programme are granted an extra year
of financial support.
1.1.2 Research Master Review 2007
In 2007 NVAO initiated a first study into this new type of
programme. This study aimed at raising awareness of and
attention for the RMA, and identifying the distinctive profile
characteristics of the RMA. The study also explored the first
experiences with and expectations for these programmes.
NVAO published the findings in a report, titled Research
Master Review 2007: The exploration of a new domain
(RMR 2007).
RMR 2007 identified three profile characteristics which
distinguish RMA programmes from regular MA programmes
offered by research-oriented universities in the Netherlands:
> Programme objectives and a curriculum specifically focused
on the acquisition of research competencies;
> An academic context of unquestionably high quality that
strives for excellence; and
> Selection of students with above-average academic
performances and ambitions.
NVAO received a large amount of applications for the initial
accreditation of RMAs: 156. This indicated a strong need for
this new type of master’s programmes among most of the
research universities in the Netherlands. Throughout the
academic community, the RMA was very well received as a
successful and promising educational innovation.
However, RMR 2007 did also signal one serious drawback:
the disappointing student enrolment. Allegedly, this was
caused by the image of the RMA as only being of interest for
those aspiring a PhD. To increase student enrolment, the
added value of the RMA should be made clear to those who
do not aspire a career within academia, it was reasoned.
NVAO Research M aster Review 20 1 1 15
1.1.3 Accreditation of RMAs as existing study
programmes
Nearly all RMAs that were positively assessed as new study
programmes in 2003-2007, have undergone a second assess-
ment as existing study programmes. From March 2008 to
March 2011, NVAO received a total of 128 applications, pre-
dominantly for the accreditation of existing RMA programmes
(116), and some for the initial accreditation of new RMA
programmes (12). Furthermore, the so-called ‘second phase’
of the accreditation system was legally effectuated on 1
January 2011. This involved some fundamental changes to
the accreditation procedures which were followed in previous
years. Recently, the new Assessment frameworks for research
master’s programmes have been established, and per
1 September 2011, RMAs are being assessed according
to the new regime.
1.1.4 Policy context
In September 2009, the then Minister of Education, Culture,
and Science, Ronald Plasterk, assigned a committee chaired
by former Minister Cees Veerman. The committee was asked
to assess the extent to which the higher education system of
the Netherlands is future-proof, compared to leading higher
education systems elsewhere. When NVAO initiated RMR
2011, the Veerman Committee had just published its advice
report, advocating a ’triple differentiation’ (in the structure of
the system, in the profiles of institutions, and in the offer of
study programmes) to make Dutch higher education more
future-proof. The committee’s report caused a stir in the
academic community, fuelling the national debate about some
fundamental changes in the structure of the Dutch higher
education system.
The Veerman committee formulated seven recommendations
to the Dutch Government. Five are relevant for RMAs:
> Selection: each institution should be given the right to select
students.
> Stimulate institutional profiling and reward accomplish-
ments: In order to stimulate distinctive institutional profiling,
the European U-Map system for multidimensional classifi-
cation should be further developed and adapted to the
Dutch context.
> A smaller share of student-bound funding: a higher share of
mission-bound funding that rewards accomplishments in
the distinctive academic profile of institutions.
> Investment in research: in order to secure the competitive-
ness of the Netherlands, investments are needed to
stimulate the integration of research throughout the higher
education system.
16 Pe aks i n s i g ht
> A new arrangement for master’s programmes: the offer of
master’s programmes should be differentiated, and the
professional master’s should be more structurally
embedded. Also, it should be explored how to enhance the
flexibility for lifelong learning.
The committee also formulated three recommendations to
the institutions:
> Choose a profile: institutions and their subdivisions should
opt for a distinctive profile on one or more dimensions of
the European classification system, based on achieved or
desired strengths.
> Give more attention to education as a core task of higher
education institutions: be responsive to the different
learning styles and backgrounds of students, make study
programmes more flexible and better organised.
Institutions and students should work together at realising
this objective.
> Invest in higher qualified personnel: institutions should
provide teachers wider opportunities, and be sensitive to
career-planning with equivalent and coherent career
tracks in education and research.
On 1 July 2011, the current State Secretary for Education,
Culture and Science, Halbe Zijlstra, issued a memorandum
titled Quality in Diversity: Strategic Agenda for Higher Educa-
tion, Research, and Science. Many of the policy measures
which are suggested in this memorandum are inspired by
the Veerman Committee’s report. In our view, RMAs seem to
answer to many of the recommendations formulated by the
Veerman Committee, and to many of the objectives stated in
the strategic agenda for higher education.
1.1.5 Global objectives
Against these backgrounds, NVAO initiated RMR 2011 with
one paramount research question in mind: What is the level of
educational quality realised in RMA programmes? Answering
this question would contribute to attaining the following
global objectives.
Evaluating the educational quality realised in
RMA programmes
NVAO collects a wealth of information about the quality of
individual RMA programmes in the form of critical self-
evaluations and external quality assessments. These sources
can be used to analyse the educational quality realised by
RMA programmes at a higher level of aggregation. This kind
of ‘meta-evaluation’ could show important developments,
trends, good practices and structural challenges within the
domain of RMA programmes.
Strengthening the continuity of RMA programmes
Although the academic community is generally enthusiastic
about the RMAs, RMR 2007 also pointed out one major
weakness: many RMAs appear to recruit (substantially) less
students than possible or desirable. Through RMR 2011, NVAO
hopes to collect ideas about how to increase the student
enrolment, in order to strengthen the continuity of RMAs.
Promoting quality improvement in higher education
Although accountability is being given full attention in the
current accreditation system, the purpose of quality improve-
ment remains a very real part of the system. Through RMR
2011, NVAO aims to promote quality improvement in higher
education, which is, of course, primarily the responsibility of
the institutions.
1.2 ⁄ Approach
The alumni perspective
Chapter 5 presents the results of a study performed by
Jorrit Snijder, managing director of the Roosevelt Academy
and researcher at CHEPS (the Center for Higher Education
Policy Studies of the University of Twente). To learn more
about the effectiveness of the RMAs, he surveyed three
cohorts of alumni from research master’s (RMA) and regular
master’s (MA) programmes from 9 Dutch universities,
between 1.5 and 2 years after their graduation.
Analysis of accreditation decisions for RMA programmes
Chapter 6 reports the outcomes of an analysis of the accredi-
tation decisions for RMAs. At the conference, the preliminary
results of this study were presented by Karl Dittrich, chair of
NVAO. Specifically, this study looks into the question which
quality criteria are the most important for distinguishing
between positively and negatively assessed applications.

RMR 2011 consisted of a series of studies, in which both
quantitative and qualitative research methods were
combined. To provide a look ahead, we discuss each
study briefly.
Enrolment, attrition and graduation
Chapter 2 provides some key statistics concerning the student
enrolment, attrition rates and graduation rates for RMA
programmes. These statistics are based on cohort datasets
provided by the Association of Universities in the Netherlands
(VSNU).
Thesis assessment study
Chapter 3 discusses a thesis assessment study (TAS), which
aims to answer the questions (a) to what extent theses written
by RMA students differ in quality from theses written by MA
students, and (b) how the quality of theses written by Dutch
MA and RMA students relate to international standards.
Crossing the borders of science
Chapter 7 is an elaboration of the speech held by Robbert
Dijkgraaf, President of the Royal Netherlands Academy of Arts
and Sciences (KNAW). The article sketches the current climate
for science, the challenges for higher education in the Nether-
lands, and the future of RMA programmes.
Interviews
The chapters reporting the results of the various studies
are alternated with interviews with students, alumni and
staff from RMAs. They offer a ‘view from inside’ and provide
valuable insight into the quality of RMAs.

Student satisfaction in MA and RMA programmes

Each year, the Dutch National Student Survey (NSE) is held,
measuring student satisfaction among almost all institutions
for higher education in the Netherlands. Chapter 4 reports the
results of an analysis of NSE data, identifying differences in
satisfaction between MA and RMA students.

Conference
On Thursday 24 March 2011, NVAO organised the conference
Research master’s: Excellent profiling à la Veerman (for an
overview of the programme in Dutch, see http://www.nvao.net/
nvao-congressen). Three of the lectures held at the
conference were converted into articles for this report,
included in chapter 5, chapter 6, and chapter 7.
NVAO Research M aster Review 20 1 1 17
Interview

18 Pe aks i n s i g ht
Jantine Boselie

Psychopathology
Jantine Boselie is enrolled on the second year of the research
master’s programme Psychopathology at Maastricht University:
“The programme is focused on mental disorders, such as
anxiety disorders, depression, and personality disorders.
For example, we examine whether a certain treatment is
effective. Or we investigate the mechanisms underlying a
disorder, like depression: is that a biological disorder, or do
social and cognitive factors also have an influence?
The research master’s programme absolutely meets my
expectations. Our teachers are leading specialists in their
fields, like our university professors who are specialised in
anxiety disorders, and experts from the USA. That interaction
is very stimulating. We are taught to think critically about
research. Besides knowing how to do scientific research,
you should also be able to critically evaluate the quality of
a study.

The Psychology department at Maastricht University has a
strong focus on research. In the bachelor’s programme, I soon
discovered that I very much liked doing research. I wanted to
experience the whole process, from formulating the research
question, to executing the study, and finally writing about it
all. It certainly helps that I am extremely curious; I want to
know people’s drives and which factors influence those drives.
I would like to continue with a PhD programme. Finding
grants, however, we have to do ourselves. Our department
offers some support, for example, by posting vacancies for
research positions on the website. But I haven’t yet found
my bagful of money.”

An important motive for me to choose this master’s

programme is its two-year duration. In the first year, we get a
lot of theory. In the second year, we have a clinical internship
(a placement at a hospital, for example), as well as a research
internship. That combination makes it interesting to me. And it
offers also that little extra to mention on your resume, which
can make a large difference in these times.

NVAO Research M aster Review 20 1 1 19

Chapter 2 ⁄
Student
enrolment,
attrition,
and
graduation

20 Pe aks i n s i g ht
How many (international) students enrol Table 2.1 ⁄ Number of entrants in 2005-2009.
on RMA programmes? How many of
2005 2006 2007 2008 2009 Total
them drop out, and how many graduate?
How long does it take RMA students to (Bio)medical Sciences 75 115 171 190 217 768
complete their programme? Behavioural Sciences 210 262 309 353 404 1,538
Earth Sciences 6 7 26 8 7 54
Objective monitoring of performance is essential for Humanities 343 371 414 507 522 2,157
evaluating and raising quality on all levels of analysis. Social Sciences 202 207 268 304 378 1,359
This chapter considers several measures which are widely
used as educational performance indicators: student Total 836 962 1,188 1,362 1,528 5,876
enrolment, attrition rates, graduation rates, and time to % of MA population 4% 3% 3% 4% 4% 4%
degree completion. All the results presented in this chapter
pertain to five student cohorts, from cohort 2005 onward.
VSNU provided and assisted with analysing the data.
Table 2.1 shows the number of entrants for the cohorts
Our analyses indicate that the number of RMA programmes 2005-2009. Or more exactly, the total number of first-time
and the number of students enrolling on RMAs have been enrolments: a small number of students are counted
growing steadily. About 78% of all entrants graduate within twice, because they are registered in two different RMA
four years after entering the programme. The average time to programmes. Over a third of all RMA students enrol on
degree is 23.7 months, and 76% of all graduates finish within programmes within the humanities. In contrast, RMA
the RMA programmes’ nominal duration of 24 months. programmes within the earth sciences had only 54 first-time
enrolments. Within five years, the total number of entrants
2.1 ⁄ Student enrolment has almost doubled: from 836 in 2005 to 1,528 in 2009.
However, there appear to be large differences in growth rates
Growth in the number of entrants between the different discipline clusters: growth is most
The growing demand for more and better researchers, and prolific in the (bio)medical sciences and virtually nonexistent
making Dutch higher education more attractive to foreign in the earth sciences. The growth in the number of RMA
talent, were among the main reasons for introducing the RMA. entrants parallels the growth in the whole population of
To assess the extent to which the RMAs are meeting these students entering a master’s programme. Therefore, the
objectives, it is necessary to know exactly how many Dutch proportion of RMA entrants within that population has
and international students enrol on RMA programmes. remained at a steady 3-4%.
But keeping track of student enrolment is also important for
didactic purposes: group size has a direct influence on the Growth in the number of RMA programmes
teaching/learning process. Table 2.2 displays the number of RMA programmes which
registered at least one new entrant for the cohorts 2005-
2009. Over the years, a few RMAs within the Humanities
and Social Sciences were discontinued and closed for new
enrolment. Still, the total number of RMAs with student
intake has steadily increased, from 94 in 2005 to 123 in 2009.
About half are RMAs within the humanities and about a
quarter are RMAs within the social sciences. Growth in the
number of RMAs is most salient in the (bio)medical sciences.

NVAO Research M aster Review 20 1 1 21

 Table 2.2 ⁄ Number of RMAs with student intake in Figure 2.1 ⁄ Variability in the number of entrants for cohort
2005-2009. 2009 (n=123).

2005 2006 2007 2008 2009 Frequency

14
(Bio)medical Sciences 6 8 10 12 15
12
Behavioural Sciences 13 15 17 17 17
Earth Sciences 1 1 2 2 2 10

Frequency
Humanities 50 55 56 57 60 8
Social Sciences 24 26 27 29 29
6

4
Total 94 105 112 117 123
2

0
0 5 10 15 20 25 30 35 40 45 50 55 60
Number of entrants
Growth in the average number of entrants Number of entrants
The average number of entrants per RMA has increased
from 8.9 in 2005 to 12.4 in 2009. Growth is most salient in Objectives regarding student intake
RMAs within the behavioural and social sciences, and less RMA programmes, on average, aim for a maximum intake of
pronounced in RMAs within the humanities and (bio)medical 25 students per cohort, with a maximum of 10 students for
sciences. Over all disciplines and cohort years, the average the smallest and 60 for the largest programmes. So with an
number of entrants per RMA programme was 10.7 (SD=8.7). average intake of 12.4 students for cohort 2009, the student
enrolment seems to be well below the programmes’ capacity.
Table 2.3 ⁄ Average number of entrants per RMA in
2005-2009. The KNAW Social Sciences Council (SWR, 2007, pp. 13-14)
reported that RMAs within the social and behavioural
2005 2006 2007 2008 2009 Total sciences, on average, aimed for a maximum intake of
23 students per cohort (SD=10.8), and a minimum intake
(Bio)medical Sciences 12.5 14.4 17.1 15.8 14.5 15.1 of 15 students (SD=9.7). Since these RMAs, on average,
Behavioural Sciences 16.2 17.5 18.2 20.8 23.8 19.5 had an actual intake of 11 students in 2006 (SD=8.5), the
Earth Sciences 6.0 7.0 13.0 4.0 3.5 6.8 SWR concluded there was plenty room for growth. In 2009,
Humanities 6.9 6.7 7.4 8.9 8.7 7.8 the average number of entrants for RMAs in the social and
Social Sciences 8.4 8.0 9.9 10.5 13.0 10.1 behavioural sciences was 17 (SD=14.4). This is slightly
above these programmes’ objectives for the minimum
Total 8.9 9.2 10.6 11.6 12.4 10.7 student intake, but still leaves room for growth.

A high proportion of international students

Table 2.4 shows that international students make up 31% of
High variability in the number of entrants the RMA entrants. The largest proportions of international
Figure 2.1 displays the high variability in the number of students are found in RMAs within the earth and social
entrants for the 123 RMA programmes with student intake sciences, the smallest proportion in RMAs within the (bio)
in 2009. Some RMAs are very small: 25 RMAs show an intake medical sciences. The proportion of international students
of less than 5 students. On the other hand, 19 RMAs show has increased from 29% in 2005 to 36% in 2009.
an intake of more than 20 students. More than half of all
RMAs (63 programmes) had an intake of 10 students or less.
More than a quarter (32 programmes) had an intake of
5 students or less, and 5 programmes had an intake of only
1 student.

22 Pe aks i n s i g ht
 Table 2.4 ⁄ Intake of Dutch and international students in Table 2.5 ⁄ Attrition rates per cohort.
2005-2009.

Dutch International Total Cohort Cohort < 1yr < 2yr < 3yr < 4yr
size
(Bio)medical Sciences 625 18% 143 19% 768
Behavioural Sciences 1,067 69% 471 31% 1,538 2005 678 11% 14% 15% 16%
Earth Sciences 32 59% 22 41% 54 2006 719 11% 15% 16% 18%
Humanities 1,519 70% 638 30% 2,157 2007 927 10% 14% 16% -
Social Sciences 820 60% 539 40% 1,359 2008 1,073 11% 16% - -
2009 1,187 11% - - -
Total 4,063 69% 1,813 31% 5,876
Total - 11% 15% 16% 17%

The most recent mobility statistics available for the Nether-

lands published by Nuffic (Richter et al., 2011) indicate that
of all the students enrolled on master’s programmes, 15%
are international students. The proportion of international
students among the RMA entrants is more than twice as high.
2.2 ⁄ Attrition rates
In contrast to most other master’s programmes, RMAs are
selective: only students with above-average motivation and
prior achievements are eligible for enrolment. Students who
are admitted may be expected to meet the high requirements
of the programme and to successfully complete the programme
within the nominal programme duration of two years. In other
words, RMA programmes should have low attrition rates and
high graduation rates. Otherwise, their selectivity would not
be justified.
Table 2.5 shows the attrition rates (the number of
deregistered students as a percentage of the cohort size)
1-4 years after first enrolment for the cohorts 2005-2009.
The percentages in this table are not cumulative: some
RMA students who deregister in a certain year return to
the programme in a later year. But the number of these
‘re-entrants’ is smaller than the number of deregistering
students: the average attrition rate increases from 11%
after 1 year (cohorts 2005-2009), to 17% after 4 years
(cohorts 2005-2006).
About a third of all students who left and did not return to
their RMA, at some time did complete a master’s programme
within the same field of science and at the same institution
as their RMA programme. About 60% left and did not enrol on
another master’s programme within the same institution as
their RMA. Some RMA programmes’ self-evaluation reports
mention several reasons for students leaving the programme:
(a) personal problems, (b) following courses or doing an
internship at a university outside the Netherlands, (c) a job
offer, or (d) being admitted to a PhD programme. However,
there are no data on drop out from regular master’s
programmes. Therefore, we lack the necessary information
to comparatively assess these attrition rates.
The calculation of attrition rates (in this section) and
graduation rates (in the next section) is based on cohort
sizes which differ from the number of entrants reported
in the previous section. For calculating the attrition and
graduation rates, we have excluded two groups of
enrolments which would confound the results:
> 429 enrolments on four RMAs within the medical sciences
which have a nominal programme duration of more than
2 years; and
> 863 late enrolments (registered after the reference date
of 1 October).

NVAO Research M aster Review 20 1 1 23

2.3 ⁄ Graduation rates
Table 2.6 displays the graduation rates (the number of
diplomas awarded as a percentage of the cohort size)
1-4 years after first enrolment. On average, 6% of all
entrants graduated within 1 year (cohorts 2005-2009),
52% graduated within 2 years (cohorts 2005-2008),
73% graduated within 3 years (cohorts 2005-2007),
78% graduated within 4 years (cohorts 2005-2006), and
79% graduated within 5 years (cohort 2005) after first
enrolment on the RMA.
Table 2.7 shows that the average time to degree completion
was 23.7 months for all 2,788 RMA graduates (SD=7.5).
There seem to be no large differences between students
within different disciplines. So on average, RMA graduates
seem to have finished exactly within the programmes’ nominal
duration of two years. However, the dataset also includes
the lateral entrants who are expected to complete the RMA
programme in less than two years. If these lateral entrants
could be excluded, the average time to degree would be higher.
Table 2.7 ⁄ Average time to degree (in months).

Table 2.6 ⁄ Graduation rates per cohort. N M SD

Cohort Cohort < 1yr < 2yr < 3yr < 4yr (Bio)medical sciences 168 25.3 7.9
size Behavioural sciences 881 23.4 5.6
Earth sciences 38 21.2 9.0
2005 678 6% 51% 72% 78% Humanities 996 24.5 8.6
2006 719 5% 54% 73% 78% Social sciences 705 22.8 7.4
2007 927 6% 52% 73% -
2008 1073 5% 53% - - Total 2,788 23.7 7.5
2009 1187 6% - - -

Total - 6% 52 73% 78%

Two factors confound these graduation rates. First, almost
all RMA programmes offer the possibility of ‘lateral entry’:
students who have completed a comparable regular master’s
programme can enter in the second year of the RMA.
This explains the graduation rate of 6% after 1 year.
Second, these graduation rates simply reflect the number
of years between students’ first enrolment and programme
completion. They have not been corrected for years that
students may have been deregistered prior to graduation.
This correction has been made when calculating the statistics
reported in the next section.
2.4 ⁄ Time to degree completion
Cumulative percentage of graduates over time
Figure 2.2 shows how the percentage of graduates cumulates
over the number of months from first enrolment to successful
degree completion (corrected for years in which students were
deregistered, but including lateral entrants). Of all RMA gradu-
ates, 11% completed the programme within 12 months, 16%
finished within 18 months, 76% finished within 24 months,
87% finished within 30 months, and 96% finished within 36
months. Only 112 graduates (4%) took more than 36 months
to complete their RMA programme.
Figure 2.2 ⁄ Cumulative percentage of graduates over
1-60 months (n=2,788).
Cumulative percentage
100%

Graduation within the nominal programme duration 80%

Cumulative percentage

The ‘time to degree completion’ is defined as ‘the number of

60%
months between first enrolment and successful completion,
corrected for years in which the student was deregistered’. 40%

The results reported in this section pertain to all RMA

20%
graduates of the cohorts 2005-2009. However, they do not
include the graduates of the four RMAs within the medical 0%
0 6 12 18 24 30 36 42 48 54 60
sciences which have a nominal programme duration of more Months
than two years. Months

24 Pe aks i n s i g ht
Some RMA programmes’ self-evaluation reports mention
reasons for students’ delay in finishing the programme:
(a) personal problems, (b) following courses or doing an
internship at a university outside the Netherlands, and
(c) choosing to spend extra time on finishing their master’s
thesis: some students extend and elaborate their theses
into a component of a PhD programme they enrol on prior
to graduation.
2.5 ⁄ Discussion
RMAs attract a relatively large proportion of international
students. International students make up 15% of all students
currently enrolled on master’s programmes, whereas the
proportion of international students among the RMA entrants
of cohort 2009 was 36%. Generally, the recruitment of interna-
tional students is organised locally by the programmes and
universities individually. To further increase the intake of
international students, the RMA programmes and institutions
could consider joining forces, and seeking co-operation with
Nuffic.

Although we have no data on attrition and graduation rates in
other master’s programmes, we conclude that the attrition
and graduation rates for RMA programmes give no cause for
concern. Actually, our findings sketch a fairly positive picture
of the results of RMA programmes: out of every 100 students
enrolling on an RMA, approximately 11 students drop out in
the first year, 8 students leave the programme in a later year,
60 students graduate within 24 months, 16 students graduate
within 25-36 months, and 5 students graduate within 37 or
more months after first enrolment.
Our findings regarding student enrolment on RMA
programmes, however, sketch a less positive picture. Both
NVAO (2007, p. 60) and the SWR (2007, pp. 18-19) reported
that the low student enrolment was among the stakeholders’
major concerns. A higher number of entrants would not only
be desirable for RMA programmes’ meeting the demand for
more researchers and PhD students. According to some,
increasing the student intake would also be necessary for
securing the viability and continuity of the RMAs.
But recruitment activities could also be aimed at increasing
the enrolment of Dutch students. To this end, further research
should shed light on the reasons students have for enrolling,
or rather, not enrolling on an RMA programme. Is it because
they are unfamiliar with this type of master’s programme? Or
do they find the programmes’ high demands and prolonged
duration unappealing? Do they think that RMAs are only for
students who aspire to enter a PhD programme? Or can’t they
meet the admission requirements? Such knowledge about the
motivation of students for not enrolling on an RMA, can inform
institutions about how to position the RMA more effectively in
the ’higher education market’.

The ambitious learning objectives which characterise RMA

programmes demand intensive modes of instruction. So when
student intake is too large, it becomes difficult for teachers to
provide the appropriate amount of challenge and guidance.
But when the intake is too small, students have few opportu-
nities for learning through peer tutoring, for learning how to
work together in different groups, and for developing leader-
ship and conflict resolution skills.

Our analyses show that the average number of entrants has

increased in recent years. However, the student intake of most
RMAs seems to be well below programmes’ capacity. Indeed,
in order to provide their students with the opportunity to reap
the benefits of collaborative learning, many RMAs would do
well to increase the enrolment, while maintaining the current
level of admission requirements.

NVAO Research M aster Review 20 1 1 25

Interview

26 Pe aks i n s i g ht
Werner Raub
Sociology and
Social Research
Prof Dr W. (Werner) Raub, professor of theoretical sociology
at Utrecht University: “I founded the research master’s
programme Sociology and Social Research at Utrecht
University, and until recently I also led this programme.
The programme attracts about 15 students each year.
That is exactly the number of students which we aim for.
Students start in September and are expected to success­fully
complete the programme within 22 months. There are no
study delays, all students finish in time. The attrition rates
are at a minimum, about 10%, of which the majority drops
out within the first weeks of the curriculum. Those are
students who have made the wrong choice.
Important factors for the success of our research master’s
programme are the relatively small groups, the intensive
guidance, and the adequate selection. Right from the start,
students know what to expect. There are no re-examinations,
each course has to be successfully completed the first time
around. Students also compare themselves with each other:
weaker students see what the stronger ones do and don’t
want to fall behind.
Our research master’s programme is a breeding ground for
the PhD trajectory. We co-operate with colleagues from
Groningen and Nijmegen in an inter-university research
school. Annually, we select about 15 new PhD students
from a total of 250-450 applicants. Each year, 7-10 research
master’s graduates apply for a position, and 60-70%
successfully so. This certainly indicates their quality.
We recruit intensively abroad, so we have around 50%
international students. That is also our target for the years
to come. Dutch universities should take into account their
competitiveness, when deciding on the tuition fees for
international students. Students from the European
Economic Area (EEA) have to pay the same registration
fees as Dutch students, but students from outside the EEA
pay a lot more, about 15,000 euro per year. They pay less for
similar programmes in Germany.
The introduction of the research master’s programmes has
been an excellent development. They contribute significantly
to a better preparation of PhD students. However, many of
these programmes are too small. The main problem is the
policy of the Dutch universities: all institutions do their own
thing. Better co-operation between universities would
present much better opportunities for viable programmes.
For example, you could think of merging similar research
master’s programmes from two or three different universities.
With the small distances and well-functioning inter-university
research schools, that would certainly be attainable.”
NVAO Research M aster Review 20 1 1 27
Chapter 3 ⁄
Thesis
Assessment
Study

28 Pe aks i n s i g ht
 To what extent is there a difference in
quality between theses written by
students enrolled on research master’s
(RMA) programmes and those written by
students enrolled on regular master’s
(MA) programmes? How does the quality
of Dutch (research) master’s theses relate
to international standards?
3.1 ⁄ Introduction
If RMAs are truly excellent programmes, then this should be
expressed in the quality of the theses produced by their
students. Supposing that RMA theses are of high quality, not
only in comparison with regular master’s theses, but also in
an international perspective, NVAO conducted a thesis
assessment study (TAS). The study was conducted in close
co-operation with master’s programmes in four disciplines:
> Psychology and Neuroscience, Maastricht University (P&N);
> Media Studies, University of Amsterdam (MS);
> Communication Science, University of Amsterdam (CS); and
> Economics and Business, Tilburg University (E&B).
For all four study sites, samples were drawn from the popula-
tion of recent theses in English (n=74). Half of the sample was
written by RMA students, the other half by students from
adjacent regular MA programmes. For two of the study sites,
samples were drawn from theses that centred on the mean
supervisor’s grades in the MA and RMA populations. The
theses in these samples represent the “average” MA and RMA
theses. For the two other study sites, we randomly selected
theses covering the full range of grades (varying from 6 to
10 on a 10-point scale). The theses in these samples reflect
the full variability in quality present in the two populations.
These panels were assisted by NVAO policy advisors, who
acted as secretaries. In order to enhance the consistency and
comparability of assessments between panels and panel
members, NVAO developed a thesis assessment framework
including nine standards. Prior to presenting the 74 theses for
assessment by the panel members, they were made anony-
mous: the names of students and supervisor(s) were removed,
as well as information that would point to the programme in
which the student was enrolled. The assessments were
performed “double-blind”: neither panel members nor secre-
taries knew which were the (R)MA theses. Nor did they know
how the theses were graded by supervisors.

Foreign experts were invited to take part in the assessment

panels. The composition of the four panel was as follows:

Psychology and – David Linden (Professor of Translational Neuroscience, School of Medicine, Cardiff University, UK), (Chair);
Neuroscience – Miguel Castelo-Branco (Director of the Institute of Biomedical Research in Light and Image,
University of Coimbra, Portugal); and
– Lars Muckli (Professor of Psychology, University of Glasgow, UK).

Media Studies – Astrid Söderbergh Widding (Professor at the Institute of Cinema Studies, University of Stockholm, Sweden), (Chair);
– Jostein Gripsrud (Professor at the Department of Media Studies, University of Bergen, Norway); and
– Matthew Fuller (Professor at the Centre for Cultural Studies, Goldsmiths, University of London, United Kingdom).

Communication – Keith Roe (Professor at the Leuven School for Mass Communication Research, University of Leuven, Belgium), (Chair);
Science – Wolfgang Donsbach (Professor of Communication at the Department of Communication at the University of Dresden,
Germany); and
– Ralf Terlutter (Professor of Marketing and International Management, University of Klagenfurt, Austria).

Business & Economics – Hannelore Weck-Hannemann (Professor of Political Economy at the Institute for Public Finance at the University
of Innsbruck, Austria), (Chair);
– Gordon Gemmill (Emeritus Professor of Finance, Warwick Business School, University of Warwick, UK); and
– Piet Sercu (Professor of International Finance, Katholieke Universiteit Leuven, Belgium).

NVAO Research M aster Review 20 1 1 29

 3.2 ⁄ Preliminary analyses Table 3.1 ⁄ Differences in supervisor’s grades between
MA and RMA theses.
3.2.1 Sample characteristics
Table 3.1 shows the differences in supervisor’s grades MA RMA
between the samples of MA and RMA theses. The 37 RMA N M SD N M SD d
theses have significantly higher grades than the 37 MA
theses included in the study. This indicates a strong effect of Psychology & Neuroscience 9 7.9 1.10 9 9.1 .60 1.41c
the type of study programme (MA or RMA) on supervisor’s Media Studies 9 7.5 .35 9 7.6 .42 .26d
grades. However, for the samples of theses of Media Studies Communication Science 9 7.5 .54 9 8.6 .43 2.27a
and those of Economics and Business, no significant Economics & Business 10 7.9 1.00 10 8.0 .83 11d
differences in supervisors’ grades between MA and RMA
theses were found. Total 37 7.7 .81 37 8.3 .82 .75b

3.2.2 Assessment panels’ ratings a

p<.0 0 1; b
p<.0 1; c
p <.0 5 ; d
p>.0 5
Table 3.2 presents the means and standard deviations for
the assessment panels’ ratings of both MA and RMA theses
on each of the nine standards of the assessment framework. 3.2.3 Internal consistency of the measurements
The assessment panels’ ratings were calculated in the With Cronbach’s α ranging between .79 and .96, the
following way. First, the assessment panel members composite scale appears to be highly reliable for all
individually rated each thesis on the nine standards, using sub-samples. The reliability coefficients ranging from
a 4-point scale (1=unsatisfactory, 2=satisfactory, 3=good, .69 to .81 found for the four assessment panel’s total
4=excellent). Secondly, an average score for each standard scores, indicate satisfactory to good levels of inter-
was calculated from the individual ratings of the three panel rater agreement in the scores on the composite scale.
members. Thirdly, the individual panel members’ scores on
the nine criteria were averaged. Finally, these average panel
member scores were averaged in turn, to calculate a ‘panel
total score’ for each thesis.

Table 3.2 ⁄ Mean scores (M) and standard deviations (SD) in assessment panel ratings.

P&N (N=18) MS (N=18) CS (N=18) E&B (N=20)

Standards M SD M SD M SD M SD

Anchoring in current knowledge 3.39 .43 2.78 .67 2.71 .65 3.25 .55
Problem definition and hypothesis 3.26 .48 2.69 .82 2.84 .60 2.90 .57
Research design 3.11 .60 2.39 .53 2.58 .77 2.84 .81
Data collection and analysis 3.17 .67 2.43 .64 2.66 .87 2.86 .54
Information and argumentation 2.94 .75 2.61 .56 2.82 .63 2.93 .65
Structure, wording, presentation 3.20 .54 2.52 .74 3.12 .70 2.85 .71
Relevance 3.15 .71 2.80 .81 2.89 .63 2.90 .59
Creativity and originality 3.04 .76 2.44 .49 2.64 .58 2.67 .76
Over-all quality 3.15 .68 2.67 .71 2.79 .63 2.92 .68

Panel total score 3.16 .55 2.59 .58 2.73 .58 2.90 .54

30 Pe aks i n s i g ht
3.2.4 Ranking the theses Table 3.3 ⁄ MA and RMA theses as ranked by the four
Each thesis included in the study was ranked according to assessment panels.
its panel total score rating. Table 3.3 presents the results for
each of the four assessment panels. As would be expected, Ranking P&N MS CS E&B
RMA theses are predominantly found in the top half of the
rankings, and MA theses are predominantly found in the 1 RMA RMA RMA RMA
bottom half of the rankings. For each of the four assessment 2 RMA RMA RMA RMA
panels, the proportion of ‘correctly’ ranked theses is expressed 3 RMA RMA RMA RMA
as a p-value, ranging from .67 for the Media Studies panel 4 RMA MA RMA RMA
to .90 for the Economics & Business panel. These results 5 RMA MA RMA RMA
indicate that the MA and RMA theses could adequately be 6 MA RMA RMA MA
distinguished based on panel total scores, suggesting that 7 RMA MA RMA RMA
the measurements have a satisfactory level of external 8 RMA RMA MA RMA
validity. 9 MA RMA RMA RMA
10 MA RMA MA RMA
11 MA RMA MA RMA
3.3 ⁄ Primary analyses 12 MA MA RMA MA
13 RMA MA MA MA
3.3.1 Differences in quality between MA and 14 RMA RMA MA MA
RMA theses 15 MA MA MA MA
Two series of analyses were performed to test for differences 16 MA MA MA MA
in quality between the samples of MA and RMA theses. 17 MA MA MA MA
Table 3.4 presents the results of a first series of t-tests, 18 MA MA MA MA
indicating that, on average, RMA theses were given 19 – – – MA
significantly higher ratings than MA theses. These statistics 20 – – – MA
indicate a very powerful effect of the type of study programme
(MA or RMA) on the ratings in all four assessment panels. RMA in top 50% 7:9 6:9 8:9 9:10
That is, significant differences in panel total scores were also P correct .78 .67 .89 .90
found for those programmes (Media Studies and Economics
& Business), in which there were no significant differences in
supervisor’s grades between MA and RMA theses.

Table 3.4 ⁄ Differences in panel total scores between MA and RMA theses for each of the four study sites.

MA RMA
M SD min max M SD min max d

Psychology & Neuroscience 2.83 .48 1.89 3.52 3.49 .42 2.81 4.00 1.47b
Media Studies 2.31 .48 1.67 3.15 2.87 .55 2.26 3.78 1.08c
Communication Science 2.25 .34 1.83 2.83 3.20 .31 2.80 3.57 2.89a
Economics & Business 2.48 .30 2.07 3.07 3.33 .37 2.75 3.89 2.51a

Total 2.47 .45 1.67 3.52 3.22 .46 2.26 4.00 1.66a

a
p<.00 1; b
p < .0 1; c
p < .0 5

NVAO Research M aster Review 20 1 1 31

 3.3.2 Differences in quality between theses with similar
supervisor’s grades
Curious about whether significant differences in the assess-
ment panels’ ratings would also be found when comparing
theses with similar supervisor’s grades, we performed a
second series of t-tests. To this end, we first analysed the
distribution of supervisor’s grades in the total sample of
74 theses. Aiming to form groups of roughly equal size, we
decided on a division of five levels, subsequently allocating
each thesis to one of these five levels: Level 1 (including
theses with supervisor’s grades 6-7), Level 2 (including
theses with supervisor’s grade 7.5), Level 3 (including theses
with supervisor’s grade 8.0), Level 4 (including theses with
supervisor’s grade 8.5), and Level 5 (including supervisor’s
grades 9-10).
Table 3.5 presents the results of these analyses. Considering
the number of theses allocated to each level, it becomes clear
that there is a large difference in the way that supervisor’s
grades are distributed in the MA and RMA samples. The large
majority (60%) of MA theses were allocated to Levels 1 and 2,
whereas half of all RMA theses were allocated to Levels 4
and 5.
Significant differences were found for three of the five groups
(Levels 2, 3 and 4). For Level 1 and Level 5, the difference in
sample sizes was too large to perform a t-test. The results
of these analyses also point to a powerful effect of the type
of study programme on the assessment panels’ ratings.
Furthermore, these results suggest that supervisors apply
stricter norms for RMA theses than for MA theses.
3.4 ⁄ Discussion
3.4.1 Conclusions
To what extent is there a difference in quality between
theses written by students enrolled on research master’s
programmes (RMA) and those written by students enrolled
on regular master’s (MA) programmes?
We have seen a marked difference in the quality between MA
and RMA theses. First, when theses were ranked according
to assessment panels’ ratings, RMA theses were in the top 3
for all study sites, and in the top 5 for all but Media Studies.
Secondly, we found that assessment panels’ ratings are,
on average, significantly higher for RMA theses than for MA
theses. Thirdly, when comparing MA and RMA theses with
similar supervisor’s grades, assessment panels’ ratings were
consistently higher for RMA theses than for MA theses.
How does the quality of Dutch (research) master’s theses
relate to international standards?
An answer to this question must be a tentative one. There is
no such thing as ’international standards’ in the sense of
an internationally accepted assessment framework with
fixed standards. However, by employing panel members
from outside the Netherlands, we have made sure that inter­
national norms have been applied in assessing the theses.
All panels agree that the MA theses assessed are in line with
international standards. To this, the Communication Science
panel adds that most of the RMA theses assessed are
certainly above average in comparison to international
standards. The other three panels all state that a (vast)

Table 3.5 ⁄ Differences in panel total scores between MA and RMA theses for five levels of supervisor’s grades.

MA RMA
Grade level N M SD min max N M SD min max d

Level 1 (6 – 7) 11 2.35 .38 1.83 2.85 4 2.77 .25 2.44 3.06 1.32d
Level 2 (7.5) 11 2.35 .45 1.67 3.15 5 3.24 .53 2.44 3.78 1.81b
Level 3 (8.0) 7 2.43 .42 1.93 3.11 9 3.14 .57 2.26 3.80 1.45c
Level 4 (8.5) 4 2.70 .38 2.26 3.19 6 3.14 .15 2.93 3.33 1.66c
Level 5 (9 – 10) 4 2.96 .52 2.27 3.52 13 3.45 .42 2.80 4.00 1.05d

Total 37 2.47 .45 1.67 3.52 37 3.22 .46 2.26 4.00 1.66a

a
p<.0 0 1; b
p < .0 1; c
p < .0 5; d
p > .05

32 Pe aks i n s i g ht
majority of theses assessed are of top international level.
In fact, the Media Studies panel rated three RMA as excellent,
although the sample included only ’average’ theses (with
supervisor’s between 7 and 8) and did not include the best
theses available (with grades of 8.5 or higher). Therefore, it
would be safe to conclude that the quality of Dutch (research)
master’s theses relate favourably to international standards.
3.4.2 Strengths and limitations of the study
Strengths
The assessment panels agreed that the overall procedure
made sense and that blind assessment is a useful method
for objectively identifying differences in thesis quality.
Neither panel members nor secretary knew beforehand which
were the (R)MA theses. However, one of the assessment
panels could identify a small number of theses as RMA,
either by indications in the text, or because of the format. In
these instances, the panel decided to ignore the indications.
A second strength is the level of internal consistency in the
measurements of thesis quality. A third and last strength is
the apparent external validity of the TAS measurements of
thesis quality. When the theses were ranked according to the
assessment panels’ ratings, RMA theses were predominantly
found in the top half of the rankings.
Limitations
One limitation of TAS lies in the design of the assessment
framework and the accompanying rating scale. Aiming for a
concise and practical framework consisting of a limited set
of quality criteria, necessarily caused several criteria to be
multidimensional.
TAS results may have been influenced by sampling artefacts.
First, given limited time and money, we could only select a
small number of theses. Larger numbers would certainly
contribute to the representativity of the sample. Second,
with the assessment panels being composed of international
experts, the sample could only include theses written in
English. Third, our sample design did not prevent large diffe-
rences in the distribution of supervisor’s grades between the
samples of MA and RMA theses. Ideally, the samples of MA
and RMA theses would have represented the full variability
in quality present in the two populations, yet displaying no
significant differences in supervisor’s grades.
3.4.3 Implications
Which factors are responsible for the difference in quality
between RMA and MA theses? As our study did not
specifically address this issue, we can only propose
tentative explanations.
At least three interacting factors can be considered to
contribute to the quality of a thesis:
> the quality of the student writing the thesis,
> the quality of the curriculum in which the student is
enrolled, and
> the quality of the teacher acting as the thesis supervisor.
Some of the salient differences between RMA and MA
programmes regarding these three factors seem obvious
candidates for explaining the difference in thesis quality:
> The quality of the student: most MA programmes have no
other admission requirement than a relevant BA-degree,
whereas enrolment into RMAs is reserved for students with
above-average academic achievements and motivation.
> The quality of the curriculum: the RMA programmes have a
considerably stronger focus on the acquisition of research
abilities than regular MA programmes. Also, MAs have a
programme length of one year (60 EC), whereas RMAs have
a programme length of two years (120 EC). This allows RMA
students more time to deepen and widen their substantive
and methodological knowledge, and more time to work on
their thesis.
> The quality of the teacher: the core teaching staff in RMAs
consists of highly qualified and experienced researchers,
constituting academic environments, which often are of a
considerably higher quality level than those in which regular
MA programmes are offered.
The underlying factor that seems to account for all the
differences in the quality of students, curricula, and teachers
is: high expectations. In comparison with regular MA
programmes, substantially higher expectations are put on
the enrolling students, on the teachers, and on the intended
learning outcomes of RMA programmes. RMA research
internship assignments appear, on average, to be more
challenging than MA assignments. Consequently, RMA
assignments would have greater potential for developing
into top theses. Some of the factors discussed above may
contribute to enabling students to actually make a
challenging assignment into a top thesis. One of these factors
could well be the very much larger amount of courses and
skills training that RMA students can build from by the time
they embark on their research assignment.
NVAO Research M aster Review 20 1 1 33
Interview

34 Pe aks i n s i g ht
Helena Cousijn
Cognitive Neuroscience
Two years ago, Helena Cousijn completed the research
master’s programme Cognitive Neuroscience at Radboud
University Nijmegen. Currently, she is enrolled on a PhD
programme at the University of Oxford: “My interest in
research was awakened well before my studies started.
I did a bachelor’s programme in Cognitive Psychology,
before I started with Cognitive Neuroscience in Nijmegen.
Research on the brain and nervous system is the main focus
of that programme. I wanted to elaborate on my foundational
knowledge of psychology by increasing my knowledge of
neurobiology. The first year of the curriculum consists of a
broad range of different courses. The second year is fully
dedicated to the research internship.
The programme exceeded my expectations. It was hard work,
because the expectations are certainly high. But the quality
was excellent; the programme goes very deeply into the
subject matter. When I now compare it with courses I did as a
PhD student at the University of Oxford, I am still amazed at
the profundity of the courses I did in Nijmegen. The internship
was very good as well; I had a high degree of autonomy and
learned a lot. It also resulted in a few publications.
During my internship, I studied a certain gene which we
supposed to be somehow related to stress. A group of
subjects was genotyped, put in a scanner, and then exposed
to some very unpleasant videos which were intended to
induce stress. We have published the results in PNAS,
Proceedings of the National Academy of Sciences of the
United States of America, an influential multidisciplinary
periodical. Someone else has now used the data from that
video study for a publication in Science, which I co-authored.
Two years ago, I started in Oxford. I continue to work in the
same field, still investigating how genes influence brain
activity. The difference is that I now study a more psychiatric
subject: schizophrenia. My main question is how a certain
gene – called ZNF804a – could lead to the development of
schizophrenia. To examine the effects of that gene, I combine
neuroimaging techniques with methods to study post-mortem
human brains on a molecular level.
Also in the future, science will continue to be my favourite
sphere of activity. I don’t know whether that will be in
the Netherlands, I have not made any specific plans yet.
Science is an international endeavour, so I can very well
imagine myself ending up somewhere else.”
NVAO Research M aster Review 20 1 1 35
Chapter 4 ⁄
Student
satisfaction
in MA and
RMA
programmes

36 Pe aks i n s i g ht
 To what extent do MA and RMA students
differ in satisfaction with their study
programme? Which aspects of the study
programme are RMA students more
satisfied about than MA students? And
which aspects are they less satisfied
about?
especially more positive about the academic skills acquired
through the programme, about their teachers and lecturers,
and about the academic guidance, support and supervision
offered by the programme. However, RMA students are less
positive about their study load, the programme timetables,
the extent to which they learn how to work together with
others, the availability and affordability of accommodation
in the city where they are studying, and about several facilities
in their institution and student-city environment.

Student satisfaction is an important indicator of the quality 4.1 ⁄ Overall ratings

of study programmes. Using data from the Dutch National
Student Survey (NSE; Muskens et al., 2010), we aimed to
identify differences in satisfaction between MA and RMA
students. First of all, the respondents were asked about their
overall satisfaction with their study programme. They were
asked to provide an overall rating for their study programme in
general. Subsequently, the respondents were asked to provide
a rating of 14 aspects of their study programme. For each of
these 14 aspects, several sub-questions followed. Each item
was rated on a 5-point scale (1 = very dissatisfied, 5 = very
satisfied). The ratings of the sub-questions were used to
calculate a ’theme score’ for each of the main aspects.
Our results indicate that MA and RMA students are generally
quite satisfied with their programme, although satisfaction is
somewhat higher in RMA students. RMA students are
Table 4.1 indicates that the overall ratings of MA students
are quite positive. However, RMA students gave significantly
higher ratings on most of these 15 items. For each item,
the standardised mean difference (Cohen’s d) is calculated,
indicating the extent to which MA and RMA students differ in
their average ratings. When asked for an overall rating, RMA
students seem especially more positive about the general and
academic skills learned in the programme, the preparation
for a professional career, the teachers and lecturers in the
programme, the academic guidance, support and supervision
offered, and about the extent to which they are encouraged to
help improve the programme.

Table 4.1 ⁄ Differences between MA and RMA students’ overall ratings.

MA RMA
N M SD N M SD d

Study programme in general 35,793 3.90 .80 2,858 4.02 .85 .15a
Content of the programme 35,042 3.91 .80 2,823 4.00 .85 .12a
General skills 34,965 3.74 .84 2,817 3.97 .84 .28a
Academic skills 34,994 3.85 .85 2,822 4.19 .86 .40a
Preparation for a professional career 34,430 3.20 1.03 2,762 3.56 1.05 .35a
Teachers/ lecturers 35,019 3.86 .80 2,822 4.07 .81 .26a
Information provided 34,954 3.48 .93 2,812 3.50 1.01 .03d
Study facilities 34,782 3.65 .93 2,811 3.80 .96 .16a
Assessment and grading 34,962 3.53 .86 2,812 3.65 .88 .13a
Programme timetables 34,854 3.70 .88 2,799 3.71 .90 .02d
Study load 34,976 3.59 .83 2,816 3.57 .87 -.03d
Academic guidance, support and supervision 34,247 3.36 1.01 2,797 3.69 1.05 .33a
Other facilities and student-city environment 34,336 3.65 .86 2,767 3.75 .88 .12a
Overall atmosphere 34,953 4.02 .82 2,818 4.12 .91 .12a
Encouragement to help improve the programme 34,229 3.40 .99 2,791 3.76 1.05 .37a

a
p<.00 1; d
p > .0 5

NVAO Research M aster Review 20 1 1 37
 Table 4.2 ⁄ Satisfaction with the content of the study programme.

MA RMA
N M SD N M SD d

Content of the programme (theme score) 34,654 3.74 .62 2,811 3.90 .68 .27a
Level of the programme content 34,656 3.89 .83 2,809 4.12 .85 .28a
Meets the initial expectations 34,582 3.75 .92 2,806 3.80 1.04 .06b
Fits in with education 33,885 3.77 .90 2,780 3.83 .95 .06b
Content is stimulating 34,635 3.81 .89 2,811 4.03 .94 .25a
Content draws on current developments 34,441 3.87 .88 2,769 3.97 .91 .11a
Coherence of the programme 34,581 3.69 .87 2,804 3.68 .95 -.02d
Instructional formats 34,534 3.67 .85 2,797 3.82 .87 .17a
Opportunity to shape your own programme 34,060 3.34 1.07 2,782 3.84 1.11 .46a
Quality of the study materials 34,560 3.82 .76 2,789 4.04 .78 .30a

a
p<.00 1; b
p < .0 1; d
p > .0 5

Table 4.3 ⁄ Satisfaction with the general skills acquired in the study programme.

MA RMA
N M SD N M SD d

General skills (theme score) 34,364 3.79 .64 2,791 3.97 .66 .28a
Critical attitude 34,396 3.96 .84 2,798 4.26 .84 .35a
Ability to function in a professional setting 34,182 3.58 .94 2,770 3.87 .95 .30a
Problem-solving skills 34,313 3.81 .85 2,779 3.96 .86 .18a
Ability to draw substantiated conclusions 34,392 3.93 .80 2,793 4.13 .82 .26a
Communication skills 34,302 3.64 .96 2,781 3.85 .92 .22a
Working together 34,163 3.73 .93 2,756 3.64 .97 -.10a
Argumentation/reasoning skills 34,268 3.84 .84 2,795 4.05 .85 .25a

a
p<.00 1

Table 4.4 ⁄ Satisfaction with the academic skills acquired in the study programme.

MA RMA
N M SD N M SD d

Academic skills (theme score) 34,225 3.68 .75 2,790 4.10 .75 .57a
Analytical thinking 34,163 3.96 .81 2,787 4.21 .83 .30a
Ability to critically evaluate scientific work 34,032 3.86 .90 2,779 4.29 .85 .48a
Ability to formulate scientific theories 33,702 3.43 .96 2,758 3.89 .97 .48a
Writing scientific papers 33,694 3.58 1.00 2,763 4.08 .95 .49a
Research methods and techniques 33,767 3.56 1.00 2,776 4.04 .98 .48a
The ability to perform independent research 33,789 3.66 .98 2,771 4.13 .93 .48a

a
p<.00 1

38 Pe aks i n s i g ht
4.2 ⁄ Content of the programme of study
Table 4.2 shows that, on average, MA students are quite
satisfied with the content of their programme. However, RMA
students are a little more positive, especially about the level
of the programme content, the extent to which the programme
is stimulating, the opportunity to shape their own programme
of study, and the quality of the study materials. No significant
difference was found in MA and RMA students’ ratings of the
coherence of the programme.
4.3 ⁄ Acquisition of general skills
Table 4.3 displays the differences between MA and RMA
students’ satisfaction with several general skills acquired in
the study programme. Again, although MA students appear
quite satisfied, RMA students gave significantly higher ratings
for most of these items. However, RMA students are less
satisfied than MA students about the extent to which they
learn how to work together with others.
4.4 ⁄ Acquisition of academic skills
As may be expected, RMA students are clearly more satisfied
than MA students with the academic skills acquired in the
programme. Table 4.4 shows that the standardised mean
differences (Cohen’s d) between MA and RMA students are
larger for these items than for all the other items in the
questionnaire.
4.5 ⁄ Preparation for a professional
career
For both MA and RMA students, the preparation for a
professional career is among the least positively rated
aspects of the study programme. However, as displayed
in Table 4.5, RMA students are moderately more satisfied
than MA students.
4.6 ⁄ Teachers & lecturers in
the programme
For both MA and RMA students, the quality and availability
of teachers and lecturers is among the most positively rated
aspects of the study programme. However, the results
presented in Table 4.6 indicate that RMA students are a
little more positive about their teachers than MA students,
especially about their teachers’ engagement with students.
4.7 ⁄ Assessment and grading
Table 4.7 shows that RMA students are more satisfied
than MA students with the extent to which assessments
and grades are aligned with programme content and the
turn-around time for assessments and grades. However,
compared to MA students, RMA students are less satisfied
with the clarity of the grading criteria. Although significant,
the differences between the two groups of students are small.
4.8 ⁄ Study load
Table 4.8 indicates that both MA and RMA students are
quite positive about their study load. However, on average,
satisfaction with study load is somewhat lower in RMA
students.
4.9 ⁄ Academic guidance, support and
supervision
As may be expected, RMA students are more satisfied than
MA students with the quality and availability of academic
guidance, support, and supervision offered by the programme.
Table 4.9 indicates that the differences in the average ratings
between the two groups of students are quite substantial.
NVAO Research M aster Review 20 1 1 39
Table 4.5 ⁄ Satisfaction with the study programme’s preparation for a professional career.

MA RMA
N M SD N M SD d

Preparation for a professional career (theme score) 32,579 3.13 1.00 2,576 3.43 .98 .30a
Professional skills 33,097 3.20 1.12 2,651 3.54 1.05 .30a
Practice-based focus 33,318 3.14 1.17 2,619 3.39 1.12 .21a
Contact with professional practice 32,762 3.07 1.21 2,589 3.37 1.19 .25a
Quality of placement supervision 22,986 3.17 1.16 1,869 3.47 1.14 .26a
Extent to which the placements fit in with the curriculum 22,507 3.32 1.09 1,870 3.60 1.04 .26a

a
p<.00 1

Table 4.6 ⁄ Satisfaction with the teachers/lecturers in the programme.

MA RMA
N M SD N M SD d

Teachers/lecturers (theme score) 33,583 3.74 .68 2,754 3.99 .67 .36a
Teachers'/lecturers' subject area expertise 33,648 4.20 .73 2,757 4.42 .71 .31a
Quality of teaching methods 33,622 3.68 .85 2,756 3.80 .85 .14a
Teachers'/lecturers' availability outside of class hours 32,487 3.67 .97 2,723 3.97 .93 .32a
Teachers'/lecturers' engagement with students 33,558 3.69 .95 2,755 4.04 .92 .37a
Quality of supervision 33,425 3.68 .88 2,736 3.95 .88 .31a
Quality of feedback 33,471 3.55 .94 2,751 3.70 .98 .17a
Extent to which teachers/lecturers inspire you 33,613 3.66 .92 2,752 3.94 .92 .30a
Teachers'/lecturers' knowledge of professional practice 33,038 3.83 .92 2,671 4.09 .89 .29a

a
p<.00 1

Table 4.7 ⁄ Satisfaction with the assessment and grading in the study programme.

MA RMA
N M SD N M SD d

Assessment and grading (theme score) 33,269 3.44 .78 2,735 3.48 .84 .04c
Clarity of the grading criteria 33,267 3.44 .95 2,737 3.40 1.01 -.04c
Assessments and grades are aligned with programme content 33,211 3.59 .87 2,724 3.67 .91 .08a
Turn-around time for assessments and grades 33,166 3.29 1.07 2,720 3.36 1.09 .07b

a
p<.00 1; b
p < .0 1; c
p < .0 5

40 Pe aks i n s i g ht
Table 4.8 ⁄ Satisfaction with the study load.

MA RMA
N M SD N M SD d

Study load (theme score) 33,123 3.49 .78 2,731 3.43 .83 -.08a
Distribution over the academic year 33,056 3.48 .97 2,729 3.41 1.01 -.07a
Manageability of deadlines 33,043 3.67 .84 2,727 3.62 .89 -.05c
Relation between EC credits and study load 32,853 3.33 .98 2,705 3.26 1.05 -.07b

a
p<.00 1; b
p < .0 1; c
p < .0 5

Table 4.9 ⁄ Satisfaction with the academic guidance, support and supervision.

MA RMA
N M SD N M SD d

Academic guidance, support and supervision (theme score) 29,167 3.53 .92 2,578 3.86 .94 .36a
Availability academic guidance, support and supervision 29,124 3.53 .97 2,576 3.85 .99 .33a
Quality of academic guidance, support and supervision 26,793 3.56 .95 2,490 3.89 .96 .35a

a
p<.00 1

4.10 ⁄ Other aspects Programme timetables

The study also covered other aspects of students’ satisfaction,
as shown in the overall ratings in Table 4.1. Since these
aspects are less central to a comparison of MA and RMA
programmes, we will cover them in less detail.
RMA students give slightly lower ratings than MA students
for the timely publication of the programme timetables and
changes in the programme timetables. However, MA and
RMA students are equally satisfied with the suitability of their
timetables.

Information provided by the institution
RMA students give higher ratings than MA students for the
information provided by the institution about the programme
and the students’ progress through the programme. However,
the differences between MA and RMA students are small.
Study facilities and information facilities
MA and RMA students are equally positive about their library,
their digital learning environment, and their information desk.
RMA students give significantly higher ratings for other study
facilities (teaching rooms, student work areas, learning
materials, ICT facilities) and their student administration unit
than MA students. However, the differences between the two
groups of students are small.
Other facilities and the student-city environment
Compared to MA students, RMA students are less satisfied
with the restaurant and canteen facilities at their institution,
the catering facilities in the city where they are studying, and
with its range of cultural facilities. RMA students do give
slightly higher ratings for the ease with which they can reach
their institution and the amount of green space in the city
where they are studying. MA and RMA students are equally
satisfied with the sports facilities at their institution.
Both MA and RMA students are somewhat negative about
the accommodation in the city where they are studying.
Moreover, RMA students give significantly lower ratings for
the availability and affordability of accommodation than
MA students, and their monthly rent is slightly higher.

NVAO Research M aster Review 20 1 1 41

4.11 ⁄ Discussion
The results discussed in this chapter, indicate that both
MA and RMA students are quite positive about their study
programme. With only a few exceptions, the average ratings
fall above the mid-point of the 5-point rating scale. For most
items in the questionnaire, however, significant differences
were found between the average ratings of MA and RMA
students. In other words, the type of programme appears to
have an effect on student satisfaction.
Positive effects (where RMA students gave higher average
ratings than MA students) were found for the following
aspects:
> the content of the study programme
> the acquisition of general skills
> the acquisition of academic skills
> the preparation for a professional career
> the teachers and lecturers in the programme
> the academic guidance and support
> the engagement with students.
Negative effects (where RMA students gave slightly lower
average ratings than MA students) were found for the
following aspects:
> the timely publication of programme timetables and
timetable changes
> the study load
> the availability and affordability of accommodation in
the city where they are studying
> the extent to which students learn to work together
with others
> the clarity of grading criteria.
Notably, RMA students are less satisfied than MA students
with the extent to which they learn how to work together with
others. This finding underscores our conclusions reported in
Chapter 2, in which we have voiced our concerns about the
very small student intake of many RMAs. For these
programmes, increasing student enrolment seems necessary,
if they are to provide students with the opportunity to reap the
benefits of collaborative learning, and learn how to co-operate
and work in teams.

Smaller positive effects were found for the information

provided by the institution, study facilities and information
facilities, and assessment and grading.

Compared to MA students, RMA students are especially more

satisfied with the opportunity to shape their own study
programme, with the extent to which they acquire a critical
attitude, knowledge about research methods and techniques,
and the abilities needed for critically evaluating scientific
work, formulating scientific theories, writing scientific papers,
and performing independent research. These positive results
are quite in line with the three distinguishing features of
RMAs (a focus on the acquisition of research abilities, a high-
quality academic context, and selection of students with
above-average academic performances and ambitions).
However, the effects were not positive for all items included in
the questionnaire.

42 Pe aks i n s i g ht
NVAO Research M aster Review 20 1 1 43
Interview

44 Pe aks i n s i g ht
Alexander Sack

Cognitive and Clinical

Neuroscience
Alexander Sack, professor of Neurocognition at Maastricht
University and a member of The Young Academy1:
“The students of our research master’s programme produce
excellent theses. About three-quarters of them enrol in a PhD
programme at a globally recognised top institution. We have
also put a lot of effort, enthusiasm, and time in designing the
curriculum. Students have a lot of face-to-face instruction,
and they are trained through problem-based learning.
They work in small groups of maximally 12 students, and they
are tutored by university professors and/or top researchers in
the field.
Also important is the hands-on training, for example working
with an fMRI scanner, or in a TMS lab. An fMRI scanner
visualises the structure and function of a working brain.
So we can see which part of the brain is active when, for
example, a subject is presented a certain picture or executes
a task. With TMS you can stimulate certain areas of the
brain with magnetic pulses, thereby influencing a subject’s
behaviour, cognition or emotion.
Maastricht University is a university with a vision. The Faculty
of Psychology and Neuroscience has several ‘Principal
Investigators’ who enjoy world-fame in their field.
Furthermore, our research infrastructure is world-class.
Currently, The Maastricht Brain Imaging Center (MBIC) has
its own fMRI scanner, two TMS labs, and five EEG labs.
The now launched Brains Unlimited project will further
expand this infrastructure by the end of 2012, adding two
additional high-field MR Scanners (7 and 9.4 Tesla) to the
overall neuroscience research environment. You will not easily
find such high-quality facilities elsewhere.
Academically, Germany and the Netherlands have a lot
in common. A salient difference, however, is that the Nether-
lands seems to be more flexible and dynamic in adapting to
change. For example, with the bachelor/master system: the
Netherlands was one of the first countries in Europe to imple-
ment the system, and did so in a swift and effective manner.
In contrast, Germany is still in the middle of that process.
The report by the Veerman Committee covered several
important issues. More diversity, less redundancy, and more
specialisation are certainly necessary. I also understand the
necessity of cutbacks on the education budget in financially
difficult times. But we also have to realise that while the
Netherlands may be a small country geographically, in the
world of science it is one of the top players.
When cutbacks have a negative influence on the research
climate, top researchers who can easily acquire positions
elsewhere, will be lost. But those are exactly the kind of
people you want to retain. It is not a good idea to reduce
expenditure on higher education in a country like the
Netherlands, which is dependent on the knowledge,
education, creativity and innovative power of the labour
force. That would mean cutting back on our future and
the knowledge of generations to come.”

1
Th e Yo u n g Ac ad em y, a n in d e p e n d e n t se ct ion of t h e Roya l
Nether l an d s Ac ad em y of Ar t s a n d S cie n ce s ( KN AW ), is a
dyn a m i c an d i n n ov at i ve p l at for m of top youn g scie n t ist s a nd
sch o l ars t h at s elect s it s m e m b e rs for t h e ir re se a rch exce lle nce,
in terd i s c i p l i n ar y app roa ch , a n d b roa d in te re st in scie n ce and
s cien c e c o m m u n i c at i on .

NVAO Research M aster Review 20 1 1 45

Chapter 5 ⁄
The alumni
perspective
Jorrit Snijder (Roosevelt Academy / CHEPS University of Twente) and Steven A. David (NVAO)

46 Pe aks i n s i g ht
To what extent do research master’s Table 5.1 ⁄ Additional admission criteria.
programmes attain their objectives?
MA RMA Total
Do they increase the number and quality
of young researchers? Do they indeed "No" 1.027 64% 92 16% 1.119 51%
provide a better preparation for a career "Yes" 568 36% 501 84% 1.069 49%
in research than regular master’s
Total 1.595 100% 593 100% 2.188 100%
programmes?

The academic community in the Netherlands has welcomed

the RMA as a successful and promising educational A high bachelor’s grade point average (BA GPA) and a high
innovation. However, little is known about the effectiveness grade for the bachelor’s thesis are the most frequently
of this innovation. To gauge the first outcomes, we surveyed mentioned additional admission requirements. Table 5.2
three cohorts of alumni from research master’s (RMA) and shows that, on average, RMA alumni have significantly
regular master’s (MA) programmes from nine Dutch higher BA GPAs and BA thesis grades than MA alumni.
universities, between 1.5 and 2 years after their graduation.
Table 5.2 ⁄ Differences in bachelor’s grade point average
Our results indicate that RMAs are clearly more selective and bachelor’s thesis grade.
than MA programmes. Our results also indicate that,
compared to MA alumni, RMA alumni: MA RMA
> are more satisfied with research related aspects of their N M SD N M SD d
study programme;
> are more satisfied with the quality and availability of the BA GPA 1,476 7.5 .61 555 7.9 .68 .72a
teaching staff and thesis supervisors; BA thesis 1,374 7.8 .81 483 8.3 .78 .62a
> have more instruction time, a higher workload, and higher
study results; a
p <.0 0 1
> are more satisfied with the extent to which their programme
prepared them for professional practice; and
> more frequently have a research oriented job. More than half 5.2 ⁄ Satisfaction with research related
of all the RMA alumni in our sample have enrolled on aspects of the programme
a PhD programme.
Respondents were asked to rate (research related aspects
5.1 ⁄ Selectiveness of the study of) their master’s programme on a scale of 1 (lowest) to 10
programme (highest). Table 5.3 shows that MA graduates, on average,
are quite satisfied with their programme. However, on
Table 5.1 clearly shows that RMA programmes are more average, RMA graduates are significantly more satisfied.
selective than MA programmes. For most RMA alumni in the The largest differences are found for research related
sample (84%) there were additional criteria for entrance aspects of the programme: direct participation in research,
into the master’s programme besides a bachelor’s degree, development of research skills, and insight in cutting edge
whereas there were no additional criteria for the majority of developments in the field. More than the MA alumni, the
MA alumni (64%). But remarkably, 16% of the RMA alumni RMA alumni in the sample think their programme provided a
also report that their programme had no additional admission good basis for writing an article for a peer-reviewed journal,
requirements besides a BA degree. and for finalising a PhD within four years.

NVAO Research M aster Review 20 1 1 47

 Table 5.3 ⁄ Differences in satisfaction with the programme.

MA RMA
N M SD N M SD d

Overall grade for study programme 1,551 7.2 1.13 576 7.8 1.25 .48a
Direct participation in research 1,565 6.6 2.00 577 8.0 1.84 .73a
Providing a solid theoretical basis 1,571 7.3 1.39 579 7.7 1.57 .29a
Insight in cutting edge developments in the field 1,564 6.6 1.65 575 7.6 1.66 .60a
Development of research skills 1,567 7.2 1.60 577 8.1 1.51 .60a
International focus 1,567 6.3 2.38 579 7.4 1.97 .48a
Information about study and career opportunities 1,569 5.2 2.01 577 5.8 2.17 .33a

a
p<.00 1

Table 5.4 ⁄ Differences in satisfaction with teachers and thesis supervisors.

MA RMA
N M SD N M SD d

Research qualities of teachers 1,568 7.4 1.49 577 8.2 1.40 .55a
Availability of teachers 1,566 7.1 1.56 575 8.0 1.62 .55a
Quality of thesis supervisors 1,458 7.2 1.80 551 7.9 1.77 .35a

a
p<.00 1

Table 5.5 ⁄ Differences in instruction time, total workload, master’s grade point average and master’s thesis grade.

MA RMA
N M SD N M SD d

Instruction time (hours/week) 1,557 11.3 7.35 573 14.5 10.16 .40a
Total workload (hours/week) 1,562 30.1 10.51 566 36.8 10.80 .63a

MA GPA 1,569 7.6 .58 574 8.0 .59 .70a

MA thesis 1,574 7.7 .72 581 8.2 .72 .78a

a
p<.00 1

48 Pe aks i n s i g ht
5.3 ⁄ Satisfaction with teachers and
thesis supervisors
Respondents were also asked to rate the quality and
availability of their teachers and thesis supervisors on a
scale of 1 (lowest) to 10 (highest). Table 5.4 presents the
differences in average ratings between MA and RMA
alumni. Again, satisfaction appears to be highest among
the RMA graduates.
Table 5.6 displays the numbers and proportions of MA and
RMA graduates that are employed in either a non-research
job, a research job, or a mixed job. The majority of MA alumni
(65%) and a minority of RMA alumni (20%) report having a
non-research job. In contrast, the majority of RMA alumni
(73%) and a minority of MA alumni (19%) report having a
research job. Only 5% of the MA alumni, but more than half
of the RMA alumni in our sample have enrolled on a PhD
programme.

5.4 ⁄ Instruction time, workload and Table 5.6 ⁄ Current employment of MA and RMA alumni.
study results
MA RMA Total
Respondents were asked to provide an estimate of the
average amount of face-to-face instruction time and their Non-research job 909 65% 105 20% 1,014 53%
total workload (including preparing for classes, following Research job 265 19% 388 73% 653 34%
courses, taking exams, making assignments, etcetera) Mixed job 224 16% 35 7% 259 13%
over the whole master’s programme (in hours per week).
The results presented in Table 5.5 indicate that RMA Total 1,398 100% 528 100% 1,926 100%
programmes are more intensive than MA programmes.
Compared to MA alumni, RMA alumni report a significantly
higher amount of instruction time (over 3 hours per week
more), and a significantly higher total workload (almost 5.6 ⁄ Satisfaction with preparation for
7 hours per week more). professional practice
Respondents were also asked to indicate their grade point Respondents were asked to rate the extent to which their
average for their master’s programme (MA GPA) and the grade master’s programme prepared them for professional practice
they had been awarded for their master’s thesis. Table 5.5 on a scale from 1 (lowest) to 10 (highest). They were also
indicates that RMA graduates have significantly higher MA asked to indicate their agreement (on a 5-point scale, with
GPAs and MA thesis grades than MA graduates. So the extra 1=strongly disagree, and 5=strongly agree) with two state-
effort seems to pay off in higher study results. ments: “Without this master's programme, I would not have
obtained this job”; and “In my current job, I can put my
5.5 ⁄ Current employment competencies into practice”. Table 5.7 summarises the results.
Compared to the MA graduates in our sample, the RMA
Our results indicate a small difference in employment rate graduates score significantly higher on all three items.
between MA and RMA alumni: the proportion of respondents
in paid employment, 1.5 to 2 years after graduation, is a little
higher for MA alumni (88%) than for RMA alumni (84%).
However, large differences are found between MA and RMA
alumni in the types of jobs in which they are employed.

NVAO Research M aster Review 20 1 1 49

Table 5.7 ⁄ Differences in satisfaction with preparation for professional practice.

MA RMA
N M SD N M SD d

Preparation for professional practice 1,566 5.9 1.98 578 7.0 1.99 .53a
Without this master's programme, I would not have obtained this job. 1,412 3.3 1.54 530 3.9 1.38 .42a
In my current job, I can put my competencies into practice. 1,415 3.5 1.24 531 4.2 1.10 .58a

a
p<.00 1

5.7 ⁄ Discussion
In an earlier study, we (NVAO, 2007) found that RMAs
distinguish themselves from regular MA programmes
in three respects: (a) a selection of students with above-
average academic performances and ambitions, (b) a
curriculum which is specifically focused on the acquisition
of advanced research abilities, and (c) an academic context
of unquestionably high quality that strives for excellence.
NVAO’s (2011) Assessment frameworks for research master’s
programmes state that “The intended learning outcomes
enable a transition to trajectories which lead to a PhD, and to
positions requiring research abilities and experience beyond
the level that can be expected on the basis of the usual link
with research within academic higher education”. Regular
MA programmes often aim to prepare for a broad range of
professions. In contrast, RMA programmes have a more
specific focus: they aim to prepare for the profession of
researcher within a certain scientific discipline.
The results of our survey are in line with these differences
between MA and RMA programmes. First of all, the proportion
of alumni reporting additional admission criteria is higher
among the RMA alumni than among the MA alumni. Compared
to MA alumni, RMA alumni have higher BA GPAs and higher
grades for their BA thesis. Second, RMA alumni are more
satisfied about research related aspects of their programme
than MA alumni. Third, RMA graduates are more satisfied
about the quality and availability of teaching staff and thesis
supervisors. Fourth, RMA alumni have had more face-to-face
instruction time, a higher workload, and higher study results.
Fifth, in comparison with MA alumni, a high proportion of RMA
alumni have a research oriented job 1.5 to 2 years after gradu-
ation. Notably, more than half of all RMA alumni in our sample
enrolled on a PhD programme. Sixth, and last, RMA graduates
are more satisfied than MA graduates about the extent to
which their study programme provided a good preparation for
professional practice. Although the results are positive for
both groups, the differences that we found between MA and
RMA alumni are substantial.
Considered from the perspective of alumni, RMAs certainly
seem to realise their aim of providing a thorough preparation
for a career in research. However, the number of RMA gradu-
ates is still small. In other words, RMAs do appear to increase
the quality, but not so much the number of young researchers.
Only through increasing the volume of their output (while
maintaining the high quality standards), will RMAs fully attain
their objectives.

50 Pe aks i n s i g ht
NVAO Research M aster Review 20 1 1 51
Interview

52 Pe aks i n s i g ht
Mirjam de Bruijn

African Studies
Mirjam de Bruijn is professor and director of the research We push our students to graduate within two years, and
master’s programme African Studies at Leiden University: often they manage to do so. Some students are delayed,
“Our programme combines social sciences and humanities. for example, when a student enrols on a PhD before
We have a strong focus on the development of students, completing the research master’s programme. Some
and the field of studies has a broad scope. For example, students need only a few months extra. We have a very
we had a group of students doing research on prisons in low attrition rate, so far only two students have dropped
Cameroon, another group studying migration of young out from the programme.
women in Chad, and a third group looking into markets and
market chains in Uganda. We try to fit students in with We have a large staff of lecturers from the Netherlands
current research projects of our staff. and abroad. That makes it difficult to manage the programme.
Recently, we have diminished the size of the teaching staff.
The programme has an interdisciplinary character; students Our objective for the future, is to further promote inter­
are trained to work with experts from a range of different nationalization efforts. We co-operate a lot with institutions
fields. Each cohort has to function as an interdisciplinary in Africa, and we aim to formalise those co-operations.
team. The sharing of knowledge between students is part Also, we want to develop a study programme in co-operation
of their acquisition of knowledge, also during the writing of with other institutions for African studies across Europe.
the thesis. This would provide our students with the opportunity to do
part of their research master’s programme abroad.”
On average, the intake amounts to 10 students per year.
Of those 10 students, about half come from the Netherlands,
two to three come from Africa, and the others come from
countries across Europe. Nearly 60% continues with a PhD
trajectory at a European university, the other 40% often
quickly finds a job in public administration or with an NGO.
We aim for an annual intake of 15-20 students. Budget
limitations form a large obstacle. Many African students
apply for the programme, but they can only enrol if they have
a scholarship. We try our best to help them with that, but
scholarships are scarce.

NVAO Research M aster Review 20 1 1 53

Chapter 6 ⁄
The
accreditation
of research
master’s
programmes

54 Pe aks i n s i g ht
How are RMA programmes accredited?
Which assessment criteria have the
highest weight on the final conclusion?
Which considerations underlie divergent
judgments of quality standards?
This chapter shows how RMA programmes are assessed for
accreditation by NVAO. NVAO’s assessment frameworks for
RMA programmes imply that RMAs have to meet higher levels
of quality than regular master’s programmes. The committees
that assess the RMAs maintain stern demands, yet are
generally quite positive about the quality of the programmes.
A negative final conclusion is generally due to shortcomings
in the curriculum, and in a few cases (also) to shortcomings
in the aims and objectives, and the quality of staff.
6.1 ⁄ The accreditation of RMA
programmes
To be accredited, all bachelor and master programmes have to
be assessed every six years. For RMAs, a special accreditation
procedure has been established, which reflects the focus on
research that is one of the main features of RMAs.
Applications for the accreditation of RMAs are assessed by
expert committees appointed by the Royal Netherlands
Academy of Arts and Sciences (KNAW). These committees are
composed of three to five Academy members, an expert
from Flanders, and an RMA graduate. The panels deliver
an assessment report with a positive or negative recommen-
dation about the application. Subsequently, based on the
recommendation of the KNAW committee, the Board of NVAO
decides whether the programme meets the requirements for
accreditation.
The framework used for the accreditation of RMA programmes
is an elaboration of the framework used for accreditation
of regular master programmes. This framework contains
requirements about six themes: the (1) aims and objectives
of the programme, (2) the curriculum, (3) staff involved,
(4) facilities, (5) internal quality assurance and (6) results
(for existing programmes), or (6) conditions for continuity
(for new programmes). Each theme encompasses two to
eight underlying standards. The assessment frameworks for
RMA programmes differ from the frameworks for regular
MA programmes in the following points:
> The intended learning outcomes enable a transition to
trajectories which lead to a PhD, and to positions requiring
research abilities and experience beyond the level that can
be expected on the basis of the usual link with research
within academic higher education.
> The research nature of the curriculum can be shown through
comparison with a regular master’s programme and through
comparative positioning in a national and international
perspective.
> Balance and coherence between discipline-specific content
and research abilities characterise the programme in both
years.
> In its admission of students, the programme enforces a
selection with regard to demonstrated knowledge and
skills on the part of the students to the effect that enrolled
students may readily be expected to meet the high
requirements of the programme and to successfully
complete the programme.
> The programme has a scope of 120 credits and a duration
of two years.
> Sufficient research capacity of verified good to excellent
level is available within the academic environment in which
the programme is embedded. Additionally, the staff must
have demonstrable experience in the effective training and
coaching of newly-qualified researchers.
> The programme is completed with a substantial test of
research ability, which can be deemed of value in the
scientific discipline concerned.
In other words, higher norms apply for RMA programmes than
for regular MA programmes, with respect to the aims and
objectives, the content and structure of the curriculum, the
admission of students, the quality of the teaching staff, and
the quality of the master’s thesis which students have to write
in order to complete the programme. Therefore, by definition,
RMAs have to meet higher levels of quality than regular MA
programmes.
6.2 ⁄ KNAW committees’ final
conclusions
In the period from March 2008 to March 2011, NVAO
processed a total of 128 applications, mostly for the
accreditation of existing RMA programmes (n=116) and only
a few for the initial accreditation of new RMA programmes
NVAO Research M aster Review 20 1 1 55
 (n=12). Table 6.1 shows that most applications were assessed
by the Humanities committee (n=56) and the Social Sciences
committee (n=37). Of the 116 applications for existing RMA
programmes 9 (8%) were judged negative, of the 12 appli­
cations for new RMA programmes 4 (33%) were judged
negative. For 7 applications no final conclusion is available,
because these applications were withdrawn before the KNAW
committee had assessed the programme. Aiming to identify
which quality criteria have the highest weight on the final
conclusion, we have analysed how the KNAW committees
have judged the themes and standards.
6.3 ⁄ KNAW committees’ judgments
of themes
A KNAW committee’s final conclusion is negative, if at least
one of the six themes is judged as negative. Table 6.2 shows
that among the applications for the accreditation of existing
RMA programmes, the KNAW committees only gave negative
judgments for Theme 1 (Aims and objectives), Theme 2
(Curriculum), and Theme 3 (Staff). Among applications for the
initial accreditation of new RMA programmes, the KNAW
committees gave negative judgments for Theme 1 (Aims and
objectives), Theme 2 (Curriculum), and Theme 4 (Facilities).
Remarkably, out of the 13 applications for which the final
conclusion by the KNAW committee was negative, 12 were
negatively judged on Theme 2 (Curriculum). The one remaining
application was negatively judged on Theme 3 (Staff).

Table 6.1 ⁄ KNAW committees’ final conclusions.

Accreditation Initial accreditation

KNAW Committee Nega Posb NAc Nega Posb NAc Total

Social Sciences 5 29 1 0 2 0 37
Behavioural Sciences 2 14 0 2 1 0 19
Humanities 2 49 3 0 1 1 56
(Bio)medical Sciences 0 8 2 1 3 0 14
Earth Sciences 0 1 0 0 0 0 1
Computational Science 0 0 0 1 0 0 1

Total 9 101 6 4 7 1 128

a
Neg= n egat i ve f i n al c on cl usion ; b Pos=p osit ive fin a l con cl usi o n; c NA=no fi nal co nclu si o n avai lab le.

Table 6.2 ⁄ KNAW committee’s judgments of themes.

Accreditation Initial accreditation

Theme Neg Pos Total Neg Pos Total

1 Aims and objectives 2 108 110 1 10 11

2 Curriculum 8 102 110 4 7 11
3 Staff 2 108 110 0 11 11
4 Facilities 0 110 110 1 10 11
5 Internal quality assurance system 0 110 110 0 11 11
6 Results 0 110 110 – – –
6 Conditions for continuity – – – 0 11 11

56 Pe aks i n s i g ht
 6.4 ⁄ KNAW committees’ judgments
of standards
The six themes of the assessment framework can only be
judged as either positive or negative. However, the underlying
standards of the assessment framework for existing
programmes can be judged as unsatisfactory (U), satisfactory
(S), good (G), or excellent (E). A theme can be judged positively,
even though one of the underlying standards is judged as
unsatisfactory. Across KNAW committees, the most common
judgment is satisfactory, given in 80% of all cases.
The judgment good is given in 15% of al cases. The judgments
unsatisfactory and excellent are the least common, given in
3% and 2% of all cases respectively. The following sections
present the KNAW committees’ judgments of the 21 standards
underlying the six themes of the assessment framework for
existing RMAs.
6.4.1 Aims and objectives
Two of the three standards underlying Theme 1 (Aims
and objectives) have a relatively high weight on the final
conclusion. Table 6.3 shows that Standard 1.1 (Subject-/
Discipline specific requirements) was judged unsatisfactory
for 5 applications. Looking more closely at the KNAW
committees’ judgments, we found that in these cases,
the final conclusion was also negative. On the other hand,
when Standard 1.1 or Standard 1.2 (Research master’s level)
was judged as either good or excellent, the final conclusion
was positive.
In the assessment reports, the KNAW committees
mentioned different reasons to judge these standards
as good or excellent, such as:
> a programme with a strong international orientation,
which fits in with recent international developments in
the particular field of studies;
> an emphasis on ‘learning to do research by doing research’,
with a thorough training in advanced research methods,
statistical modelling, and philosophy of science; and
> an integrated multidisciplinary and/or interdisciplinary
approach, which crosses the borders between scientific
disciplines and sub-disciplines;
In contrast, some of the KNAW committees’ reasons to
judge these standards as unsatisfactory mentioned in
the assessment reports were:
> intended learning outcomes which do not sufficiently fit
in with international standards for research oriented
master’s programmes within the discipline concerned;
> learning objectives which are not sufficiently specified,
or which are specified only for separate tracks and not
for the programme as a whole; and
> a programme which sets its learning objectives
unrealistically high, aiming to cover a broad range of
domains, but lacking integration and focus.
6.4.2 Curriculum
Two of the eight standards underlying Theme 2 (Curriculum)
have a relatively high weight on the final conclusion:
Standard 2.3 (Consistency of the curriculum) and Standard
2.8 (Learning assessment). When any one of these standards
was judged unsatisfactory, the final conclusion was negative.
On the other hand, when these standards were judged as
either good or excellent, the final conclusion was positive.

Table 6.3 ⁄ KNAW committees’ judgments of the aims and objectives.

Standard U S G E

1.1 Subject-/discipline specific requirements 5 5% 87 79% 16 15% 2 2%

1.2 Research master’s level 0 0% 95 86% 15 14% 0 0%
1.3 Academic orientation 0 0% 95 86% 14 13% 1 1%

Total 5 2% 277 84% 45 14% 3 1%

NVAO Research M aster Review 20 1 1 57

 Table 6.4 displays the large variation in the KNAW committees’
judgments of the standards underlying Theme 2 (Curriculum).
In the assessment reports, the KNAW committees
mentioned different reasons to judge these standards
as good or excellent, such as:
> a curriculum which largely corresponds with that of similar
programmes offered by internationally recognised top-
performing universities;
> a curriculum which covers a broad range of topics, while
maintaining an adequate amount of focus;
> an active recruitment policy, for example, by inviting the
10% top-performing bachelor’s students to apply for
admission to the RMA, and by tapping the international
network of teaching staff; and
> offering a compulsory course in quantitative methods,
before the start of the first semester.
In contrast, some of the KNAW committees’ reasons to
judge these standards as unsatisfactory mentioned in
the assessment reports were:
> a curriculum which is insufficiently elaborated to be
adequately assessed, for example, lacking a comparison
with similar programmes offered by foreign universities;
> a curriculum which insufficiently reflects the current
research by the teaching staff;
> a curriculum which is insufficiently distinct from that of a
regular MA programme, for example, when RMA students
follow too many courses together with regular MA students,
without having any additional requirements for course
completion;
> a curriculum which is fragmented, lacking focus and
consistency;
> a didactical concept emphasising students’ active
participation in small groups, which can not work in
practice, because of the student intake being too small;
> admission requirements which are too lenient, for
example, a bachelor’s grade point average of below 7
(on a 10-pont scale), or admitting students who have
too many deficiencies in prior knowledge to successfully
participate in the programme; and
> shortcomings in the assessment of master’s theses,
for example, different guidelines and assessment
criteria for different tracks within the same programme,
or a lack of variability in grades, which are sometimes
disproportionately high.
6.4.3 Staff
Two of the three standards underlying Theme 3 (Staff) have a
relatively high weight on the final conclusion. In 5 out of the
6 applications for which Standard 3.1 (Staff requirements for
academic orientation) was judged unsatisfactory, the final
conclusion was also negative. On the other hand, a good or
excellent judgment of Standard 3.1 or Standard 3.3 (Quality of
staff) often coincided with a positive final conclusion. Table
6.5 shows that for a relatively large proportion of applications,
these standards were judged as good or excellent.

Table 6.4 ⁄ KNAW committees’ judgments of the curriculum.

Standard U S G E

2.1 Requirements for academic orientation 4 4% 76 69% 22 20% 8 7%

2.2 Correspondence between aims and objectives and the curriculum 8 7% 84 76% 12 11% 6 5%
2.3 Consistency of the curriculum 8 7% 83 75% 13 12% 6 5%
2.4 Workload 0 0% 101 92% 9 8% 0 0%
2.5 Admission requirements 4 4% 95 86% 11 10% 0 0%
2.6 Credits 0 0% 109 99% 1 1% 0 0%
2.7 Coherence of structure and contents 6 5% 83 75% 19 17% 2 2%
2.8 Learning assessment 2 2% 100 91% 8 7% 0 0%

Total 32 4% 731 83% 95 11% 22 3%

58 Pe aks i n s i g ht
 Standards regarding staff were judged to be good or excellent
when the programme was considered to be embedded in an
academic context of outstanding quality. That is, when the
programme’s teaching staff is composed of researchers who
are internationally recognised leaders in their field. In a few
cases the quality of the teaching staff was judged to be
unsatisfactory, for example, because the programme did not
by default deploy the best teachers and researchers available
to the institution.
6.4.4 Facilities
The standards underlying Theme 4 (Facilities) have a relatively
low weight on the final conclusion. Table 6.6 shows that for
a relatively large proportion of applications, Standard 4.2
(Tutoring) was judged as good or excellent. Looking more
closely at the committees’ judgments, we found that in most
of these cases the final conclusion was also positive.
In contrast, some of the KNAW committees’ reasons to
judge these standards as unsatisfactory mentioned in
the assessment reports were:
> planned cutbacks in the budget of the university library,
leading to a loss of high-level expertise in the library
staff; and
> insufficient capacity of staff to provide students with the
desired amount of tutoring.
6.4.5 Internal quality assurance system
As displayed in Table 6.7, only one of the standards
underlying Theme 5 (Internal quality assurance system)
was judged as unsatisfactory, and for one application only.
Standard 5.2 (Measures for improvement) is judged as
good for almost a quarter of all applications. However, no
systematic co-variation could be observed between the
judgment of these standards and the final conclusion.

In the assessment reports, the KNAW committees mentioned
different reasons to judge these standards as good or
excellent, such as:
> an advanced research infrastructure, including modern
laboratory and library facilities;
> a well-designed tutoring system, assigning personal
mentors to each student; and
> providing international students with ample assistance for
requesting a student visa, opening a Dutch bank account,
and finding accommodation.
In the assessment reports, the KNAW committees mentioned
different reasons to judge these standards as good or
excellent, such as:
> clear and ambitious quality objectives, and a sustained
effort toward improvement;
> a highly systematic approach which takes both the separate
courses as well as the whole curriculum as objects of
evaluation;
> quality assessments using concrete measures and clear
criteria;

Table 6.5 ⁄ KNAW committees’ judgments of the staff.

Standard U S G E

3.1 Requirements for academic orientation 6 5% 70 64% 29 26% 5 5%

3.2 Quantity of staff 0 0% 103 94% 7 6% 0 0%
3.3 Quality of staff 0 0% 83 75% 20 18% 7 6%

Total 6 2% 256 78% 56 17% 12 4%

Table 6.6 ⁄ KNAW committees’ judgments of the facilities.

Standard U S G E

4.1 Facilities 2 2% 93 85% 14 13% 1 1%

4.2 Tutoring 0 0% 81 74% 28 25% 1 1%

Total 2 1% 174 79% 42 19% 2 1%

NVAO Research M aster Review 20 1 1 59

> follow-up evaluations of the extent to which previous
measures for improvement have indeed had a positive
effect on achieved learning outcomes or study progress;
> active involvement of staff, students, alumni, and the
professional field; and
> quick and adequate communication concerning the
outcomes of evaluations and planned measures for
improvement to staff, students and other stakeholders.
6.4.6 Results
The KNAW committees’ judgments of the two standards
underlying Theme 6 (Results) are remarkable. As displayed
in Table 6.8, Standard 6.1 (Achieved learning outcomes) is
the standard most often judged as good to excellent of all
21 standards. In contrast, Standard 6.2 (Study progress) is
the standard most often judged as unsatisfactory. However,
in not a single case this has resulted in a negative judgment
of the theme Results. Moreover, no systematic co-variation
could be observed between the judgment of these standards
and the final conclusion.
In the assessment reports, the KNAW committees mentioned
different reasons to judge these standards as good or
excellent, such as:
> low attrition rates, for example, a drop out of 11% of all
incoming students;
> high graduation rates, for example, 87% of all incoming
students successfully completing the programme within
the nominal duration of two years; and
> students producing theses of a very high quality, which
often result in articles worthy of publication, or have actually
been accepted for publication in internationally renowned
peer reviewed journals.
In contrast, some of the committees’ reasons to judge these
standards as unsatisfactory mentioned in the assessment
reports were:
> objectives regarding study progress which are less
ambitious than may be expected, for example, a target
graduation rate of 50% within two years; and
> high attrition rates and low completion rates compared
to the intensive, virtually personal guidance offered to
students, for example, three students dropping out from
a total student intake of eight in two years.

Table 6.7 ⁄ KNAW committees’ judgments of the internal quality assurance system.

Standard U S G E

5.1 Periodical evaluations 0 0% 96 87% 10 9% 4 4%

5.2 Measures for improvement 1 1% 84 76% 25 23% 0 0%
5.3 Involvement of staff, students, alumni and the professional field 0 0% 101 92% 9 8% 0 0%

Total 1 0% 281 85% 44 13% 4 1%

Table 6.8 ⁄ KNAW committees’ judgments of the results.

Standard U S G E

6.1 Achieved learning outcomes 0 0% 72 65% 32 29% 6 5%

6.2 Study progress 14 13% 68 62% 23 21% 5 5%

Total 14 6% 140 64% 55 25% 11 5%

60 Pe aks i n s i g ht
6.5 ⁄ Discussion
The norms which RMA programmes have to satisfy in order to
be accredited by NVAO are defined by the assessment frame-
works. These norms clearly surpass those which have to be
met by regular MA programmes. Therefore by definition,
accredited RMAs demonstrate higher levels of quality than
regular MA programmes.
The KNAW committees who assess the RMAs appear to
maintain stern demands: for a considerable proportion of
applications, especially those for the initial accreditation
of new RMAs, the KNAW committees’ final conclusion is
negative. On the other hand, the KNAW committees appear
to be quite positive about the quality demonstrated by RMAs:
somewhere between a fifth and a third of all applications are
judged as either good or excellent on standards regarding:
> the intended learning outcomes;
> the orientation, content and structure of the curriculum;
> the quality of the teaching staff deployed in the programme;
> tutoring and information provision;
> measures for improvement based on the outcomes of
internal evaluations; and
> study progress, and the extent to which the intended
learning outcomes are achieved.
The previous Research Master Review (NVAO, 2007) indicated
that the major shortcomings of negatively assessed
applications for new RMAs lay in three areas:
> Curriculum, aims and objectives (the proposed RMA
programmes were considered to be insufficiently
elaborated, or not distinct enough from regular master’s
programmes);
> Staff deployment (the proposed RMA programmes did not
meet the requirement for quantity and/or quality of teaching
staff, or the applications could not make the required quality
and quantity of teaching staff sufficiently clear); and
> Selection of students (KNAW committees objected to the
proposed method of student selection, considered the
proposed selection procedure not described clearly enough,
or found the admission requirements too lenient or not
transparent).
The results reported in this chapter largely re-iterate, but
also elaborate on these earlier findings. A negative final
conclusion by the KNAW committees is due to shortcomings
in the curriculum, and in a few cases (also) due to short­
comings in the aims and objectives and the quality of staff.
The judgments of six of the standards underlying these three
themes appear to systematically co-vary with the final
conclusions: the final conclusion is negative when any of
these six standards is judged as unsatisfactory, yet positive
when they are judged as either good or excellent. No such
systematic co-variation is observed between the final
conclusion and the judgments of the standards underlying the
other themes. A negative final conclusion about applications
for existing programmes was never due to shortcomings in the
facilities, internal quality assurance system, or results, even
though Standard 6.2 (Study progress) is the standard most
often judged as unsatisfactory. Apparently, the extent to which
the intended learning outcomes are achieved weighs heavier
on the KNAW committees’ judgments, than high attrition rates
and low graduation rates.
The large variability in KNAW committees’ judgments makes
it difficult to identify common strengths and weaknesses
which are shared by all RMA programmes. Instead, we can
provide a sketch of the stronger programmes among RMAs.
Stronger RMA programmes:
> have a coherent curriculum which is strongly orientated on
research, and which can compete with similar master’s
programmes at internationally rewcognized top universities;
> have an effective recruitment policy, and offer preparatory
courses which are intended to help students overcome
deficiencies in prior knowledge;
> are embedded in an academic context of excellent quality,
which is composed of researchers who are internationally
recognised as leaders in their field;
> have an outstanding research infrastructure;
> have a well-designed tutoring system, providing
personalised student guidance;
> have clear and ambitious quality objectives, a sustained
effort toward improvement, and a highly systematic
approach of internal quality assurance;
> have an attrition rate of about 10% and a graduation rate of
about 90%, with the large majority of students finishing
within the programme’s nominal duration of two years; and
> have a respectable proportion of students producing
excellent theses, which can be deemed of value within the
scientific discipline concerned.
NVAO Research M aster Review 20 1 1 61
Interview

62 Pe aks i n s i g ht
Bojou Neecke
Health Sciences
Bojou Neecke has almost finished the research master’s
programme Health Sciences, which is offered by NIHES (the
Netherlands Institute for Health Sciences) at the Erasmus
Medical Center in Rotterdam: “It is a master’s programme in
clinical epidemiology. My specialisation is focused on head
and neck tumours in patients aged 65 and older. I study the
prevalence of these tumours in combination with other
diseases which are common among people in this age group,
such as diabetes, cardiovascular diseases, or hypertension.
This gives me a good picture of the general health condition
of these patients. I investigate the extent to which other
diseases influence the chances of survival in the cancer
patients. We also look at other known predictors, such as
the sex and age of the patients, and the location and the
development stage of the tumour. For example, has the
tumour already spread to the lymph nodes? Based on all
the information that I collect, I can make a good prediction
about the chances for survival of the patients.
In the second year of my medicine studies, I was given the
opportunity to enrol on the research master’s programme.
That was an opportunity I certainly did not want to pass.
Previously, I studied Biology and did an internship at the
department for otorhinolaryngology at Leiden University
Medical Center. In my research master’s programme,
I elaborated on the research which I worked on as a
volunteer in Leiden. The programme taught me how to
adequately design research: How do I formulate research
questions? How do I analyse the data that I collect? How do
I draw the right conclusions?
My research in Health Sciences involves human subjects,
but I did a lot of research on animal subjects during my
biology studies. Because of the low variability among
laboratory animals (genetically, they are virtually identical)
it is comparatively easy to draw reliable conclusions.
This made me aware of the importance of a good research
design, when studying human subjects. That awareness
also caused me to look more critically at other studies,
for example at studies researching the arguments for and
against chemotherapy. At the start of our first course,
Professor Albert Hofman of NIHES told us: ‘I will take you
to the other side of the bridge’. At the time, I didn’t know what
he meant by that, but now I do. My perspective on research
has changed radically.
The research master’s programme has exceeded my
expectations, primarily regarding the extent to which it has
increased my awareness. It also provided the opportunity to
do additional courses and elective subjects. Such as the
Summer Programme of three weeks, that offered courses of
a week, half a week, or a day. Often, famous experts were
invited to give lectures, for example, professors from Harvard
University. That is what I liked the most, because it makes you
aware of the high quality of education that you get.
Within two months, I will finish my research master’s
programme. I am currently working on an article about my
research. Afterwards, I will continue my studies in a PhD
programme. In January 2012, I will start my PhD at the
department of otorhinolaryngology at the Erasmus Medical
Center in Rotterdam. The research for my dissertation will
be an extension of my current research. It will also focus on
head and neck oncology, but with palliative patients.
There is an increasing amount of attention for this group
of patients. Sometimes they live for quite some time,
although they are aware that they will eventually die from
the tumour. The research is aimed at providing better care
for these patients and a higher quality of life.”
NVAO Research M aster Review 20 1 1 63
Chapter 7 ⁄
Research
master’s:
Crossing the
borders of
science
Robbert Dijkgraaf (KNAW), Steven A. David (NVAO) and Ans Vollering (KNAW)

64 Pe aks i n s i g ht
7.1 ⁄ The fitful climate for science
The inestimable value of research master’s programmes can
more readily be appreciated by considering several global
and national trends that affect science and higher education
in the Netherlands. A rough sketch of the current climate for
science sounds like a typically Dutch weather forecast:
“Bleak winds and scattered showers, but with fair chances
of sunny spells in places, later on.”
7.1.1 The global scientific climate
Scientific research takes place in a world where growing
numbers of people are persistently looking for information on
a wide range of issues that concern them – information which
is drawn to a lessening extent from traditionally trusted
sources. Modern information and communication technology
has greatly benefited R&D, but media such as Internet forums,
weblogs and Twitter, with their abundance of information
that widely varies in its nature and weight, complicate the
scientific debate. The authoritative position of science is being
tainted by the suspicion of researchers pushing aside alternative
hypotheses, manipulating data, and restricting dissidents.
How can science regain its authority, when so many people
with so little scepticism rely on the opinion of Doctor Google?
Let’s start with trying to understand the underlying forces.
The world is becoming increasingly complex, as are the issues
in science, policy and practice. However, many a nation’s
education system lags behind the growth in our knowledge.
At the same time, science is becoming increasingly relevant,
interwoven with society, and subjected to market forces.
Policy makers are impatiently demanding concrete solutions
from science, whereas scientists are primarily “dealers in
doubt”. A famous quote by the physicist Richard Feynman
puts it in a nutshell: “Science is belief in the ignorance of
experts”. In the Google era, it seems crucial to guard that
belief in the ignorance of experts will not be replaced by a
belief in the expertise of ignorants.
Scientists should first and foremost reckon with the fast-
pace change in the environment that we find ourselves in.
What seems necessary to adapt to these global trends,
are transparency and maximum integrity, an unyielding
confidence in the power of science, strong leadership that
balances the impartiality of scientific research with a
profound engagement with the problems of society, and
adequate equipment. Furthermore, scientists need a
relentless striving towards excellence, and teach such a
focus on academic excellence in whom they tutor. The Dutch
research master’s programmes can very well inform scientists,
also those abroad, about how this tutoring is done.
7.1.2 The scientific climate in the Netherlands
Progress in knowledge and innovation is a policy priority that
enjoys broad support from social interest groups in the
Netherlands, including the labour unions and employers’ fede-
rations. All political parties in the House of Representatives
endorse the ambition to put the Netherlands back in the top 5
of most competitive economies worldwide. The importance
and primacy of knowledge and innovation are warranted by
an honest analysis of the present and future climate of the
Netherlands science and innovation system. The outcomes
of that analysis are both disquieting and encouraging.
On the positive side, the Dutch knowledge production is
characterised by outstanding levels of quantity as well as
quality (NOWT, 2010). Since the year 2000, the publication
output volume has increased 47%, and in 2008, Dutch
scientists together produced almost 30.000 research
publications. The Netherlands produces 2.8% of all research
publications within 18 benchmark countries which are either
among the most advanced countries worldwide in terms of
R&D performance, innovation and economic competitiveness,
or showing a very rapid rate of improvement, i.e. China.
In contrast: the Netherlands accounts for only 0.8% of the
total population in these benchmark countries. Also, judging
from the citation impact of Dutch research publications
within the international scientific and scholarly literature,
the science base in the Netherlands is of outstanding quality.
The Netherlands ranks fourth, closely following top
performers Switzerland, Denmark and the US (NOWT, 2010).
The Dutch scientific climate in terms of production has a
fairly good starting position for the march forwards, and in
the latest edition of the Global Competitiveness Report the
Netherlands moves up two positions to 8th place. But the
report also states that: “Dutch businesses are highly
sophisticated (ranked 5th) and are among the most
aggressive internationally in absorbing new technologies for
productivity enhancements (ranked 3rd for their technological
readiness). The country’s excellent educational system
(ranked 8th and 10th for the two related pillars) and efficient
factor markets, especially goods markets (ranked 8th), are
highly supportive of business activity. The Netherlands is also
characterized by a comparatively stable macroeconomic
environment, improving on a relative basis compared with
last year. The country’s competitiveness would be further
enhanced by introducing more flexibility into the labour
market (ranked 80th on this sub-pillar).” (Schwab, 2010, p. 23).
Therefore, it will certainly not be easy to attain the goal of
placing the Netherlands in the top 5 of most competitive
economies worldwide.
NVAO Research M aster Review 20 1 1 65
 Figure 7.1 ⁄ Total R&D intensities and 5-year total Figure 7.1 indicates how severely the Netherlands is falling
R&D growth rates behind the most advanced nations in terms of R&D
R&D intensity (%) expenditure. This begs the pertinent question how – and
4,5 whether – the Netherlands is going to realise its top 5
FOLLOWERS LEADERS
ambition. Indeed, the international competition is growing
4,0 (%) (%)
fiercer, and research is becoming increasingly complex
25 60
23 and expensive.
22 53
21
JPN FIN KOR 50
20
3,5 SWE 19 With respect to R&D-relevant human resources in the 44
18 18
16 40 Netherlands, some of the structural problems37 38 are
39 a shortage
15 34 34
15
3,0 14 14 14 of attractive job opportunities,
31 unattractive working
CHE 29 30
US 30 conditions 27and career prospects. These problems need to
25 25 25
10 AUT
10 GER DNK
be
20 thoroughly addressed. Figure 7.2 indicates how delicate
2,5 20 20
20 the Dutch 18 19 18
17 knowledge base actually is.
12
5 FRA AUS 10 9
10
2,0 EU15 7
CAN BEL
UK
0 NLD EU27 0
NOR
GER

BEL

SWE
CHE

FRA

IRE
JPN

FIN
1,5
DNK
NLD

NOR
UK
GER

BEL

AUT
SWE
CHE
FRA

IRE

FIN

DNK
NLD

NOR
UK
AUT

IRE

1,0
Business sector Universities
2000 2006

CATCHING-UP
0,5 FALLING BEHIND
0 1 2 3 4 5 6 7 8 9 10

Average annual growth rate real R&D expenditures (%)

Figure 7.2 ⁄ Share of researchers within the total labour force

Share of researchers within the total labour force (‰)

(%)
20 base year

15,0 2005
15
11,9 increase
10,6
9,3 9,8
10 9,0
7,3 7,4 7,6 7,9
6,8 7,1 7,2
5,8 5,8 6,1
4,8
5

-5
GER

BEL

SWE
CHE

FRA
IRE

JPN
KOR

FIN
AUS
CAN

DNK
US
NLD

NOR
UK

AUT

Data source: OECD MSTI database. Data treatments: UNU-MERIT.

The baseline refers to the period 1999-2001, except for Denmark and Austria (2002), and France
2000-2001). In the case of Switzerland, Australia and Canada the reference year is 2004 and
1998-2000 as a baseline; for Switzerland the baseline is 2000. Baseline data were unavailable for
Finland, Netherlands and Sweden.

66 Pe aks i n s i g ht
7.2 ⁄ The knowledge society and
future-proof higher education
For any knowledge society, an effective and efficient higher
education system is of paramount importance. Promoting the
quantity and quality of that system’s output, should be the
very foundation for human and social capital policy. So what
can be said of higher education in the Netherlands? At first
glance, the Dutch higher education system seems to be
performing quite well: the proportion of the Dutch population
that has attained tertiary education is above the OECD and EU
average (OECD, 2010).
On the other hand, the results from OECD’s Programme for
International Student Assessment (PISA) are very disturbing.
The 2006 edition (OECD, 2007) indicates that Dutch 15-year-
olds enrolled in secondary education generally perform
significantly above average on a standardised test of science
knowledge and skills, but report significantly below-average
expectations of a science-related career at age 30.
Furthermore, they report a below-average general value of
science, a below-average personal value of science, a below-
average general interest in science, a below-average
enjoyment of science, a below-average instrumental
motivation to learn science, and a below-average future-
oriented motivation to learn science.
In September 2009, the Netherlands Minister of Education,
Culture and Science sent a letter to the Chair of the House of
Representatives, to inform the members of parliament about
his plans to assign a committee that would assess the extent
to which the higher education system of the Netherlands is
future-proof, in comparison with leading higher education
systems elsewhere. The Committee on Future-Proof Higher
Education, chaired by Cees Veerman (former Minister of
Agriculture, Natural Preservation and Fishery), presented its
report Triple Differentiation in Higher Education, in April 2010.
The report was very well received by the academic community
and by those involved in higher education policy, despite its
grim conclusion that the Dutch higher education system
appears not to be future-proof. Attrition rates are too high,
talent is not being optimally challenged, and the system lacks
the flexibility to accommodate the labour market demands
and the expected growth in enrolment of students with
increasingly diverse backgrounds, without compromising
educational quality. The “Veerman Committee” advocates a
triple differentiation for the sake of quality and diversity in
higher education: in the structure of the system, in the profiles
of institutions, and in the provision of study programmes.
The wide acclaim for the Veerman Committee’s report might
well be the crucial stepping stone towards the kind of national
coherence, which the Royal Netherlands Academy of Arts
and Sciences (KNAW) has advocated for several years. The
Netherlands is just too small a country for a fierce internal
competition between institutions. To stand our ground against
the pressures from within and outside Europe, a certain
economy of scale is required, with clear and distinctive
institutional profiles and concerted efforts.
But whereas the Veerman Committee’s report identifies the
need for a unidimensional differentiation in the profiles of
institutions, we propose that profiling and co-operation can
and should be realised on multiple levels or dimensions.
In the horizontal plane, junctions between departments of
institutions that are active within a single discipline, and in
the vertical plane, junctions between departments that are
active in different disciplines within a single institution.
Together, these are the warp and weft from which the fabric
of the academic community is woven. Both kinds of thread
need to be strengthened.
The warp, the junction within disciplines, can be strengthened
through contemporary style sector planning, founded on
today’s and yesterday’s successful examples, and whole-
heartedly endorsed by the researchers involved. Therefore,
it would be highly desirable to revitalise the inter-university
research schools that in the past few years have been
gradually marginalised by the advent of the institutionally
embedded graduate schools.
In order to strengthen the junction within institutions, the
weft, higher education institutions should opt for a clear and
distinctive profile. Starting from secondary education all the
way through the three cycles of tertiary education, students
should proceed through a coherent system of ever smaller
funnels, as it were being sieved through finer-meshed strains
at each stage of transition to the next phase. Such narrowing
of focus is more easily effectuated in master’s programmes.
In contrast, bachelor’s programmes should rather be broadened
in scope, so that students can gradually set their course
without severe detriment. Furthermore, both master’s and
bachelor’s programmes should include research elements for
students, preferably set up in regional collaboration with
business and societal partners.
NVAO Research M aster Review 20 1 1 67
If higher education institutions are to provide a truly
transformative experience for their students, several
conditions should be met: removing unnecessary thresholds
between organisational entities; a clear focus on academic
excellence; safeguarding a critical mass; bachelor and master
programmes that are closely connected and integrated with
research endeavours; and active societal engagement by
students and higher education institutions.
Ultimately, the Dutch science and higher education systems
should attend to the junction on the European and global
level. In our view, the Netherlands should aspire to be a nexus
in international knowledge networks – an ambition which
seems both challenging and feasible, judging from the
outcomes of the recent study by the British Royal Society
(2011) Knowledge, networks and nations: Global scientific
collaboration in the 21st century. The report displays the
prominent position that the Netherlands take in worldwide
scientific collaboration, which seems to have intensified in
recent years.
More metaphorically speaking, scientists should also
excavate, explore, exhume, expose, exhibit, explicate, exfoliate,
expedite, extrapolate, extend and expand the current
frontiers of our knowledge. The KNAW has contributed to
the realisation of these objectives, with the publication of
The Netherlands Science Agenda (2011), that formulates
49 of the most pertinent questions in science today and for
which Dutch scientists seem exceptionally apt to provide
answers. And perhaps research master’s students sooner
or later will contribute to some of those answers.
Whatever changes to the higher education system are
suggested, the choices between different strategic options
should be well-founded on empirical evidence. As shown by
a recent parliamentary inquiry on system-wide reforms in
secondary education chaired by the social-democratic
representative Jeroen Dijsselbloem, educational innovations
can have gravely detrimental effects if they are based on not
much else than the latest fads and fashion in learning theory
or instructional design. System-wide educational reforms
should be ‘Dijsselbloem-proof’, which is to say that policy
innovations or measures should only be implemented if their
superior effectiveness and efficacy have been evidenced
through rigorous study.
68 Pe aks i n s i g ht
7.3 ⁄ The future of research master’s
programmes
Interestingly, the research master’s programmes already
answer to many of the recommendations formulated by
the Veerman Committee. Specifically, research master’s:
> are selective, establishing a harder “cut” between the first
cycle at the bachelor’s level and the second cycle at the
master’s level of higher education, in comparison with
regular “continuation” master’s programmes;
> clearly have a functional role in the differentiation in
institutional profiles;
> are an investment in the professional development of
future academic staff.
Since their conception, research master’s programmes have
taken on a pivotal role for the majority of Dutch universities in
at least three ways. First, because of their unequivocal striving
toward excellence, research master’s programmes have
given a system-wide quality impetus to tertiary educational
provision in the Netherlands. Second, because they are
embedded in and organised by institutions’ most prestigious
research units, research master’s programmes function as
“unique selling points” through which institutions can position
themselves on the international market for higher education.
Third, research master’s programmes appear to have
enhanced the contiguity between the second cycle master’s
level and the third cycle doctorate level of higher education.
However, there are also some serious concerns revolving
the research masters. The continuity of research master’s
programmes are contingent on how the Dutch universities
live up to the major intellectual challenge of finding the right
balance between focus and mass. This is to say that research
master’s programmes should maintain their programmatic
focus on their institutions’ high-profile research activities,
while ensuring the critical mass in student enrolment. It is
not so much the number of accredited research master’s
programmes, as it is the number of research master’s
students that should count as a performance indicator,
besides the (international) visibility of the research masters.
The critical mass in the student body could be realised by
ensuring that the research master is a study programme that
leads to a full-fledged and prestigious end-qualification at
master’s level, as well as a study programme that thoroughly
prepares for a PhD-trajectory. The necessary focus could be
conserved through inter-institutional fusion of or co-
operation between research master’s programmes within
similar disciplines. Intra-institutional fusion of research
master’s programmes runs the risk of watering down the
focus and diluting the coherence of the study programme,
whereas diversity in higher education and distinctive
profiling of higher education institutions are required.
Earlier, we expressed this in terms of the warp of the
fabric that constitutes the academic community, junctions
in the horizontal plane between institutional subdivisions
which are active within a single scientific discipline.

Another concern is how the introduction of the research

master has affected the PhD-trajectories. Openings for PhD
positions often state a preference for research master’s
students. As yet, it is unclear whether students from regular
MA programmes are still considered eligible for entry in PhD
programmes. Next to the introduction of research master’s
programmes, with a year extra in duration compared to
regular MA programmes, the Netherlands higher education
system has witnessed an increase in the proportion of PhD
programmes with a duration of three years instead of four.
The Netherlands should carefully observe that this time
reduction has no negative impact on the quality of these
trajectories that lead to third cycle qualifications.

Finally, another concern is that the double layered motivation

has to be clear from the beginning until the end of the
research master’s programme. The programme is meant to
prepare students properly for a PhD-trajectory as well as
for the labour market outside the university setting.

7.4 ⁄ In closing
Since the very start in 2003, KNAW committees have advised
NVAO about the (initial) accreditation of research master’s
programmes. The accreditation lends research master’s
programmes an unambiguous certificate of quality. In our
view, the research master is an outstanding innovation,
developing talent and stimulating excellence in science
and higher education. These programmes are no less than
exemplary in the extent to which they hold high expectations
of both students and teachers, integrate research in the
curriculum, and deliver what they promise. The KNAW can
only hope to continue the role that it currently plays in the
accreditation of these extraordinary master’s programmes.

NVAO Research M aster Review 20 1 1 69

Interview

70 Pe aks i n s i g ht
Thijs Porck
History: Society and
Institutions
In 2010, Thijs Porck successfully completed the research
master’s programme History: Society and Institutions at
Leiden University: “My cohort consisted of about 12 students.
We had two general courses: Historical Methodology and
Historical Conventions. The focus was on historiography and
the philosophy of science: how does science work, and how
do you apply that to the study of history? The programme
offers five specialization tracks, such as the Middle Ages,
modern history, and social history. The courses within
these tracks were attended by students from the research
master’s programme as well as students from the regular
master’s programme. I was one of the students in the
Middle Ages track.
I chose to enrol on the research master’s programme for
two reasons. First, I was invited to apply for admission,
because I had high grades during my bachelor’s studies.
Second, I aspired to enrol on a PhD programme and continue
doing research. The research master’s programme seemed
to be the obvious path. Another advantage is the degree of
freedom. Whereas the regular master’s programme has only
10 ECs of elective subjects, the research master’s programme
has 20 ECs. Also, the research master’s programme gives
more weight to the master’s thesis: 40 instead of 20 ECs.
My expectation was that al my peers would enrol on a PhD
programme, but eventually only four did. In that respect, the
programme offered a good preparation, but did not lead to
the result which we had hoped for. The main reason for this is
the limited number of openings for PhD positions. The content
of the curriculum was good. Courses which were exclusively
for students in the research master’s programme had a high
level of quality. The students in these courses are highly
motivated and come well-prepared, which leads to better
group discussions. Students have a large influence on the
level of difficulty of their programme, because the content
of half of the curriculum (thesis and electives) depends on
their own choices.
Currently, I am a PhD student and teacher at the department
of English Language and Culture. One half of my time I spend
on doing research, the other half I spend on teaching. I have an
appointment for five years, which means I have a net amount
of 2,5 years time to dedicate to my dissertation. My research
focuses on the concept of old age in early medieval England,
the period from 700 to 1350 AD. I study how old age was
conceived, which conceptions are passed down through texts.
I include literary texts as well as functional texts, such as legal
texts, sermons and encyclopaedias.
The research master’s programme gave me a good foundation
for analytical thinking. What is a good research question, how
do you handle sources, how do you make a good selection?
The programme taught me a lot about research methodology.”
NVAO Research M aster Review 20 1 1 71
Chapter 8 ⁄
Synthesis

72 Pe aks i n s i g ht
 “The conclusion seems clear: experienced
experts possess pedagogical content
knowledge that is more flexibly and
innovatively employed in instruction;
they are more able to improvise and to
alter instruction in response to contextual
features of the classroom situation;
they understand at a deeper level the
reasons for individual student success
and failure on any given academic task;
their understanding of students is such
that they are more able to provide
developmentally appropriate learning
tasks that engage, challenge, and even
intrigue students, without boring or
overwhelming them; they are more able
to anticipate and plan for difficulties
students are likely to encounter with new
concepts; they are more able to generate
accurate hypotheses about the causes
of student success and failure; and they
bring a distinct passion to their work.”
John Hattie, Visible learning, p. 261.
8.1 ⁄ What did we find?
Positive appraisal from experts
The series of studies conducted within the framework of RMR
2011 indicate that the quality of RMA programmes is high.
NVAO’s assessment frameworks for RMA programmes
establish higher norms for RMAs than for regular MA
programmes. These norms refer to the aims and objectives,
the content and structure of the curriculum, the admission of
students, the quality of the teaching staff, and the quality of
the master’s thesis. This implies that RMA programmes have
to meet higher levels of quality than regular MA programmes
to be accredited.
The committees appointed by the Royal Netherlands
Academy of Arts and Sciences (KNAW) to advise NVAO on the
accreditation of RMAs maintain stern demands. However, they
are generally quite positive about the overall quality of RMAs,
and about the young research talent that these programmes
so impressively nurture. That research talent has also been
clearly recognised by the panels composed of international
experts, who participated in the thesis assessment study
(TAS). These panels conclude that a large proportion of the
RMA theses which they assessed are of top international level.
Positive appraisal from students and alumni
The high quality of RMA programmes is not only recognised
by experts. Students and alumni of RMAs report significantly
higher satisfaction with their study programme, than students
and alumni of regular MA programmes. RMA students and
alumni are especially more satisfied with the academic skills
acquired in their studies and with research related aspects
of their programme. Compared to alumni from regular MA
programmes, RMA alumni more frequently have a research
oriented job. A large proportion of RMA alumni enrol on a PhD
programme.
A range of peaks in sight
Drawing from a multitude of sources, we feel confident to
conclude that true academic excellence is being realised
in RMA programmes. Carrying the habitual geographical
metaphor a bit further, NVAO believes that the Dutch higher
education system can surely be characterised as a plateau.
A plateau, however, that has a much more distinctive relief
than our current charts display. RMR 2011 certainly puts
the RMA more clearly on the map as a whole range of peaks.
However, NVAO is convinced that if further steps would be
taken to systematically explore the academic landscape of
the Netherlands, more peaks will come in to sight in other
domains.
Favourable results, but low student enrolment
On average, RMA programmes have low attrition rates and
high graduation rates. The vast majority of RMA graduates
seem to finish within the nominal programme duration, or
with little delay in degree completion. However, the low
student enrolment remains a challenge for many RMAs.
The annual student intake is often (far) below the programmes’
capacity.
8.2 ⁄ What are the implications?
Educational policy
The Netherlands encounters fierce contrary winds from both
East and West, from well-established competitors as well as
some rapidly developing nations. Therefore, the Netherlands
should work hard to keep its position in the world ranking of
innovative economies.
NVAO Research M aster Review 20 1 1 73
The RMA programmes provide an answer to the Veerman
Committee’s call for more selection, sharper profiling,
investments in research and higher qualified personnel, and
more attention to the provision of education as a core task
of universities. The RMA appears to be a trustworthy brother
in arms for making the Dutch higher education system more
future-proof. The Dutch government and policy makers should
cherish the RMA as a valuable educational intervention,
display the RMA as an illuminating example for future
system reforms, and make an effort to assure the continuity,
accessibility and quality of RMA programmes.
RMAs and the Veerman Commitee’s
recommendations
Selection (√)
Stimulate institutional profiling and reward
accomplishments (√)
A smaller share of student-bound funding (√)
Investment in research (√)
Effectuate the Associate degree (X)
Design a new arrangement for master’s programmes (√)
Introduction of an unambiguous system of titles (X)
Institutions: Choose a profile (√)
Institutions: Give more attention to education as a
core task (√)
Institutions: Invest in higher qualified personnel (√)
Enhancing continuity and quality through increasing student
enrolment
Increasing the student intake seems necessary for securing
the viability and continuity of RMA programmes. Furthermore,
many RMAs should aim to increase enrolment, in order to
provide their students with the opportunity to reap the
benefits of collaborative learning.
It seems fair to conclude that educational institutions in the
Netherlands could do more and better, to infuse more positive
attitudes towards science and research in their students.
After all, the stability and progress of a knowledge society
depends on the extent to which teachers manage to have
students not only acquire the necessary scientific knowledge
and skills, but also to internalise the value and importance
of science. Challenging research seminars as part of the
bachelor’s programmes could be a good starting point to
foster a deeper and lasting interest in research and science.
74 Pe aks i n s i g ht
Another promising strategy, as shown in some of the
interviews, is scouting: approaching talented students at
an early stage and giving them a taste of what an RMA
might bring them.
In any case, Dutch universities should pursue with providing
the context in which RMA programmes can continue to realise
true academic excellence. In order to find ways to make RMA
programmes more attractive to larger groups of students, the
co-operation between universities could be strengthened.
Invigorating the inter-university research schools seems to
be a logical starting point. To recruit more foreign research
talent for the RMAs, the universities should aim for a closer
collaboration with Nuffic. However, we hasten to add that
increasing the student intake should not be realised through
lowering admission requirements.
Didactical approach
The different studies performed for RMR 2011 consistently
indicated that the RMA has proven to be a highly effective
educational innovation. This begs the question of why RMAs
are so successful. One explanation for the success of RMAs
could be that these programmes correspond to a didactical
approach that has proven its effectiveness throughout
millennia: the master/apprentice model of instruction and
academic socialisation. We have found further explanations
for the success of RMA programmes in John Hattie’s (2009)
magnum opus Visible learning: A synthesis of more than 800
meta-analyses relating to achievement. It appears that several
distinctive attributes of RMAs are among the most powerful
influences on achievement, as identified by Hattie:
> highly motivated students who have a strong learner
identity, and who are ready to put a considerable effort
into mastering their study programme’s subject matter,
out of interest, enjoyment, and their attaching a high
personal value to academic achievement;
> a programme which clearly sets challenging learning goals,
and which demands students’ active participation and
engagement;
> a captivating delivery of that content by highly qualified
and passionate teachers, many of whom are internationally
renowned experts involved in ground-breaking research in
their specific field of science;
> teachers and students forming a cohesive community of
learners.
Recommendations
Improving the quality of higher education requires a concerted
effort. We would like to conclude with some recommendations
to a number of relevant parties.

Programmes:
> Provide your students with the opportunity to reap the
benefits of collaborative learning, in a manner geared
towards learning to co-operate in (multidisciplinary/
interdisciplinary) teams.
> Ensure a degree of multidisciplinarity/interdisciplinarity in
RMA programmes that constitutes a true reflection of the
level of multidisciplinarity/interdisciplinarity in the type of
research environments where your graduates may be
employed.

Institutions:
> Pursue with providing the context in which RMA
programmes can continue to realise true academic
excellence;
> Invigorate the inter-university research schools;
> Strengthen the co-operation between universities in order to
find ways in which RMA programmes can be made attractive
to a larger group of students.

Government and policy makers:

> Cherish the RMA as a valuable educational intervention
and help making our nation’s higher education system
more future-proof;
> Display the RMA as an illuminating example for future
system reforms; and
> Take care of assuring the continuity, accessibility and
quality of RMA programmes.

> Nuffic: Feature the RMA as a unique selling point of the

Dutch higher education system, in your efforts to recruit
more foreign research talent.

> Employers: Acknowledge the added value of study

programmes that have a strong focus on the acquisition
of advanced research abilities, such as the RMAs.

> KNAW committees: Remain critical, not only about the

quality of RMA programmes, but also about the quality
and consistency of the assessments performed by each
committee.

NVAO Research M aster Review 20 1 1 75

Interview

76 Pe aks i n s i g ht
Anne-France Pinget
Linguistics
Anne-France Pinget successfully completed the research
master’s programme Linguistics at Utrecht University in
July 2011. On 1 October 2011, she started her PhD: “The
Linguistics programme consisted of several semesters.
The first two semesters were very theoretical, covering
all fields of language studies. In the third semester,
I could choose a specialisation. My major was language
acquisition, and my minor was experimental psycho­
linguistics.
During the research master’s programme, I applied for a
PhD position at the Utrecht Institute of Linguistics OTS.
On 1 October, I started my research on language variation
and change. I received a scholarship from NWO (the
Netherlands Organisation for Scientific Research) for
my project Solving the actuation problem: The role of
perception in sound change.
I have only just started my doctoral studies, which will
span four years. My research focuses on language change
throughout the years. My interest is not so much in the
development of new words, but in language change at
a phonetic level. For example, why do people in the
Netherlands increasingly use a voiceless ‘f’ instead of
a voiced ‘v’, or a retroflex approximant ‘r’ instead of a
trilled, tapped or flapped ‘r’? I aim to catch some of these
language changes ‘in the act’, trying to explain why
they happen exactly on this location, and at this moment
in time.
I am very satisfied about the research master’s programme
Linguistics at Utrecht University. I did a bachelor’s and a
one-year master’s programme in Belgium. That bachelor’s
programme was focused on applied linguistics, and I
expected the same from the research master’s programme.
However, a large part of the curriculum turned out to be
focused on theoretical linguistics, and that was interesting
too. It introduced me to a whole new side of the linguistic
sciences. So in that respect I was pleasantly surprised by
the research master’s programme.”
NVAO Research M aster Review 20 1 1 77
References

78 Pe aks i n s i g ht
Committee on Future-Proof Higher Education. (2010).
Triple differentiation in higher education. The Hague:
The Netherlands Ministry of Education, Culture and Science.
Dijkgraaf, R. (2009). Knowledge and crisis. [Kennis en crisis:
Jaarrede voor de Verenigde Vergadering van de Koninklijke
Nederlandse Akademie van Wetenschappen gehouden op
25 mei 2009]. Amsterdam: Royal Netherlands Academy of
Arts and Sciences.
Dijkgraaf, R. (2010). The climate for science. [Het klimaat voor
de wetenschap: Jaarrede van de president van de Koninklijke
Nederlandse Akademie van Wetenschappen gehouden op
10 mei 2010]. Amsterdam: Royal Netherlands Academy of
Arts and Sciences.
Dijkgraaf, R. (2011). On the borders of science. [Over de
grenzen van de wetenschap: Jaarrede van de president van
de Koninklijke Nederlandse Akademie van Wetenschappen
gehouden op 30 mei 2011]. Amsterdam: Royal Netherlands
Academy of Arts and Sciences.
Hattie, J. A. C. (2009). Visible learning: A synthesis of over 800
meta-analyses relating to achievement. London: Routledge.
KNAW. (2011). The Netherlands science agenda
[De Nederlandse wetenschapsagenda]. Amsterdam:
Royal Netherlands Academy of Arts and Sciences.
KNAW Social Sciences Council. (2007). Research master’s
programmes in the social sciences: First experiences.
[Onderzoeksmasters in de sociale wetenschappen: Eerste
ervaringen.] Amsterdam: Royal Netherlands Academy of
Arts and Sciences.
Muskens, M., Pass, J., & van den Broek, A. (2010). The Dutch
national student survey: Facilitating transparency and course
choice. Paper presented at EAIR 32nd Annual Forum in
Valencia, Spain.
NOWT [The Netherlands Observatory of Science and
Technology]. (2008). Science and technology indicators
2008. The Hague: The Netherlands Ministry of Education,
Culture and Science.
NOWT [The Netherlands Observatory of Science and
Technology]. (2010). Science and technology indicators
2010. The Hague: The Netherlands Ministry of Education,
Culture and Science.
Richters, E., Roodenburg, S., & Becker, R. (2011).
Mapping mobility 2011: International mobility in Dutch
higher education. The Hague: Nuffic.
NVAO. (2007). Research master review 2007: The exploration
of a new domain. The Hague: Accreditation Organisation of
the Netherlands and Flanders.
OECD. (2007). PISA 2006: Science competencies for tomorrow’s
world. Paris: Organisation for Economic Co-operation and
Development.
OECD. (2008). Education at a glance 2008. Paris: Organisation
for Economic Co-operation and Development.
OECD. (2010). Education at a glance 2010. Paris: Organisation
for Economic Co-operation and Development.
The Royal Society. (2011). Knowledge, networks and nations:
Global scientific collaboration in the 21st century. London:
The Royal Society.
Schwab, K. (Ed.) (2010). The global competitiveness report
2010-2011. Geneva: World Economic Forum.
NVAO Research M aster Review 20 1 1 79
RMR 2011 ⁄
A collective
effort

80 Pe aks i n s i g ht
Miguel Castelo-Branco Lineke van Bruggen
Wolfgang Donsbach Steven David
Matthew Fuller Karl Dittrich
Gordon Gemmill Astrid Koster
Jostein Gripsrud Guido Langouche
David Linden Jan Willem Meijer
Lars Muckli Fred Mulder
Keith Roe Research team
Piet Sercu
Astrid Söderbergh Widding Lieselotte De Zutter
Ralf Terlutter IJda van den Hout
Hannelore Weck-Hannemann Conference organisation
TAS assessment panel members
Sandra Louwersheimer
Charles Forceville Nathalie van den Dobbelsteen
Media Studies, UvA Franka Paijers
Mahdy Kabbani
Peter Neijens Support
Communication Science, UvA
Erik van der Spek
Frans de Roon Interviews & text editing
CentER, UvT

Alexander Sack
Cognitive and Clinical Neuroscience, UM

Martin Nieuwenhuizen
Petra Pieck
Institutional research VSNU

NVAO Research M aster Review 20 1 1 81

Colophon
Peaks in sight - Research Master Review 2011
© March 2012

Editors: Steven A. David and Erik van der Spek

Lay-out: Smidswater, The Hague
Offset: Albani Drukkers

NVAO
Accreditation Organisation of the Netherlands and Flanders

Parkstraat 28 / 2514 JK Den Haag

P.O. Box 85498 / 2508 CD The Hague
The Netherlands

T +31 (0)70 312 23 00

F +31 (0)70 312 23 01
info@nvao.net
www.nvao.net
 Research Master Review 2011

Peaks in sight

Peaks in sight - Research Master Review 2011 - NVAO

NVAO
Accreditation Organisation of the Netherlands and Flanders

Parkstraat 28 / 2514 JK Den Haag

P.O. Box 85498 / 2508 CD The Hague
The Netherlands

T +31 (0)70 312 23 00

F +31 (0)70 312 23 01
info@nvao.net
www.nvao.net