Summary Record of meeting to discuss OECD support for National PISA 2018 Data Analysis

And Reporting

27 September 2019 Paris, France

With the support of the Open Society Foundation (OSF) and through its PISA for Development
(PISA-D) initiative, the OECD is supporting seven countries (Albania, Bosnia-Herzegovina,
Indonesia, Moldova, Panama, Serbia and Ukraine) in analyzing their PISA 2018 data and, in the case
of Panama, its PISA-D data, interpreting the results and producing national reports. These reports will
be launched concurrent with the launch of OECD’s international report on PISA 2018 on 3 rd
December 2019. With the national reports in the final stages of preparation, the OECD held a meeting
in Paris on 27 September 2019 that brought together the Lead Analysts, OECD Secretariat,
representatives of OSF and representatives of the concerned Ministries of Education for the purpose
of reviewing the key findings of the national reports and discussing communication strategies for
disseminating the results of PISA 2018 (see Annex 1 of this Summary Record for details of the
participants).

The main aims of the meeting were to:

 provide the opportunity for the OECD and the Lead Analysts to share their experiences of
the programme with participants in the meeting;

 present to the participants in the meeting and to receive feedback on: (a) the key results and
messages from the national reports; and (b) the suggested communication strategies,
including the launches on 3rd December 2019; and

 enable the participants in the meeting to discuss and reach agreement on timelines to support
the production and launching of the national reports on 3rd December 2019 and subsequent
dissemination of the results, policy dialogues and use of the PISA 2018 data in the countries.

SUMMARY RECORD

1. Michael Ward, Senior Policy Analyst in the OECD Directorate for Education and Skills and
Development Co-operation Directorate and manager of the Lead Analysts Programme, welcomed
the delegates and opened the meeting. Michael facilitated a round of introductions, presented the
agenda for the meeting and confirmed the main aims of the meeting as noted above.

2. Andreas Schleicher, OECD Director for Education and Skills made some opening remarks. After
congratulating the Lead Analysts on their achievements, Andreas emphasised that participating in
PISA is a costly exercise and the OECD considered that it should do more to help countries to
analyse their data, interpret the results and produce national reports to inform policy dialogue and
support evidence-based policy-making. Andreas also appreciated that the national reports and the
launches of these are very important, but he recognised that to have real impact it will be
necessary to sustain the dialogue about the results and to inform policy dialogues and
development in the months leading up to and following the launches.

3. In her opening remarks, Katherine Lapham, Deputy Director, OSF (Open Society Foundations)
Education Support Programme, also congratulated the Lead Analysts on their achievements.
Katherine looked forward to the launches of the national reports in the participating countries and
emphasised the importance of the ‘last mile’ in respect of communicating the key findings and
policy messages and supporting national evidence-based policy dialogue and decision-making.

4. In the second session of the meeting Michael Ward presented the Lead Analysts Programme,
highlighting the following milestones:
 ● The first period of the residence at the OECD for the Lead Analysts (April) included
training on PISA data analysis and reporting; the characteristics of international
assessments and examples of research conducted using their initial PISA data files; how
to handle international studies’ databases and technical documents; how to replicate PISA
results and answer research questions using relevant software (such as Mplus, R, STATA,
SPSS or SAS); and practical experience on how to formulate, analyse, interpret and
present results relevant to educational research using PISA data.

● In the second period of the residence (May to August) the Lead Analysts have applied the
skills acquired through the training to analysis of their own PISA 2018 data and, with the
mentoring and support of the OECD’s PISA-D manager, PISA Analysts and Consultants
have interpreted their results and prepared a report on the basis of the OECD’s national
report template.

● During the third period of the residence (September), with the support of the OECD the
Lead Analysts have developed a communication strategy for the preparation of key
stakeholders and the dissemination of results.

5. Michael praised the Lead Analysts for their hard work and dedication and also thanked the PISA
teams and Ministries of Education in the countries for their support for the work as they had been
reviewing drafts of the chapters as they emerged. Michael also highlighted the inputs of the
external experts and OECD colleagues that had contributed to the programme. Michael invited the
Lead Analysts to reflect on their experiences in the programme and, in particular, to share with
the meeting what they had learned and achieved during their period at the OECD.

6. In the discussion of the Lead Analysts Programme, the Lead Analysts confirmed that their
greatest achievement was the production of the national reports. They each appreciated the value
of the template OECD had provided for the national report and the mentoring, training and
support that had been provided. Ministry and OSF representatives expressed satisfaction with the
programme and Michael explained that each Lead Analyst has been asked to complete a detailed
evaluation questionnaire so that lessons from the Programme can be fed into the design of future
activities. Michael confirmed that five countries had already signed up for the PISA 2021 Lead
Analysts Programme that is scheduled to run from April to September 2022.

7. In the third session, the Lead Analysts presented the key findings and main policy messages from
their national reports. These presentations were well received by the participants in the meeting
and the Lead Analysts were praised for the quality of their work. The participants in the meeting
considered that the key findings and policy messages would be valuable contributions to national
policy dialogue and evidence-based decision-making in the countries.

8. In the fourth session, Michael Ward and Julia Himstedt presented the template prepared by the
OECD for a national PISA 2018 national report communication plan (attached as Annex 2 of this
Summary Record). Michael and Julia showed how this template includes the context for the
national reports, a list of key stakeholders, key messages, suggested associated communication
activities, including the preparation of products such as press releases and social media and key
events, the timeline to 3rd December 2019, including national report production and
communication milestones leading up to the launches in each country of the national reports. The
participants discussed the sections of the template, especially the challenging political contexts for
the national reports in some countries and the key stakeholders that will need to be addressed with
specially tailored messages. The country representatives AGREED to use the OECD template for
their communication plans and to confirm to the OECD the details of their focal points for the
purpose of communication and report production. Michael CONFIRMED that Julia would be
available right through to 3rd December to help the countries finalise and implement their
communication plans.
9. The participants discussed how the reports would be produced in the countries and the steps
involved in report clearance. Michael emphasised the deadline of 3rd November for the OECD to
receive the final versions of the reports for checking prior to publication. The representatives of
the countries CONFIRMED that:

● They would like to include the OECD logo on the covers of their reports.

● The deadlines for finalising the reports in time for publication on 3rd December 2019
seem feasible.

10. The OECD’s external expert, Tijana Prokic-Bruer, presented the Netherlands’ State of Education
launch event and suggested possible formats for the PISA 2018 national report launch events in
December. The participants discussed their ideas for the launches in the seven countries and
NOTED Ukraine’s plan to hold a three-day national education conference immediately after the
launch; and several countries’ intentions to complement their launch event with smaller meetings
of key stakeholders. Michael invited the countries to prepare their plans for the launches bearing
in mind the release date and time of the PISA 2018 international dataset and international report
Volumes I-III by the OECD at 9.00.a.m. (Paris-time) on 3rd December.

11. Michael CONFIRMED that the OECD would be represented at each country’s national report
launch on 3rd December by the following people:

Country OECD Representative

Albania Alfonso Echazarra
Bosnia-Herzegovina Tanja Bastianic
Indonesia Yuri Belfali
Moldova Michael Ward
Panama Catalina Covacevich
Serbia Tijana Prokic-Breuer
Ukraine Lucia Tramonte

12. In the fifth session, Michael facilitated a discussion of next steps, including, the timeline to 3rd
December 2019, the launch events and beyond. The AGREED next steps and timeline are
reflected in the communication plan template included at Annex 2 of this Summary Record.

13. Michael then presented the conclusions from the meeting as reported in this document and
participants AGREED that the aims of the meeting had been achieved.

14. On behalf of OSF, Daniel Pop, Senior Education Advisor, OSF Education Support Programme,
made some closing remarks. Daniel thanked the OECD for the very efficient management of the
Lead Analysts Programme and the preparation for and administration of the meeting. Daniel
congratulated the Lead Analysts on the quality of their work and their dedication to the task of
preparing the national reports. Daniel looked forward to the launches on 3rd December and
emphasised that following the launches it would be necessary to sustain the evidence-based policy
dialogues and decision-making in each country to extract the maximum yield from the PISA data.

15. Yuri Belfali, Head of the OECD’s Early Childhood and Schools Division closed the meeting with
some final remarks. Yuri thanked the lead analysts for all the work they have done over the past
six months and their excellent presentations in the morning. Yuri also thanked the country
representatives for their participation in and contributions to the meeting. Yuri also appreciated
the OSF colleagues for their support for the programme and their contributions to the meeting,
acknowledging that without this support the Programme would not have been possible. Yuri
thanked her PISA colleagues and external experts for their support for the programme.
16. In closing the meeting, Yuri looked forward to helping the countries to finalise and produce their
national reports; to supporting the countries in their communication efforts; and, with OSF
colleagues, standing together with the Ministers of Education and their officials in each of the
seven countries on 3rd December 2019 to launch the national reports.

17. A photograph of the participants was taken:

ANNEX 1: PARTICIPANTS IN THE MEETING

Lead Analysts

Erind Çela, Data Manager PISA, Ministry of Education, Albania

Žaneta Dzumhur, PISA NPM and TIMSS and PIRLS NRC, Agency for Pre-School, Primary and
Secondary Education, Bosnia and Herzegovina

Januar Pribadi, Centre for Educational Assessment, Indonesia

Stanislav Ciubotaru, National Agency for Curriculum and Evaluation, Ministry of Education, Culture
and Research, Moldova

Tania Johnson, Ministry of Education, Panama

Marina Videnovic, Researcher, Institute of Psychology, Faculty of Philosophy at the University of

Belgrade, Serbia

Mariia Mazorchuk, Professor of Information Technology, Statistics Methods, Psychometrics and Data
Analysis, National Aerospace University, Ukraine

Country PGB representatives for meetings on 26th and 27th September 2019

Rezana Vrapi
Deputy Director
Department of Programmes for Development of Education, Sports and Youth
Ministry of Education, Sports and Youth
23 Rruga e Durrësit
AL-1001 Tirana
Albania

Alisa Ibrakovic
Agency for Pre-School, Primary and Secondary Education
Kneza Domagoja bb
88000 Mostar
Bosnia and Herzegovina

Totok Suprayitno
Head of Agency for Research and Development
Ministry of Education and Culture
Building E, 2nd Floor
Jalan Jenderal Sudirman
Senayan
Indonesia

Rahma Wati
Ministry of Education and Culture
Building E, 2nd Floor
Jalan Jenderal Sudirman
Senayan
Indonesia

Anatolie Topala
Director of National Agency for Curriculum and Evaluation
Ministry of Education
Piata Marii Adunari Nationale 1
MD-2033 Chisinau
Republic of Moldova

Nadia de Leon
Ministry of Education
PO Box: 0816-04049
Villa Cardenas, Ancon
Panama City
Panama

Anamarija Viček
State Secretary
Ministry of Education, Science and Technological Development
Nemanjina 22-26
11000 Belgrade
Serbia

Tetiana Vakulenko
Methodologist Research and Analytic Department
Ukrainian Center for Education Quality Assessment (UCEQA)
5 V. Vynnychenko street
04503 Kyiv Ukraine

OSF Representatives

Katherine Lapham, Deputy Director, OSF Education Support Programme

Daniel Pop, Senior Adviser, OSF Education Support Programme

Nóra Henter, Program Admin Officer, OSF Education Support Program

Dritan Nelaj, OSF, Albania

Dzenana Trbic, OSF, Bosnia and Herzegovina

Ana Coretchi, OSF, Moldova

Tatjana Stojic, OSF, Serbia

Hryhoriy Baran, OSF, Ukraine

OECD Representatives

Andreas Schleicher, Director for Education and Skills

Yuri Belfali, Head of Early Childhood and schools Division, Directorate for Education and Skills

Michael Ward, Senior Policy Analyst, Directorate for Education and Skills

Filippo Besa, Consultant, Directorate for Education and Skills

Julia Himstedt, Consultant, Directorate for Education and Skills

Tijana Prokic-Bruer, External Expert, Directorate for Education and Skills

 ANNEX 2: COMMUNICATION PLAN TEMPLATE

PISA 2018
National Communications Plan
[your country]
POLITICAL ENVIRONMENT

Drawing on Chapter 1 of your national report, please use this section to summarise the political
landscape in the area of school policy and education in your country. What are the main challenges
as identified by the government and the public? What are the current policy and reform priorities of
the government? What are the main education issues debated in public.

If possible, please describe when and under which circumstances the government decided to
participate in PISA. Is PISA 2018 the first PISA round for your country? What insights would the
government like to get from the results? Is your country participating in other international student
assessments?

Is your country using student assessments on a national level? Is data and statistics used to shape
education policy? If yes, what kind of data is used? Is there public/political debate around data in
education (student performance, spending on education, staffing of schools…)?

KEY MESSAGES FROM PISA 2018

Drawing on the Executive Summary of your national report, please list here six to nine main findings
from PISA2018 for your country plus three policy recommendations/implications. Look at overall
performance but also at performance in relation to peer countries. Include results from subsections
of the PISA domains and difference between student groups (boys vs girls, advantaged vs
disadvantaged students, migrants vs non-migrants, etc.). Include relative strength and challenges in
your list.

Describe how these findings relate to the government strategy/policies in the area of education. Are
they supporting the current policy priorities or are they suggesting a different focus?

Based on the findings and the current government policy, develop a set of messages you think the
government should/could put forward when communicating on the PISA2018 results.
AUDIENCES / STAKEHOLDERS

In this section, you should provide a brief description for the main audiences and stakeholders for
the PISA2018 results and your report and outline a strategy how to get them engaged in discussing
the findings. These stakeholders may include the Minister of Education and junior ministers of
education, senior officials of the Ministry of Education, other political leaders, ministers and senior
officials of other ministries, sub-national governments, school principals and principals associations,
teachers and teacher associations and unions, civil society, media, education researchers and
academics more generally, students, parents, the wider population.

Traditional media

Please describe how mainstream media (TV, radio, newspapers) in your country deals with
education and schools. What are the hot issues? Is data around education used and debated on a
regular basis? What would be a good strategy to engage with mainstream media about the PISA
results? Is it possible to identify a group of senior journalists, who set the tone in the public debate
around education?

Non-traditional media, blogger, influencer, social media

Are there any influential actors in non-traditional media who discuss education and school policy?
What would be a good way to engage with them?

Specialised media

Please list the main specialised media outlets for teachers and professionals in the education sector
in your country. How could you make sure that PISA findings will be discussed in these outlets (op-
ed, interviews, etc).

Education administration

Who within the education administration would be interested in the PISA results? Would the results
be useful for school inspectors? Will they be able to build on the PISA results for their day-to-day
work with schools?

Education researchers

What would be a good approach to communicate PISA results within the education research
community (journals, conferences, annual meetings)? How to encourage researcher to use PISA data
for further analysis.

Professional associations in education

Please describe the most important professional bodies in education such as teacher associations,
school principal associations or teacher unions. Describe, how these organisations and their
members might get engaged around the PISA2018 results.
Teachers, Principals, Parents and students

How can you reach out to schools, teachers and principals? Do parents and students have a voice in
education policy in your country? How could you engages with them about the PISA results? Are
there elected parent and student representatives on local or national level? Are there important
parent associations who might be interested in the results?
COMMUNICATION ACTIVITIES FOR PISA2018

Use this section to list and describe the main communication activities to promote PISA2018 results.

Pre-launch

You may want to consider the following activities before the launch of the results on December 3rd

Contact reporters for stories on the purpose of PISA

PISA is a complex project with a specific contribution to improving education. It is useful to get some of
these aspects into the public to prepare your audience for the release of the results. Reports like this
could focus on the history of PISA, its focus on competences rather than on curricula or the impact of
previous rounds of PISA in your country or in a reference country. You could try to reach out to reporters
in television or print and encourage reporting along these lines.

Pre-launch media seminar/webinar

Two to four weeks ahead of the launch you could organize a seminar on site or online to brief reporters
about the upcoming results. This could be an opportunity to communicate about the concept of PISA but
also about sound ways to communicate the results. Such a seminar could also help to discourage the use
of league tables when reporting on PISA

12
Pre-launch social media material

Ahead of the launch, you may want to post messages on social media about the upcoming results.
Material you prepare could include a quiz on what is PISA and on some of the published test items but
also a video that explains the purpose of the study. Material like this is also prepared by the OECD in Paris
which you may adapt for your country or region.

Sharing PISA results with reporters under embargo

In order to give reporters time to prepare their articles about the PISA results, it can be useful to share
with them the report and main findings under embargo. Sharing embargoed material with reporters will
be managed centrally by the OECD. You are invited to refer journalists to the OECD media department for
this service.

Launch

For the launch of the results on 3rd October you may want to prepare the following material and activities

Main news conference

This will be the event to communicate the key findings. Ideally, your minister or a high-ranking official will
present the findings to the press, together with a representative from the OECD. Your minister should
also be available for follow-up TV or radio interviews.

Media material for the launch

In addition to your national report we recommend to have the following material ready for the launch:
News release, summarizing the key findings in the style of a news report in a newspaper, a collection of
key charts, media could use to illustrate the results. The charts should be compelling but also avoid
showing league tables. It is also a good idea to make the data from the charts available in Excel Files.

Social media outreach

Tweetbank, key charts, video….

Post-launch

Activities after the main launch event should help to sustain communication around the PISA findings and
make sure that the findings are contributing to national debates that shape education policy-making. OSF
and other international actors could contribute to these events.

13
Stakeholder conference

You could organize a conference with key stakeholders in education (high ranking officials from education
ministries, researcher, teacher unions, parent associations, etc.) to discuss the PISA findings and their
implications for policy-making.

Op-eds

You could consider writing op-eds in national newspapers, providing more details about the PISA findings
or encourage other researchers to discuss the findings from their point of view.

Social media outreach

Even after the main launch event, you can promote key findings from the national report on social media.
You could also engage in other outreach and dialogue formats such as Facebook live to respond to
questions and remarks from your social media community.

14
TENTATIVE TIMETABLE

15
16
17
Sample News release

Performance growth has declined in recent years, with some signs of a downward trend - Equal
opportunities have improved, but remain a challenge

(Paris/Berlin, 6 December 2016) - Pupils in Germany achieve results slightly above the OECD average in all
areas tested by PISA. However, the gap to the PISA leaders in Asia and Europe remains large. At the same
time, the increase in performance has declined in recent years, with some signs of a downward trend.
Equal opportunities have improved, but remain a challenge. This is the conclusion of the results of the
PISA 2015 survey published today.

In the priority area of natural sciences, the performance of 15-year-olds in Germany continues to be
above the OECD average and has hardly changed compared to 2006, when the natural sciences were the
last PISA focus. The German results are comparable with those in Korea, New Zealand, Australia, Great
Britain and Switzerland. However, they do not come close to the performance levels of the PISA leaders
Singapore, Japan, Estonia, Finland or Canada.

The report shows that successful PISA countries have high expectations of all students. They concentrate
on good teaching and invest their resources primarily in pupils and schools with difficult conditions. All
this on the basis of a coherent and long-term strategy.

The study also shows that the world is no longer divided into rich, well-educated and poor, poorly
educated countries: in Vietnam, for example, the 10 percent of the most disadvantaged students achieve
results that are in line with the OECD average.

While PISA leader Singapore and Portugal, which was hit by a severe economic crisis, have been able to
continuously improve the performance of their students in the natural sciences over the past decade, the
results in Germany are at best stable, with a slight downward trend. A similar picture emerges for
mathematics. In contrast, reading comprehension performance in Germany has improved significantly in
recent years.

The development in equal opportunities is also encouraging. In the natural sciences, the correlation
between social background and educational success has weakened in recent years, but remains well
above the OECD average. At the same time, the proportion of pupils who perform very well despite
disadvantageous social and economic conditions has increased significantly.

"Greater equality of opportunity in education is not only a challenge to social justice, it is also a way of
making better use of people's potential," said OECD Secretary General Angel Gurría. "A higher level of
education prepares the ground for more growth and promotes social cohesion".

Less positive, on the other hand, is the development of pupils with a migration background. Here the
performance gap remains clear. Even taking into account the differences in social background and the
18
language spoken at home, the performance gap between migrant children (first and second generation)
and pupils without a migrant background is almost one school year.

Despite the good overall level of achievement, only comparatively few pupils in Germany can imagine
taking up a career in science and technology at a later date. The gap between girls and boys is also
greater here than in an international comparison. At the same time, young people in Germany are less
open-minded about science than in most other countries because of the way they gain knowledge.

Likewise, the gender of pupils in Germany has a greater impact on their achievements in the natural
sciences. Among the best pupils there are more boys than girls. In addition, fewer girls expect to work in
the natural sciences later than boys, concentrating primarily on careers in the health sector.

Some 540,000 15-year-old schoolchildren from 72 countries and economic regions took part in PISA 2015.
The German sample comprised around 6,500 pupils from 245 schools.

19