Monitoring and Evaluation Handbook/Toolkit

Proposal for a Joint Initiative by Media Development Implementers

In recent years, the debate on if and how media development contributes to democratic,
human and economic development has increased among donors, the academic
community and media assistance groups. Media development programmes have been
around for years with donor and public spending on media assistance increasing
significantly in the last 15 to 20 years largely to support programmes focussed on
creating a democratic media environment in transitional countries, especially in former
Soviet bloc countries.

With more funding going into media development, it is inevitable that donors and
governments ask whether the money has been well spent. Do media freedom and
journalistic independence really make a contribution to creating democratic societies and
achieving sustainable development?

At the same time the media development community, convinced that free, independent,
and pluralist media do make a difference, are looking at ways to better evaluate the
impact of their programmes.

The idea is to bring together interested media development organisations, key media
researchers and existing initiatives to prepare a handbook or toolkit on monitoring and
evaluation of media development programmes.

Building on the discussions within the Global Forum for Media Development and
building on the paper prepared by Global Partners for UNESCO on measuring impact of
media development and building on the conclusions of the Measuring Impact conference
in Bad Honnef in September and of the Measuring Press Freedom and Democracy
conference at the Annenberg School of Communication in November, it is proposed to
develop a handbook or toolkit to assist media development organisations in project
design as well as monitoring and evaluation.

This initiative would work closely and assist in the implementation of the Defining
Indicators of Media Development initiative launched by the IPDC of UNESCO.

It is the aim of the handbook/toolkit to provide advice and tips to implementers of media
development programmes. It is not the aim of the handbook to proscribe one particular
way of designing programmes or measuring impact.

The main aim of the handbook/toolkit is to provide advice on collection of data,

methodologies and tools for measuring change.

Bettina Peters, Director 1

Global Forum for Media Development, IPC, Résidence Palace, Bloc C 2/215,
155 rue de la loi, 1040 Brussels, Belgium
director@mediagfmd.org
The field of media development is complex and large covering many different sectors of
media landscapes. As the list of indicators developed by Global Partners for the
UNESCO background paper shows, without being fully complete and dealing with five
sections of desired outcomes, already some 150 key questions are listed. No one media
development organisations will be able to address all sectors of the media and journalism.
The handbook would assist in providing practical advice to implementers and provide a
suggested framework into which individual projects can be put.

Given the complexity of the field, it is not the aim of the toolkit to provide the ultimate
list of indicators or the ultimate project design. While aiming to be as comprehensive as
possible and using the conclusions of the discussions initiated by UNESCO, the
handbook would give implementers a framework and guidance.

But it is on the ground that organisations in a given country would decide to add to the
list. The handbook/toolkit would provide the fullest possible picture of indicators, tools
for measurement and assessment but it would not and could not replace discussion in-
country by implementers and policy makers on deciding priorities for action.

All media development implementers are interested in making our work better, increasing
impact and creating sustainability. It is an ambitious initiative to try and bring
implementers to work together on this issue but it is a much-needed initiative in order to
increase the positive impact of our work and to provide evidence-based arguments for the
recognition of media development as a crucial part of overall development and
democratisation strategies.

It is the aim of the initiative to involve all interested media development organisations
and other relevant groups in the preparation of the handbook/toolkit, in their different
areas of geographical and sectoral expertise.

This will be achieved through using the network of 400 media development organisations
of the GFMD as well as through inviting other media development organisations to join
the initiative. Relevant media researchers active in the field will be invited to join the
initiative.

As a result of the conference in Bad Honnef the German forum for media development
and CAMECO have offered to host a wiki-space to collect and share information. The
new GFMD web-site will be linked to this space and feed information from its members
into the discussion.

These initial steps can be used to collect relevant information and data to build the
handbook/toolkit.

Bettina Peters, Director 2

Global Forum for Media Development, IPC, Résidence Palace, Bloc C 2/215,
155 rue de la loi, 1040 Brussels, Belgium
director@mediagfmd.org
The handbook/toolkit would address to distinct and important areas of media
development programmes:

1. Assessment -- programme and project design

This section would build on the work done by IWPR with the proposal to use the logical
framework as the design framework for media development programmes and projects. It
would use and further refine the indicators proposed in the UNESCO background paper.

In obtaining data to assess media landscapes the media development community can
build on the already existing data that is being collected for the indices measuring press
freedom. Both the Freedom House and IREX MSI index measure and provide data on
many aspects of the media landscape in a given country. Through their involvement
individual data collected can be fed into programme and project design. While these
indices cover many of the issues and indicators listed, for instance, in the UNESCO
background paper, they do not address all of the indicators and additional data collection
would be required.

One of the main problems facing media development organisations in developing

programmes is that there in many areas there is little statistical or other quantifiable data
available. In order to measure impact of programmes over time the situation at the start of
the programme has to be measured. In many countries there is no reliable data available
on, for instance, actual ownership of media, size of the media market, size and
distribution of the advertising market. It is often also difficult to obtain data on the
practice of media law (rather than the letter), which is crucial in order to assess if media
and media professionals can operate freely.

An important part of the handbook would be to include references and possible sources
for data collection, especially in areas where this is difficult. The involvement of local
media development organisations and local research expertise is therefore crucial to the
success of the initiative.

At the same time the handbook would include sections on qualitative data, how to obtain
it and how to organise base-line studies. A lot of work has been done in this area and
media development organisations with a larger research capacity, such as, for instance,
the BBC World Trust, would be invited to share their experience on methodologies for
measurement.

The result would be a toolkit with advice on project and programme design that for
different sectors of the media system and different categories of desired outcomes would
propose indicators and key questions and identify sources to obtain data for assessment as
well as providing advice on how to organise methodologies for measurement.

Bettina Peters, Director 3

Global Forum for Media Development, IPC, Résidence Palace, Bloc C 2/215,
155 rue de la loi, 1040 Brussels, Belgium
director@mediagfmd.org
Groups of media development organisations could use the toolkit as a base to develop
joint, large programmes (similar to the “super-project” idea on media and governance
proposed in the IWPR paper) addressing several sectors and desired outcomes.

The handbook could also be used for the design of individual projects that address only
one sector or sub-sector with the relevant indicators identified. The tools proposed would
be useful for individual projects as well, plus individual projects could place themselves
into larger contexts by using some of the data sources dealing with other sectors of the
media landscape but potentially impacting on the proposed project.

2. Monitoring and evaluation of impact – indicators, data sources and tools for
measuring

The assessment data collected in the project design phase would provide the base for
measuring change.

For each indicator and key questions identified the toolkit would develop advice on what
type of sample indicators are needed to measure change.

It would propose methodologies for measuring impact for each indicator both in terms of
quantitative as well as qualitative data collection.

It would propose sources for data collection and provide advice on how to measure
change in different indicators. Again, the involvement of local organisations in this work
is crucial as possible data sources will vary considerably from country to country.

Building on the experience of media development organisations and using the

information exchanges organised in the GFMD/German media development
forum/CAMECO/IWPR wiki spaces and web-sites into which all interested media
development organisations can feed information, the toolkit will provide

• advice on what types of methodologies for measuring change have worked in

different projects,
• tips for data collection,
• advice on programme and project adaptation to respond to data collected,
• Ideas for cooperation with other sectors to increase the data pool (for instance, in
the area of media and governance, referencing to data from the World Bank or the
Transparency International National Integrity System (which includes a small
section on media).

The result would be guidance, tips, advice on running programmes and projects,
monitoring and measuring their impact using quantitative and qualitative data.

Bettina Peters, Director 4

Global Forum for Media Development, IPC, Résidence Palace, Bloc C 2/215,
155 rue de la loi, 1040 Brussels, Belgium
director@mediagfmd.org