This document outlines a framework for understanding research evaluation systems. It discusses how national evaluation systems are organized sets of procedures for assessing publicly funded research. These systems permeate research spaces and fields by institutionalizing rules, responsibilities, procedures and consequences for repeated evaluation. They are generally conceived and governed within research spaces to manage the performance of funders, research organizations, teams, researchers and knowledge. The document also notes some broad conditions for evaluation systems, including ensuring a fit between overall and local practices, establishing appropriate timelines, and facilitating sharing of evaluation experiences.
Original Description:
Original Title
20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
This document outlines a framework for understanding research evaluation systems. It discusses how national evaluation systems are organized sets of procedures for assessing publicly funded research. These systems permeate research spaces and fields by institutionalizing rules, responsibilities, procedures and consequences for repeated evaluation. They are generally conceived and governed within research spaces to manage the performance of funders, research organizations, teams, researchers and knowledge. The document also notes some broad conditions for evaluation systems, including ensuring a fit between overall and local practices, establishing appropriate timelines, and facilitating sharing of evaluation experiences.
This document outlines a framework for understanding research evaluation systems. It discusses how national evaluation systems are organized sets of procedures for assessing publicly funded research. These systems permeate research spaces and fields by institutionalizing rules, responsibilities, procedures and consequences for repeated evaluation. They are generally conceived and governed within research spaces to manage the performance of funders, research organizations, teams, researchers and knowledge. The document also notes some broad conditions for evaluation systems, including ensuring a fit between overall and local practices, establishing appropriate timelines, and facilitating sharing of evaluation experiences.
research evaluation Maria Nedeva, MIoIR the University of Manchester Warsaw, December 3, 2010 Outline of presentation Science/innovation systems and evaluation systems Evaluation framework Evaluation practice Broad conditions for evaluation What to watch for Science/innovation systems Science/innovation systems Research spaces and fields Research organisation(s) FO1 FO2 FO3 KC1 KC2 KC3 KP2 KP1 KP3 KEY: KC = knowledge community KP = knowledge pool FO = funding organisation KCs interactions with different research spaces Research space Research field Research fields and policy spaces Research space is a policy space Relationships between regulators, funders and performers of research Funders can be layered e.g. Government organisations, executive agencies etc. National Evaluation System: definition Research evaluation systems are organised sets of procedures for assessing the merit of research undertaken in publicly funded organisation that are implemented on a regular basis, usually by state or state-delegated agencies...[NESs vary] in terms of their frequency, formalisation, standardisation and transparency. R. Whitley, 2007 (The Changing Governance of the Sciences) National Evaluation System Permeates research spaces and fields Institutionalised: rules, responsibilities, procedures, consequences Repeated National Evaluation System: organisation Who commissions Who organises Who carries out Who applies the results Example the RAE in the UK Organisation of science and evaluation systems Answers depend on the organisation of the research space (national) As a rule of the thumb: Political organisations commission (Ministries, Govnt Departments) Intermediaries (agencies) organise Multiple actors carry out (contractors, panels, experts, in- house specialists) Political organisations and agencies apply Organisation of science and evaluation systems NESs are conceived, developed and governed in the Research Space They can target social phenomena in research spaces and research fields Generally evaluations are about performance management: of funders, research organisations, research teams, researchers and knowledge Evaluation as management practice Information Judgement Action Information About what? Object Types of information How collected? Methodologies Sources Who collects? In-house Outsourced Judgement Criteria Standards Multiple criteria Layered Depend on dominant ideological beliefs Action Funding decisions Regulation Change Closure Evaluation practice Distortions Indicator based Peer review based Evaluation methodology(ies) Broad conditions of evaluation Fit between the overall and localised evaluation practices Time-lines Sharing of practice Experience Comparison Relying on others Pitfalls Over- evaluation Distortions of the science system knowledge management Assumptions Thanks for listening! Any questions? Maria.Nedeva@mbs.ac.uk