Qualitative Metaevaluation Form Using The Program Evaluation Standards, 3rd Edition

MEDT 8480

Evaluator: Group 8
Metaevaluator Melissa Brown, Jason Garofalo, 4-9-17
: Thomas Kratowicz, Frederick Wright
NAME DATE

Instructions: Rate the relevance of each standard as it currently applies to the present evaluation effort. Then provide
feedback to the evaluator on each standard by highlighting where the evaluator addresses compliance with the standard
statement in the evaluation plan, and the extent that the standard is being met. If in your opinion the evaluator has not
or has insufficiently addressed the standard, indicate so and provide constructive feedback or a suggestion as to how to
improve the plan. Finally, in the Overall Comments section found below, summarize your feedback as it pertains to the
evaluations Utility, Feasibility, Propriety, Accuracy, and Evaluation Accountability. Try to come up with an overall
statement of the evaluation plans merit taking into consideration the context of the evaluation, the relevance of the
evaluation standards to the evaluation effort, and the extent that the standards are adhered to and met by the
evaluator in the evaluation plan. NOTE: This particular assignment is to provide students an experience with an
important and often overlooked aspect of evaluation: metaevaluation. Your metaevaluative conclusions and feedback
will in no way negatively impact the course grade of the particular evaluator; however, they should be a constructive, yet
fair assessment so as to help improve the overall evaluation effort.

When addressing each standard, consider the whole evaluation plan including appendices. This is particularly true of the
Accuracy Standards. Are the evaluation instruments (if any) and methods sufficient to answer the evaluation questions?
If you have additional feedback (e.g., formatting, editing, APA), please comment in the Additional Feedback section.

Overall Comments:
Overall, your plan is well thought out. In the beginning of your plan, you gave good amount of researched information
on the assessments that are tied into the program that you are evaluating, but you did not give a lot of information on
the two remediation components of the program which are System 44 and Read 180. Also, it may be beneficial to
explain the differences in the two programs and how they remediate the students. This helps for clarity in the overall
understanding of the purpose of the evaluation and what evaluators are looking for. Adding timelines for collecting data
will strengthen the Utility standards, as will adding the contract. The contract also should address how the data will be
analyzed and the findings communicated to stakeholders. The contract will also strengthen the standard of Accuracy,
A8, Communication and Reporting.

Additional Feedback:
Some spacing issues are present, but these are minor. For example, between sentences there is a mix of one, two and
three spaces. The title page has a heading at the bottom, and the References need to start on a new page. These are all
minor grammar issues, however. Overall, great job formatting.
 STANDARDS STANDARD STATEMENTS RELEVANCE TO QUALITATIVE FEEDBACK/COMMENTS
THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
UTILITY
U1 Evaluator Evaluations should be conducted by X U1 Evaluator Credibility is highly relevant and the standard is met
Credibility qualified people who establish and through the 4 educators serving on the evaluation team who hold a
maintain credibility in the evaluation combined 30 years of experience in K-12 education.
context.

U2 Attention to Evaluations should devote attention to X U2 Attention to Stakeholders is highly relevant and the standard is met
Stakeholders the full range of individuals and groups in the evaluation of educational software for improving reading. The
invested in the program and affected evaluation is inclusive to include input from teachers and students, and
by its evaluation. the standard is met.

U3 Negotiated Evaluation purposes should be X U3 Negotiated Purposes is clearly relevant and the standard is met
Purposes identified and continually negotiated through negotiations with the evaluation client, a 5th-grade teacher who
based on the needs of stakeholders. utilizes System 44 and Read 180 programs with students for
remediation. The negotiated contract will be an added component of
the final Evaluation Plan.

U4 Explicit Values Evaluations should clarify and specify X U4 Explicit Values is clearly relevant in an evaluation of educational
the individual and cultural values software for improving reading. All three of the evaluation questions
underpinning purposes, processes, and meet Alkins first priority of relevance for local needs. Values are
judgments. considered and the standard is met.

U5 Relevant Evaluation information should serve X U5 Relevant Information is highly relevant in an evaluation of
Information the identified and emergent needs of educational software for improving reading. This standard is met;
stakeholders. quantitative data will be collected through the Scholastic Assessment
Management System and expressed through 15 reports available in the
areas of progress monitoring, diagnostics, instructional planning, and
alerts and acknowledgements.

U6 Meaningful Evaluations should construct activities, X U6 Meaningful Processes and Products is highly relevant in the
Processes and descriptions, and judgments in ways evaluation of educational software for improving reading. The standard
Products that encourage participants to is met in this evaluation; the formative evaluation will assist the client in
rediscover, reinterpret, or revise their
determining if the first year of implementation of System 44 and Read
understandings and behaviors.
180 are meeting the programs goals to help students improve their
reading.

U7 Timely and Evaluations should attend to the X U7 Timely and Appropriate Communicating and Reporting is highly
Appropriate continuing information needs of their relevant in the evaluation of educational software for improving
Communicating multiple audiences. reading. The standard is met; reporting delays are not indicated or
and Reporting expected to occur in this evaluation.

U8 Concern for Evaluations should promote X U8 Concern for Consequences and Influence is clearly relevant in an
Consequences responsible and adaptive use while evaluation of educational software for improving reading. This standard
and Influence guarding against unintended negative is partially met; the final Evaluation Plan will include inclusion of a
consequences and misuse. contract.
 STANDARDS STANDARD STATEMENTS RELEVANCE TO QUALITATIVE FEEDBACK/COMMENTS
THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
FEASIBILITY
F1 Project Evaluations should use effective x F1 Project Management is not met. In this evaluation plan, you are
Management project management strategies. missing timelines, time estimates on when you will collect data and
analyze data, and how you plan on communicating with your
stakeholders. You have strong purposes, goals, and objectives, just
missing the time and communication element.

F2 Practical Evaluation procedures should be x F2 Practical Procedures is partially met. Your evaluation plan has strong
Procedures practical and responsive to the way bones when it comes to program goals and program reports. The
the program operates. weaknesses are in setting into place procedures to make sure that the
data that you will be collecting from students is permitted. Since these
are school age children, there are laws to protect their privacy. What
steps will you put into place to make sure that they are protected. You
did a great job making sure that the stakeholders understanding of the
programs purposes, goals and procedures matched your evaluation
goals.
F3 Contextual Evaluations should recognize, monitor, x F3 Contextual Viability is met for this standard. By adding in the survey
Viability and balance the cultural and political to teachers and getting their input on the function, training, reports,
interests and needs of individuals and and the program itself, you have given them opportunities to have their
groups. opinions and concerns about the program told. Since this is the first
year the program has been in use, there is no previous data to consider.

F4 Resource Use Evaluations should use resources x F4 Resource Use is not met this standard. In your written evaluation,
effectively and efficiently. you have not mentioned any costs nor a budget for implementing this
evaluation. You have stated that you will be using a survey. Will they
survey cost anything? You have also stated that you will be using data
provided by the system. Who will analyze the data? Will that cost?
 STANDARDS STANDARD STATEMENTS RELEVANCE TO QUALITATIVE FEEDBACK/COMMENTS
THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
PROPRIETY
P1 Responsive and Evaluations should be responsive to P1 Responsive and Inclusive Orientation is met. Stakeholders are
Inclusive stakeholders and their communities. X identified and have participated in the process of evaluation questions
Orientation and planning. Good detail is given about relevant SRI district,
history/purpose of the program, and evaluators skill sets; team
members have ample education experience, understanding of various
educational testing data, and local population. Descriptions of sensitivity
to data and listening stakeholder needs are mentioned, but not
developed.
P2 Formal Evaluation agreements should be P2 Formal Agreements is not met.
Agreements negotiated to make obligations explicit X No formal agreement or MOU is present.
and take into account the needs,
expectations, and cultural contexts of It is clear that the team has worked with stakeholders on what will
clients and other stakeholders. indicate success of the program in regards to the evaluation questions.
However, no formal documentation is identified.

P3 Human Rights Evaluations should be designed and
and Respect conducted to protect human and legal
rights and maintain the dignity of
participants and other stakeholders.
P3 Human Rights and Respect is met.
X There are no major privacy concerns or data sharing concerns are
indicated. Data is already shared between stakeholders and would be
unlikely to require special consideration.
To improve this standard, a statement about the results focused on the
rights of the participants, would help clarify intentions and
accountability. Some of the population may have cultural sensitivities
that need to be addressed as irrelevant.

P4 Clarity and Evaluations should be understandable P4 Clarity and Fairness is mostly met.
Fairness and fair in addressing stakeholder X
needs and purposes. Teachers Experiences and opinions, in the form of qualitative survey
data, will be collected and considered.
Children involved in the program are not given an opportunity to share
experiences as participants. Adding interviews, or other survey data,
could ensure childrens experience with the program is considered.

P5 Transparency Evaluations should provide complete
and Disclosure descriptions of findings, limitations,
and conclusions to all stakeholders,
unless doing so would violate legal and
proprietary obligations.
P5 Transparency and Disclosure is mostly met.
X
Openness and communication with stakeholders are clearly indicated.
Presentation of the logic model, evaluation questions, and data
collection processes are clearly discussed.

How will teachers survey responses be used? Reporting of results is also

not addressed in the plan. Ensuring results are disclosed and explained
clearly would need to be added to the plan to meet this standard.

P6 Conflicts of Evaluations should openly and P6 Conflicts of Interest is not met.

Interest honestly identify and address real or
perceived conflicts of interest that may
compromise the evaluation.
X
While there are no warranted concerns presented in the plan, it is not
addressed.
Adding a statement about how to ensure results are delivered with care
is needed. Teacher participant evaluator interest does not seem to
conflict, but his may need to be clarified.
To improve this standard, address formal conflict of interest in MOU
documentation.

P7 Fiscal Evaluations should account for all X P7 Fiscal Responsibility is not yet met.
Responsibility expended resources and comply with No statement of budget or funding has been indicated. What resources
 sound fiscal procedures and processes. will be used to pay the evaluation team?

STANDARDS STANDARD STATEMENTS RELEVANCE TO QUALITATIVE FEEDBACK/COMMENTS

THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
ACCURACY
A1 Justified Evaluation conclusions and decisions A1 Justified Conclusions and Decisions is mostly met.
Conclusions should be explicitly justified in the x
and Decisions cultures and contexts where they have For this standard you need to clarify which stakeholders will draw
consequences. conclusions and make decisions. You do a good job of explaining the
role of the teacher in the evaluation program but need to define the
responsibilities of other stakeholders in the program. You also do a good
job describing who is being evaluated in the program and how it will
happen.
A2 Valid Evaluation information should serve A2 Valid Information is met.
Information the intended purposes and support x
valid interpretations. It seems that you have clearly identified the programs that will be used
to determine the validity of your program.

A3 Reliable Evaluation procedures should yield A3 Reliable Information is met.

Information sufficiently dependable and consistent
information for the intended uses.
x
The tests you use to gather information should provide you with the
reliable information you need to determine whether or not the program
works. One thing you can do to improve this standard is listing sources
of error that could occur in your evaluation.

A4 Explicit Evaluations should document A4 Explicit Program and Context Descriptions is mostly met.
Program and programs and their contexts with x
Context appropriate detail and scope for the It seems that you could provide more detail about the programs READ
Descriptions evaluation purposes. 180 and System 44 and what they entail. Be specific about what goes on
in these programs. You do provide good information through your logic
model. Is there any input that would be relevant through stakeholders
or will this be found in the survey?

A5 Information Evaluations should employ systematic A5 Information Management is not met

Management information collection, review,
verification, and storage methods.
x
For this standard is all the data collected in either Read 180 or in System
44? And if so how is the data categorized? A final goal is stated at the
end of the program but are their other goals for the individuals? You
need to incorporate more information from your Propriety standards
and once that is completed this part will be done.

A6 Sound Designs Evaluations should employ technically A6 Sound Designs and Analyses is mostly met.
and Analyses adequate designs and analyses that x
are appropriate for the evaluation For this standard you hit on the logistics of the programs. You could
purposes. provide more specific details about the programs in the sense of cultural
and political aspects. Are there any other constraints that you need to
consider? costs? validity?

A7 Explicit Evaluation reasoning leading from A7 Explicit Evaluation Reasoning is not met.
Evaluation information and analyses to findings, x
Reasoning interpretations, conclusions and How will you go about analyzing the data? Is it only interpreted by the
judgments should be clearly and teacher? What are your assumptions and arguments about the quality
completely documented. of reasoning?
A8 Communicatio Evaluation communications should A8 Communication and Reporting is not met.
n and have adequate scope and guard x
Reporting against misconceptions, biases, You do report how you will gather quantitative and qualitative data, but
distortions, and errors. how will you communicate the results that you get to stakeholders? Are
there any bias you could face with your program? If so how would you
protect against them. Will a formal report be given to stakeholders? Do
you only need one report?

STANDARDS STANDARD STATEMENTS RELEVANCE TO QUALITATIVE FEEDBACK/COMMENTS

THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
EVALUATION ACCOUNTABILITY
E1 Evaluation Evaluations should fully document x For E1 Evaluation Documentation, you have listed the types of
Documentation their negotiated purposes and documentation in your writing that you will use in your evaluation, but
implemented designs, procedures, you have not put it into chart form as suggested in this standard. As you
data, and outcomes. stated in your plan, there are types of reports that the system uses that
you will be able to utilize for your data. Growth reports, Comparison of
Milestone Data, and amount of time on program reports will all be
utilized. All of this data will assist in answer your three evaluation
questions. My one caution is to make sure that even though there are
many reports offered by your program, you use the ones that are most
aligned to answering your evaluation questions.
E2 Internal Evaluators should use these and other x E2 Internal Metaevaluation is not applicable in this part of the process.
Metaevaluatio applicable standards to examine the Some things to ponder when you do perform your evaluation:
n accountability of the evaluation Are your resources accurate for the data you are trying to collect?
design, procedures employed, Were the activities practical and complementary to the program you are
information collected, and outcomes. evaluating?
Was the evaluation responsive to stakeholders and the community?
Did the evaluation encourage use of processes and findings?

E3 External Program evaluation sponsors, clients, x E3 External Metaevaluation is not applicable in this part of the process.
Metaevaluatio evaluators, and other stakeholders Some things to ponder when you do perform your evaluation:
n should encourage the conduct of Are your expectations clarified about the needs and purposes that the
external metaevaluations using these external metaevaluation will serve? Did you identify the relative
and other applicable standards. advantages and disadvantages of the different types of meta
evaluations?
Did you list resources required for the external metaevaluation?
Identify credible metaevaluation experts. Be in constant collaboration
with stakeholders. Identify values, standards, and other dimensions of
quality. Communication is key.