Analysis

of the New Mexico School Grading Technical Guide

Presented to the Legisla<ve Educa<on Study CommiAee ********************

Coali<on for Excellence in Science and Math Educa<on

kimber@comcast.net Lisa Durkin earthnskynlight@msn.com Walt Murn murf2345@aol.com
August 2012 M. Kim Johnson

Background
CESE, a non-prot, non-parJsan 501(c)3 corporaJon, performed this study
Members include Sandia NaJonal Laboratory reJrees, an industry physicist, and a high school science teachers We have categorized school performance, in terms of comparison to standards and with respect to demographically neutral expectaJons

CESE has analyzed New Mexico public educaJon data for over a decade

CESEs primary focus is to help schools improve

Approach
CESE was requested to use its experJse in the eld of staJsJcs and related numerical analysis to Provide
feedback on:
The ability of schools to eecJvely use the Public EducaJon Department (PED) publicaJon: New Mexico, School Grading Technical Guide, CalculaJon and Business Rules ReplicaJon of school grade calculaJons by experts at math/staJsJcal modeling.

We have also esJmated how well a non-expert in data analysis could use the publicaJon. Our conclusions are based upon our direct experience in working with a number of schools and districts in New Mexico
3

We simply asked our science teacher to act in this role as a proxy for a typical district superintendent, principal, etc.

Caveats and General Comments

CESE did not have all of the data that PED uses
Exact replicaJon of results was therefore not possible and not a]empted

If the grade calculaJons were formulaic in nature, we simply assessed our ability to perform them We have provided comments where improvement in the technical manual appears to be needed
4

Comments/Ques<ons
Grades need to be ve]ed with a knowledgeable individual from the school district i.e. a sanity check before being publicized

Manual SecJon VI.E.2 through VI.l lacks clear and/or consistent deniJons

One error can propagate through the enJre calculaJon process for all the growth areas The mathemaJcal knowledge and soaware required to perform the calculaJons are too complex for most school districts

The calculaJon for growth requires an expert and most probably, custom soaware not available to school districts (even power users of Excel may not be able to perform these calculaJons). Will the enJre database used for SecJons VI.E.2 through VI.l be made available to each district? Will the soaware used to perform the grade calculaJon be provided to schools districts with training? Some specic issues:
SecJon I.I School Growth. Cohorts Mixed eects is inconsistent with the SAS Manual SecJon 3.2 SecJon I.J What is Panel? The deniJon of growth is inconsistent with the ABCDF Act NMSA (1978) 22.2E-2. DeniJons, and 22-2E-4. Annual RaJngs There is no way to tell a before/aaer school by code number for reorganized schools

Comments/Ques<ons (con<nued)
Some specic issues (conJnued):
Data ValidaJon FAY must be consistent with data in School Accountability Reports Where are the formulae in E.2 and F? Where are the deniJons? VI.F: mulJple quesJons
It appears there is a notaJon change from equaJon (1) to equaJon (2) perhaps Mixed models for school evaluaJon seem inconsistent It is not clear it is consistent with the ABCDF Act

In general, there is inadequate explanaJon in the PED manual about the growth model The Table of Point Boundaries is not at all explained
How were they derived? Were the lowest and highest performers accounted for?

In General, once all the details are derived, they are put together according to another set of rules to arrive at a grade How can one de-convolve the eects of demographics (VI.F, G, H, and I) and then jusJfy pulng them back into the mix to form a part of the grade ascribed to a given school? Expanding on this, adding the eects of demographically neutral growth to the performance against standards is something like adding oranges and cows to derive pickup trucks. The result is not obviously meaningful.
Each serves a separate purpose, and cannot be simply added together.

For ALL the factors in the manual, we have no evidence that they are weighted according to any set of criteria other than this seems OK
6

Comments/Ques<ons (concluded)
The PED uses a VAM model to derive a CondiJonal score (aka Residual as per the CESE method of calculaJng actual score minus expected score.
The reason to look at residuals is to tell which school (or teachers) are performing above expectaJons based on removing demographic eects that are uncontrolled by the school or a teacher Schools that outperform calculated expectaJons should be studied for best pracJces, for a given demographic classicaJon, to apply to other schools with similar demographics

Factors involving performance compared to a standard tell a school where it stands regarding its overall test performance, which is useful in a dierent way The addiJon, in any manner of a condiJonal score (demographics removed) with an uncondiJoned score makes no mathemaJcal sense as far as we have been able to determine.

Manual Calcula<ons Sec<on VI. B. ParJcipaJon VI. C. A]endance VI. D. GraduaJon VI. E Current Standing VI. F. School Growth VI. G. Student Growth VI. H. Highest QuarJle Student Growth VI. I. Lowest QuarJle Student Growth VI. J. Opportunity to Learn VI. K. College and Career Readiness

Results
Yes Yes Yes

Expert Ability to Replicate

Es<mated Non-Expert Ability to Replicate Yes Yes Yes No No No No No Yes Yes

8

Dicult, but Probable with QualicaJons Dicult, but Probable with QualicaJons Dicult, but Probable with QualicaJons Yes, if deniJon is claried Yes, if deniJon is claried Yes Yes

Specic Recommenda<ons
The exisJng PED manual needs substanJve improvement
It should be peer reviewed by knowledgeable, independent educaJonal staJsJcians The methodology should be clearly and completely dened, and all data and soaware should be provided to districts provided to districts prior to grade distribuJons

It would be easier for districts to use a simpler model for the current VAM used:
values to opJmize removal of demographic eects Others

Canonical correlaJon model that automaJcally weights dierent

The PED should not try to put everything together in one grade, e.g., demographically neutral growth and prociency residuals do not combine with non-demographically neutral prociency scores in a meaningful way. These are two dierent measures with two dierent outcomes.

Specic Recommenda<ons (Concluded)

Any factor may or may not pertain to performance outcome. Even if they do, it is not clear if they are redundant or actually add informaJon.
If they do not add informaJon, or do not pertain to outcome, then
they are not appropriate to use.

Even if the weighted factors may be appropriate, there is no

indicaJon of how the weighJng was decided. There May be other opJons (perhaps mathemaJcal) rather than using best guesses as it now appears. ClaricaJon is necessary for:
How the ABCDF Act has been followed using the manual The method for selecJng and combining performance factors.
10

Characteris<cs of a Valid Grading System Not Apparent from the Manual

The PED grading system should include the following traits:
A defensible, clearly dened, and more easily replicable mathemaJcal process that districts can also use A transparent, defensible process (possibly mathemaJcal) to determine the opJmum grading factors weighJng The weighJng factors are necessary for combining the various similar, meaningful performance output factors to derive a grade

Without these characterisJcs the grading system loses meaning and doesnt necessarily help educaJon reform
11