Scribd Logo
Upload

Welcome to Scribd!

  • 818 views

    Modelling Cohen's Method For Standard Setting Progress Tests in The Cardiff Medical School

    Uploaded by

    Cardiff University School of Medicine C21 Curriculum Showcase
    Dr Saadia Tayyaba, Professor Kamila Hawthorne, and Professor Phil Smith Institute of Medical Education, School of Medicine, Cardiff University Background: Over fifty standard setting methods have been proposed to date for setting performance standards on medical exams but we do not have a gold ‘standard ‘ yet and are often faced with the dilemma of getting different and contrasting standards from different methods. Within Cardiff Medical School, we utilize both test and student centred methods to ascertain the pass mark for knowledge and clinical assessments, however, with the introduction of progress tests, there is a need to investigate an appropriate method to reliably standard set these exams as they are different from the traditional knowledge assessments in that they involve a longitudinal (progress) series of assessments set at a level appropriate for a new graduate, but they are taken by students at all levels of educational program(Ricketts, 2009) and cannot be standard set reliably with the traditional criterion methods. Unfortunately, there is very little research on standard setting progress tests to show appropriateness of a single method that might be more acceptable than the conventional methods. Objective: to investigate a new relative standard setting (Cohen’s) method to see if it could be used in its original or modified form to ascertain pass marks for the progress tests in Cardiff Medical School. Method: modelling historical knowledge test data from the available primary, intermediate, and final MB assessments to test the assumptions if the performance of the average, highest, or 95th percentile student at various reference points (i.e., 60%, 57.5% or 55%) produces stable pass mark with less variation in failure rate compared to Angoff method Conclusion: The method could be modified to suit individual medical school needs on choice of an appropriate reference point, relating standards to existing criterion reference methods and therefore could be proposed as the mainstream standard setting method for progress tests in Cardiff. References: Cohen-Schotanus J, van der Vleuten CPM. (2010). A standard setting method with the best performing students as point of reference: Practical and affordable. Medical Teacher; 32: 154–160. Ricketts, C., Freeman, A. C. and Coombes, L. R. (2009), Standard setting for progress tests: combining external and internal standards. Medical Education, 43: 589–593.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content

    You might also like

    Modelling Cohen's Method For Standard Setting Progress Tests in The Cardiff Medical School

    Uploaded by

    Cardiff University School of Medicine C21 Curriculum Showcase
    818 views1 page
    Dr Saadia Tayyaba, Professor Kamila Hawthorne, and Professor Phil Smith Institute of Medical Education, School of Medicine, Cardiff University Background: Over fifty standard setting methods have been proposed to date for setting performance standards on medical exams but we do not have a gold ‘standard ‘ yet and are often faced with the dilemma of getting different and contrasting standards from different methods. Within Cardiff Medical School, we utilize both test and student centred methods to ascertain the pass mark for knowledge and clinical assessments, however, with the introduction of progress tests, there is a need to investigate an appropriate method to reliably standard set these exams as they are different from the traditional knowledge assessments in that they involve a longitudinal (progress) series of assessments set at a level appropriate for a new graduate, but they are taken by students at all levels of educational program(Ricketts, 2009) and cannot be standard set reliably with the traditional criterion methods. Unfortunately, there is very little research on standard setting progress tests to show appropriateness of a single method that might be more acceptable than the conventional methods. Objective: to investigate a new relative standard setting (Cohen’s) method to see if it could be used in its original or modified form to ascertain pass marks for the progress tests in Cardiff Medical School. Method: modelling historical knowledge test data from the available primary, intermediate, and final MB assessments to test the assumptions if the performance of the average, highest, or 95th percentile student at various reference points (i.e., 60%, 57.5% or 55%) produces stable pass mark with less variation in failure rate compared to Angoff method Conclusion: The method could be modified to suit individual medical school needs on choice of an appropriate reference point, relating standards to existing criterion reference methods and therefore could be proposed as the mainstream standard setting method for progress tests in Cardiff. References: Cohen-Schotanus J, van der Vleuten CPM. (2010). A standard setting method with the best performing students as point of reference: Practical and affordable. Medical Teacher; 32: 154–160. Ricketts, C., Freeman, A. C. and Coombes, L. R. (2009), Standard setting for progress tests: combining external and internal standards. Medical Education, 43: 589–593.

    Original Title

    Modelling Cohen’s Method for standard setting Progress Tests in the Cardiff Medical School

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    Dr Saadia Tayyaba, Professor Kamila Hawthorne, and Professor Phil Smith Institute of Medical Education, School of Medicine, Cardiff University Background: Over fifty standard setting methods have been proposed to date for setting performance standards on medical exams but we do not have a gold ‘standard ‘ yet and are often faced with the dilemma of getting different and contrasting standards from different methods. Within Cardiff Medical School, we utilize both test and student centred methods to ascertain the pass mark for knowledge and clinical assessments, however, with the introduction of progress tests, there is a need to investigate an appropriate method to reliably standard set these exams as they are different from the traditional knowledge assessments in that they involve a longitudinal (progress) series of assessments set at a level appropriate for a new graduate, but they are taken by students at all levels of educational program(Ricketts, 2009) and cannot be standard set reliably with the traditional criterion methods. Unfortunately, there is very little research on standard setting progress tests to show appropriateness of a single method that might be more acceptable than the conventional methods. Objective: to investigate a new relative standard setting (Cohen’s) method to see if it could be used in its original or modified form to ascertain pass marks for the progress tests in Cardiff Medical School. Method: modelling historical knowledge test data from the available primary, intermediate, and final MB assessments to test the assumptions if the performance of the average, highest, or 95th percentile student at various reference points (i.e., 60%, 57.5% or 55%) produces stable pass mark with less variation in failure rate compared to Angoff method Conclusion: The method could be modified to suit individual medical school needs on choice of an appropriate reference point, relating standards to existing criterion reference methods and therefore could be proposed as the mainstream standard setting method for progress tests in Cardiff. References: Cohen-Schotanus J, van der Vleuten CPM. (2010). A standard setting method with the best performing students as point of reference: Practical and affordable. Medical Teacher; 32: 154–160. Ricketts, C., Freeman, A. C. and Coombes, L. R. (2009), Standard setting for progress tests: combining external and internal standards. Medical Education, 43: 589–593.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as pdf or txt
    818 views1 page

    Modelling Cohen's Method For Standard Setting Progress Tests in The Cardiff Medical School

    Uploaded by

    Cardiff University School of Medicine C21 Curriculum Showcase
    Dr Saadia Tayyaba, Professor Kamila Hawthorne, and Professor Phil Smith Institute of Medical Education, School of Medicine, Cardiff University Background: Over fifty standard setting methods have been proposed to date for setting performance standards on medical exams but we do not have a gold ‘standard ‘ yet and are often faced with the dilemma of getting different and contrasting standards from different methods. Within Cardiff Medical School, we utilize both test and student centred methods to ascertain the pass mark for knowledge and clinical assessments, however, with the introduction of progress tests, there is a need to investigate an appropriate method to reliably standard set these exams as they are different from the traditional knowledge assessments in that they involve a longitudinal (progress) series of assessments set at a level appropriate for a new graduate, but they are taken by students at all levels of educational program(Ricketts, 2009) and cannot be standard set reliably with the traditional criterion methods. Unfortunately, there is very little research on standard setting progress tests to show appropriateness of a single method that might be more acceptable than the conventional methods. Objective: to investigate a new relative standard setting (Cohen’s) method to see if it could be used in its original or modified form to ascertain pass marks for the progress tests in Cardiff Medical School. Method: modelling historical knowledge test data from the available primary, intermediate, and final MB assessments to test the assumptions if the performance of the average, highest, or 95th percentile student at various reference points (i.e., 60%, 57.5% or 55%) produces stable pass mark with less variation in failure rate compared to Angoff method Conclusion: The method could be modified to suit individual medical school needs on choice of an appropriate reference point, relating standards to existing criterion reference methods and therefore could be proposed as the mainstream standard setting method for progress tests in Cardiff. References: Cohen-Schotanus J, van der Vleuten CPM. (2010). A standard setting method with the best performing students as point of reference: Practical and affordable. Medical Teacher; 32: 154–160. Ricketts, C., Freeman, A. C. and Coombes, L. R. (2009), Standard setting for progress tests: combining external and internal standards. Medical Education, 43: 589–593.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as pdf or txt
    of 1

    Modelling Cohens Method for standard setting Progress Tests in the Cardiff Medical School

    Dr Saadia Tayyaba, Professor Kamila Hawthorne, Professor Phil Smith Institute of Medical Education, School of Medicine, Cardiff University

    Background
    Over fifty standard setting methods have been proposed to date for setting performance standards on medical assessments but we do not have a gold standard yet and are often faced with the dilemma of getting different and contrasting standards from different methods. Within Cardiff Medical School, we utilize both test and student centred methods to ascertain the pass mark for knowledge and clinical assessments. However, with the introduction of progress tests, there is a need to investigate an appropriate method to reliably standard set these exams as they are different from the traditional knowledge assessments and cannot be standard set reliably with the traditional criterion methods.

    Objectives of the Study


    The purpose of this study is to investigate the suitability of Cohens method for the recently introduced progress tests in the Cardiff Medical School and to find out if any further modification of this method could be implemented to ascertain the pass marks based on the modelling of available local data.

    Figure 1: Failure rates using various standard setting procedures

    Method: Application of Cohens method to Cardiff progress tests:


    Data were modelled from three recent SBA exams across the intermediate and final years. The original method was modified to find out if modelling various reference points and local multipliers produces stable pass mark and less variation in the failure rates.

    Figure 2: Comparison of pass rate using highest score as reference point

    Figure 3: Comparison of pass rate using 95th percentile as reference point

    Cohens standard setting Method


    Based on an analysis of the historical data including 54 norm-referenced tests from Maastricht and 52 criterion referenced tests from Groningen University, CohenSchotanus and Van der Vleuten (2010) introduced a new compromise method of standard setting which uses the best performing student as the reference point to set the pass mark. Using this method, the pass mark is calculated as 60% of the score of student at 95th percentile after correcting for guessing. The method is developed under the assumption that combining a pre-fixed cut-off score with a relative point of reference reduces the disadvantages of the conventional criterion and norm referenced methods. A recent modification (Taylor, 2011) of this method has shown application of this method reduces the variation in failure rate when compared to using a fixed pass mark

    We were particularly interested in modelling the following combinations to determine what would happen if we set the pass mark at: Based on the original Cohens method 60% of the highest score on the exam 60% of the 95th percentile score Following the modified method (Taylor, 2011) 60% of the 90th percentile score New proposed modification 57.5% of the 95th percentile 55% of the 95th percentile For comparison we modelled the pass mark at: 60% of the 90th percentile The standard norm reference method (Average-SD) Angoff pass mark.

    Figure 4: Comparison of pass rate using 90th percentile as reference point

    Results
    The new modification (57.5% of the 95th percentile) produced reasonable compromise across years in terms of pass rate consistency. Y04 SBA showed lower fail rate at the first reference point (i.e., 60% of the highest score) compared to Y03 and Y05. Failure rates in terms of percent are relativity consistent for the finals across other reference points on Cohen's method as well as the relative (Mean-SD) method. Table1: Failure rate across Years

    Conclusion
    Failure rate varies across years when using various reference points. Using an indirect criterion-referenced standard by changing the 57.5% multiplier to the score of 95th percentile student provides a better compromise. The assumption of the modified method (Taylor, 2011) that the performance of the 90th percentile student produces the best compromise is not supported, as a further modification based on our data shows 57.5% of the 95th percentile is the best compromise. Results of this study showed Cohens method is more acceptable compared to the conventional norm-referenced method (Average-SD) and also affordable in terms of time and cost. The method is flexible and could be modified to suit individual school policies on the choice of fixed percent multipliers (e.g., 57.5%), point of reference (95th percentile, highest score), and relating the standard to the existing criterion reference methods (e.g., using some Angoff-like discussions to decide on a local multiplier). Further research is indeed important to establish the validity and defensibility of this method for standard setting progress tests.

    Purpose and rationale of the Progress tests in Cardiff


    As a part of ongoing development of the MBBCH programme, regular progress tests have been introduced from 2013 to test the knowledge and skills of the tomorrows doctors in depth and to monitor their progress throughout the programme.

    Format of the Cardiff Progress tests


    Cardiff progress tests are based on a systematic blueprint covering all the core curriculum areas The tests are introduced as one formative and two summative tests in year 2 and one summative knowledge assessment in Y3 Y5. The test are taken three times a year across years (2-5), all the years students take the same test at the same time. Each test comprises 140 clinical vignette based SBAs style questions.

    Determining the results on Progress tests


    Each of the progress tests(Y2-Y4 ) are standards set using a norm-referenced method and converted to aggregate grades Students with an aggregate grade of unsatisfactory would be considered to have failed the year Final years Final years students take two tests and a third test for those with an unsatisfactory grade after the previous two tests. The first two progress tests in the finals contribute to awards of honours and commendations.

    Failure rate at the highest reference point is 17.6% for Year 3, (9.5%) for Year 5 and (4.2%) for the Year 4 SBA, respectively. Thus, Year 3 SBA (with average %correct of 66%) came up with highest fail rate with Cohen's method; the fail rate with relative method for this exam is similar to the finals.

    References
    Cohen-Schotanus J, Van der Vleuten C. (2010). A standard setting method with the best performing students as point of reference: Practical and affordable. Medical Teacher, 32, 154-160. Taylor CA. (2011).Development of a modified Cohen method of standard setting. Medical Teacher, 33: e678-682. Contact information: tayyabas@cardiff.ac.uk smithpe@cardiff.ac.uk hawthornek@cardiff.ac.uk

    You might also like