An Introduction Into (English) Language Testing

An introduction to (English) Language testing
An introduction into (English) Language testing

Bart Deygers Cel Diversiteit & Gender / Taalbeleid @ Ghent University CNaVT
What is assessment?
Why assess? Assess what?
History Concepts Construction Criteria Teaching Closing
A bit of history
Language testing history [Until about 1980], language was basically seen to be grammar: that eventually came to be regarded as too distant, too abstract.
(Davies 2008)
Language testing history [In the 1980s], language was reckoned to be a set of real life encounters and experiences and tasks, a view which took real life testing so seriously that it lost both objectivity and generality.
(Davies 2008)
Language testing history [From the 1990s] there has been a compromise between these two positions, where language is viewed as being about communication but in order to make contact with that communication it is considered necessary to employ some kind of distancing from the mush of general goings on that make up our daily life in language.
(Davies 2008)
Language testing now

Focus on: Methodology Practical advances Performance-affecting factors Performance assessment Ethical issues
(Bachman 2000)
Some key concepts
History Concepts
Test Evaluation Assessment Reliability Validity Face validity Authenticity
Language testing definitions

Test
An often formalised (collection of) task(s), designed to determine a test takers ability, knowledge or intelligence.
(Cf. Dochy 1996, 2002)
Construction
Criteria Teaching Closing
History Concepts

Test
Evaluation
The judgement made about a test takers ability, knowledge or intelligence, based on his/her test performance.
(Cf. Douglas 2000, Lynch 2003)
Construction
History Concepts

Test
Evaluation
Assessment
Judging the ability of a learner based on a test or otherwise and using this judgement as a constructive element in learning over time.
(Cf. Gipps 1994, Lynch 2005)
Construction
History Concepts
Language testing concepts

Reliability
Construction
History Concepts

Reliability If a test w, taken by student x, is graded twice by teacher y, student x will receive two identical scores. If a test w, taken by student x, is graded by teacher y and teacher z, student x will receive two identical scores.
Construction
History Concepts

Reliability Do test scores correctly reflect the learners actual ability? How can you draw conclusions based on test results if you are not sure about the results?
Construction
History Concepts

Reliability
Construction
History Concepts

Increasing reliability through
Identical criteria for students and tutors Transparent scoring No chain questions Rubric
Construction
History Concepts
Construction
History Concepts

Validity
Construction
History Concepts

Validity Write an essay on the consequences of climate change. Time: 30 minutes
Construction
History Concepts

Validity To what extent does the test really test what it is meant to test? How can you evaluate a specific ability if you are not measuring that ability?
Construction
Make sure you and your students know what you want to test!
History Concepts
Construction
History Concepts

Face validity The learners perception of how valid a test is. How can you expect test takers to take the test results seriously if they do not take the test seriously?
Construction
History Concepts

Face validity
Construction
History Concepts

Face validity
Construction
History Concepts

Authenticity Does the test include situations that are similar to what the learners will face in real life? How can you determine somebodys language performance in reality in the task does not correspond to reality? Authenticity matters mainly in productive, communicative tasks.
Construction
History Concepts

Authenticity
Construction
We dont use if-clauses. If you know something, you write about it. If you dont know for sure, you dont mention it.
History Concepts
Construction
Some thoughts on test construction
Test construction: questions

WHY WHAT HOW

WHY
Determine entry level Student evaluation Motivational Punishment
WHAT HOW

WHY WHAT
Test purpose
Test specifications:
-
HOW
What learners? Target language situation? Which skills? Which methods?

WHY WHAT
HOW Test purpose
Test specifications
Task types

Task types
discrete point / integrated / non authentic / simulated authentic / genuine authentic / multiple choice / ranking / hotspot / true-false / matching / structuring / fill in the gaps / cloze / C-cloze / semi-open / open answer / diary / portfolio / syllabus task / problem-based task / product assessment / process assessment / oral / written / computer-based / paper-based / self assessment / peer assessment/ co assessment/ tutor assessment / in-class observation / fixed-point testing / norm referencing / criterion referencing /
A word or two about criteria
History Concepts Construction Criteria

CEF Rubric
What about the CEF?
Teaching
Closing
The CEF
CEF Rubric
2001 / Council of Europe Goals: - Encouraging reflection - Fuelling discussion - Creating common language
One of the aims of the Framework is to help partners to describe the levels of proficiency required by existing standards, tests and examinations in order to facilitate comparisons between different systems of qualifications.
Teaching
Closing
CEF: system
CEF Rubric
Skilled User
C2 C1 B2 B1 A2 A1
Teaching
Independent user
Closing
Basic user
Full text Overview
CEF: influence
CEF Rubric
DIALANG
IELTS TOEFL CNaVT
Teaching
Closing
Handboeken
CEF
Didactiek
EUROPASS
Talenscholen
CEF: influence
CEF Rubric
DIALANG
IELTS TOEFL CNaVT
Teaching
Closing
Handboeken
CEF
Didactiek
EUROPASS
Talenscholen
CEF: influence
CEF Rubric
DIALANG
IELTS TOEFL CNaVT
Teaching
Closing
Handboeken
CEF
Didactiek
EUROPASS
Talenscholen
CEF: influence
CEF Rubric
DIALANG
IELTS TOEFL CNaVT
Teaching
Closing
Handboeken
CEF
Didactiek
EUROPASS
Talenscholen
CEF: influence
CEF Rubric
DIALANG
IELTS TOEFL CNaVT
Teaching
Closing
Handboeken
CEF
Didactiek
EUROPASS
Talenscholen
CEF: influence
CEF Rubric
DIALANG
IELTS TOEFL CNaVT
Teaching
Closing
Handboeken
CEF
Didactiek
EUROPASS
Talenscholen
CEF: problem solved?

CEF Rubric
Teaching
Closing
CEF: give it a go
CEF Rubric
www.ceftrain.net
Teaching
Closing
CEF: relating tests

CEF Rubric
Step 1 Specification
Step 2 Standardisation (CEF-training)
Step 3 Validation (Test analysis)
Internal validity
Teaching
Linking benchmarked items to the CEF Linking test answers to the CEF
Verifying psychometric test quality Independent study
Closing External validity
Implementation
Confirmation

CEF Rubric
Holistic rubrics
Teaching
Closing

CEF Rubric
Dichotomous rubrics
Well-paced flow Yes 1 No 0
Teaching
Closing
Message is clear Acceptable pronunciation Effective use of grammar Effective use of vocabulary .
1 1 1 1
0 0 0 0

CEF Rubric
Band rubrics
Teaching
Closing
Language testing and Language teaching
Testing and teaching

Washback

Washback

Washback

Motivation

Motivation
Reliability Authenticity (Face) Validity
// Fairness // Realness // Credibility

Motivation
[teaser: testing the test]
Descriptive statistics
How difficult is my test? Average
How difficult is my test? Average Standard Deviation
How difficult is my test? Average Standard Deviation
Voorbeeld: max = 100 | Ave = 50 | StDev = 10 68,3 % = 40 60 95,4 % = 30 70 99, 7% = 20 - 80 68,3% max 1 x SD 95,4% max 2 x SD 99,7% max 3 x SD
Average or median?
How well-off is the average employee of Peters & Sons?
How difficult is my test? Average or median?
Correlations Is this test as difficult as last years test? Is group A as proficient as group B?
Correlations
Test 1 1 2 3 4 5 6 7 8 Test 2 1 2
8 12 10
3 4 5 6 7 8
2 6 Series1 Series2
0 0 2 4 6 8 10 12
9
10
9
10
Corr = + 1
Correlations
Test 1 1 Test 2 10
2
3 4 5 6 7 8 9 10
9
8 7 6 5 4 3 2 1
Corr= - 1
Correlations
1,00 2,00 3,00 4,00 5,00 6,00 7,00 8,00 6,00 3,00 5,00 1,00 6,00 8,00 2,00 4,00
Corr= + .05
Correlations
Correlations
Split-half reliability
Is the level of difficulty consistent within the test?
Split-half reliability Correlation between 2 test halves
Set 1 #1 #3 #5
Set 2 #2 #4 #6
TOT_100
80 ,0 0

70 ,0 0

60 ,0 0

11 ,0 0
12 ,0 0
13 ,0 0
14 ,0 0
15 ,0 0
16 ,0 0
Corr = + 1
TOT_20
Split-half reliability
Set 1 #1 #3 #5
Set 2 #2 #4 #6
16 ,0 0
SCORE_ADMITTED
14 ,0 0
12 ,0 0
10 ,0 0 2,00 3,00
4,00
5,00
6,00
Corr = + .66
EX_4
Discriminating potential Does this question separate high achievers from weaker students?
Cronbachs Alpha
Cronbachs Alpha
Cronbachs Alpha
Something to take home
Task 1: Increase your self-esteem

Know what the CEF is!
http://www.coe.int/t/dg4/linguistic/Source/Framework_EN.pdf
Know what TOEFL and IELTS are!

www.toefl.org www.ielts.org
Remember something about reliability and validity!
Task 2: Test construction For one of your classes, create a test which is motivating, valid and reliable.

An Introduction Into (English) Language Testing

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

An Introduction Into (English) Language Testing

Uploaded by

Copyright:

Available Formats

An introduction to (English) Language testing