ExcelSoft Knowledge Labs Author : Ranganath S

IRT Based Automated Test Assembly

Introduction:
Test Information Function (TIF) plays a very important role in test construction. TIF
indicates how well the test can measure at certain ability level. TIF is analogous to classical test
theory reliability but TIF has the advantage of providing the measure of precision at each trait
or ability level rather than a global measure.

The Standard Error of Measurement (SEM) is defined in terms of Information as below

SEM = 1/information

Peak in the information function indicates less measurement error which will help in achieving
the desired measurement goal with the test. The decision on the shape of the TIF and its peak is
driven by the goal of the test. Possible example goals are

 Placement of students in courses,

 Diagnosis of the intelligence of children in the lower tail of a given distribution,
 Selection of applicants for a job and
 Assembly of a new version of a test that is parallel to the current version.

Page 1
ExcelSoft Knowledge Labs Author : Ranganath S

(1) Test used for admission decisions with cut-off score theta=0.

(2) Diagnostic test over lower range of abilities.

(3) Test with an information function that follows a given population distribution.

The Y-axis in the above plot indicates the information, I(θ) and the X-axis plots the ability scale
θ.

The test construction/assembly can be both manual and automated. In both the cases it
is required to input various criteria/parameter(Information and no of items) of the test at
various abilities. The automated process selects the items which satisfy the mentioned criteria,
however in the case of manual process both setting the criteria and item selections are manual.
Lord outlined procedure, originally conceptualized by Birnbaum for the use of item information

Page 2
ExcelSoft Knowledge Labs Author : Ranganath S

functions in the test building process is listed below. Basically, this procedure entails that a test
developer take the following four steps:
1. Describe the shape of the desired test information function over the desired range of
abilities. Lord calls this the target information function.
2. Select items with item information functions that will fill up the hard-to-fill areas under
the target information function.
3. After each item is added to the test, calculate the test information function for the
selected test items.
4. Continue selecting test items until the test information function approximates the target
information function to a satisfactory degree.

Linear Programming to Test Construction:

The test construction is more than a mere picking of items for the test to be
solved in the given time. It needs to comply with lot of constraints which are in the
direction of achieving the desired measurement goals of the test. The author in the
manual test construction has to balance between many such constraints to come out
with the set of test questions. This set of questions may not provide the optimal set
from the given item bank which satisfies the test goal. This activity is also time
consuming considering the number of test constraints. This problem of test construction
has an answer in the Linear Programming, a mathematical method which is widely used
in industry and business for decision making. Applying Linear programming techniques
to achieve the test construction problem is innovative and new method in the area of
educational measurement.
The Binary Linear Programming which solves the test item selection problem
takes in the user specification of the test. The user specification composes of objective
and constraint. These are defined below.
Objective - A test specification is an objective if it requires an attribute to take a
minimum or maximum value possible for the given item bank/pool.

Page 3
ExcelSoft Knowledge Labs Author : Ranganath S

Constraint - A test specification is a constraint if it requires an attribute (eg: Time

for test, Test Information, words allowed in the test, etc) to satisfy a specified
upper and lower bound. Say for eg; no of items in the test not to be less than 58
and not more than 62, similarly with no of words in the item.

Since the Automated Test Assembler is IRT based, the obvious selection of the objective
to achieve is the target information curve set by the user. Some of the constraints at various
levels are listed below.

Test Level
1. Number of lines /words in the test.
2. Total Number of items in the test not exceeding the prescribed limit.
3. Total time (Sum of the item time averages) not exceeding the allocated exam time.

Item Level
5. Excluding the enemy Items.
6. IRT parameters’ range.
7. Ceiling on the number of question types in the test or in each of the sub-section (like
History, Geography etc).
8. Inclusion of logically related items i.e. if with the selection of one item the other
item(s) is required to be selected it is called as logically related. (Logical Constraint).

Item -Set Constraints

8. Number of Items from each sub-section (like History, Geography etc)
9. Other Item Set constraints are behavioral taxonomy, item author, presence of certain
auxiliary material in a stimulus like graphs, tables or mathematical equations.

Page 4
ExcelSoft Knowledge Labs Author : Ranganath S

User Interactions:

The above UI is to set the set of constraints, objective function for the LP being to
optimize the difference between the set target information function and that obtained from
selection of items from the item bank. The following are the constraints in the Test Level that
can be set for the test being constructed.
1. Test Time – Takes the test time in seconds. This is not an optional constraint and the
user needs to set this.

Page 5
ExcelSoft Knowledge Labs Author : Ranganath S

The remaining are optional constraint and the checkbox in its front needs to be checked for
constraint inclusion.

2. Test Length – Indicates the number of items in the test. The upper and lower tolerances
are also provided. The “Available Items in the Item Bank” shows the number of available
items in the item bank.

3. Sub-Section Length - This is to set the length of each section. The assembler presents
various section’s item present in the item bank for the author’s selection. The upper and
lower limit on the selected sub-section is set and added to the test. The assembler
warns the author, if the sum of the number of items in the lower and upper limits
crosses the allowed tolerances to the test length. On selecting an domain for inclusion in
the test, the number of items available in the item bank on the selected domain is
displayed. This helps the author in setting the limits.

4. Question-Type – This section sets the constraint as to the limits of question type in the
test. The selected question type with its number limits is added as a constraint to the
test. The assembler also displays the number of items in the selected question type in
the item bank which aides the author in setting the limits.

5. Graphics Items – If the author requires some number of items in the test which carries
graphical items like figure, chart or picture, this can be accomplished by setting this as a
constraint to the test. The tolerances to the desired number of graphical items are also
set. The assembler displays the number of items with graphics available in the item
bank.

The below is the screen shot of the UI for setting the Item Level constraints

Page 6
ExcelSoft Knowledge Labs Author : Ranganath S

6. Enemy Items – The inclusion of one item in the test helps in answering another item in
the same test. These items are known as Enemy Items. Enemy Items are to be identified
while item authoring and set as meta-data. If the “Exclude Enemy Items” radio button is
checked, the assembler will exclude the enemy items being in the same test otherwise
will not check for the enemy items.

7. Physical Limits – Limits the number of words in the items selected for the test. On
selecting a particular word length and the tolerance limits, the assembler displays the

Page 7
ExcelSoft Knowledge Labs Author : Ranganath S

number of qualifying items in the item bank in each of the knowledge domain for the
set word constraint, which gives an idea to the author.

8. Item Exposure – When set the number of exposures in the previous tests, the assembler
lists the qualifying items in each of the knowledge domains.

9. Item authored date – When the item author date is set, items which are authored
within this set date are enumerated and displayed in each of the knowledge domains.

10. The below UI sets one of the most important specification for the test construction i.e.
Test Information function.

Page 8
ExcelSoft Knowledge Labs Author : Ranganath S

The user is given the flexibility to set the information curve for the test on which the
items are selected for the test. The “Target Information Curve” is the space where the user sets
the Test Information function. The user can fine tune the values set by defining a maximum of 5
theta points in the TIF which are of significance for the measurement goal in the “Select a
maximum of five important theta points in the curve” section and get the information at these
points. These values are later used as one more constraint for the test construction.
The objective function minimizes the sum of the (positive)
differences between the test information function and target values at a series of theta values,
k=1,...,K. Because information functions have a well-behaved continuous form, only a few

Page 9
ExcelSoft Knowledge Labs Author : Ranganath S

theta values are necessary. Practical experience has shown that a choice of three-five
values is generally sufficient.
The Main Objective set for the assembler is to minimize the information difference
between the target information value set above and the information function obtained by
picking the optimal set of items conforming to all constraints set above from the given item
bank.
On selecting the Construct, the test is constructed and displays the set of items. Cancel

quits the test construction. The selected test items are listed in the area marked “Selected Test
Items”. The target test information function and the information curve for the selected set of
items are displayed in the area “Test Information Curve”. On selecting an item in the above list
the assembler lists set of items in the “Item Bank” which matches the statistics of the selected

Page 10
ExcelSoft Knowledge Labs Author : Ranganath S

item. The user can replace the item selected for test with the one in the Item Bank. The “Test
Information Curve” displays the modified information curve. The other 2 functionalities of
“Undo Last Replacement” and “Undo All Replacements” are for undoing the least recent and all
the changes made respectively.

Reference:
1. Linear models of optimal test design By Wim J van der Linden.

Page 11